# test-trust

Tells AI coding agents (Claude Code, Codex, OpenHands, Cursor, or a local model —
agent-agnostic via [MCP](https://modelcontextprotocol.io), the Model Context
Protocol) not just *which* tests cover a change, but whether those tests can
actually be **trusted** to catch a regression there.

"CI is green" and "this change is safe" are treated as the same thing by every
agent today. They aren't. A test that covers a function isn't the same as a test
that would actually fail if that function broke — over-mocked tests, snapshot
tests that just re-record whatever the new output is, and assertion-free tests
all show green while catching nothing.

## How

1. **Mutation testing** (wraps mature existing engines, not reimplemented —
   mutmut for Python, StrykerJS for JS/TS, gremlins for Go, cargo-mutants for
   Rust): deliberately introduce small bugs into the changed code, check
   whether the covering tests actually fail. Test selection per mutant comes
   from each engine's own coverage data, not reimplemented here — mutmut and
   Stryker (where a supported runner: jest/mocha/vitest) select only the
   specific tests that cover each mutant; gremlins and cargo-mutants skip
   mutants with zero coverage but re-run the full relevant test suite for
   every mutant that has any, a coarser (and slower on large repos) but
   still correct mechanism.
2. **Diff scoping** (ours): maps a git diff's changed lines to the enclosing
   function via each language's own parsing, so both the mutation run and the
   trust score are scoped to what actually changed, not a whole file/repo.
3. **Fusion** (the new part, not built anywhere else — verified): combine the
   above into one trust score per changed function, exposed as an MCP tool
   any agent can call before treating a green test run as evidence of safety.

See `examples/weak-test-fixture/` (Python), `examples/weak-test-fixture-js/`
(JS), `examples/weak-test-fixture-go/` (Go), `examples/weak-test-fixture-rust/`
(Rust), `examples/plug-and-play-fixture/` (zero config at all), and
`examples/multi-file-fixture-js/` (a function that imports a sibling module,
proving mutation-scoping doesn't break cross-file imports) for working, live
demonstrations: a function covered by a test that passes today but wouldn't
catch a real bug.

## Validated against real repos, not just fixtures

Every fixture above is a toy. Before trusting the concept, this was also run
zero-config against real, external, unmodified repositories in all four
languages: `psf/requests` (Python, 37 real functions scored, e.g.
`resolve_proxies` correctly flagged at trust 0.0 — genuinely zero tests
reference it), `kind-of` (JS, ~50M weekly downloads, `isArray` and
`isRegexp` flagged despite all 36 tests passing), `dustin/go-humanize` (Go),
and `chronotope/humantime` (Rust, a genuine multi-module crate — confirms
mutation-scoping doesn't break on real cross-module code, not just our own
fixture). That process caught and fixed several real bugs in three of the
four — including one where a failed mutation run was silently reported as a
false 100% trust score, and a scaling bug where a request scoped to one file
silently mutated an entire real package instead. Rust's real-repo check
didn't surface a new bug, consistent with `cargo-mutants -f` being the
cleanest, most directly-supported scoping mechanism of the four. See
`docs/architecture.md` for the full account.

## External tools this wraps (install once, per language you use)

- Python: `mutmut` — installed automatically as a dependency of this project.
- JS/TS: `@stryker-mutator/core` — fetched automatically via `npx` on first use.
- Go: `gremlins` — `go install github.com/go-gremlins/gremlins/cmd/gremlins@latest`
- Rust: `cargo-mutants` — `cargo install cargo-mutants`

## Try it

**Not yet published to PyPI.** Every command below assumes a local clone of
this repo, run from inside it (`uv run --directory <path> ...` from outside
it, as in the agent-embedding example further down). Once published, all of
that collapses to a plain `pip install test-trust` / `uvx test-trust`, runnable
from anywhere — no clone, no path to remember, no `uv run --directory`. That
single change is the last step between "works, with setup" and "actually
easy to adopt"; nothing about how the tool behaves changes, only how you get
it.

Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/). Per-language
mutation engines (see above) are only needed for the languages you actually
use — the Python example below needs nothing beyond `uv sync`.

```
uv sync
uv run test-trust check examples/plug-and-play-fixture inventory.py
```

No setup file, no manual mutation-testing run — this one command auto-detects
the language, auto-writes config, auto-runs mutation testing scoped to that
file using the repo's own real test command, and prints the trust score.

To run against a different file or repo:

```
uv run test-trust check <path-to-repo> <source-file> [--changed-file F] [--base-ref REF] [--threshold T]
```

**Changing the threshold** (default `0.5`) — what counts as adequately
tested. Raise it (e.g. `0.8`) to flag anything short of near-total mutation
coverage; lower it to only flag the worst gaps. Same parameter, three
surfaces:

- **CLI**: `--threshold 0.8`, as shown above.
- **MCP tool**: `threshold` is a `get_test_trust` argument, e.g.
  `get_test_trust(repo_path=..., source_file=..., threshold=0.8)`. You don't
  call this yourself — the agent does — so setting it means telling the
  agent to (in your prompt, or as a standing instruction in `CLAUDE.md`:
  "call get_test_trust with threshold=0.8"). Left unset, it uses `0.5`.
- **GitHub Action**: the `low-trust-threshold` input in your workflow file:
  ```yaml
  - uses: ./.github/actions/test-trust-pr
    with:
      github-token: ${{ secrets.GITHUB_TOKEN }}
      low-trust-threshold: "0.8"
  ```

## Embed it in an agent

One-time setup, registered globally rather than per-project — since
`repo_path` is a parameter on every call rather than fixed at startup, one
registration serves every project you open afterward, not just the one it
was set up in. For Claude Code, add to `~/.claude.json` (user-level, not a
project's `.mcp.json`):

```json
{
  "mcpServers": {
    "test-trust": {
      "command": "uv",
      "args": ["run", "--directory", "/absolute/path/to/test-trust", "test-trust", "mcp"]
    }
  }
}
```

Same `command`/`args` shape works for Codex, Cursor, or any other MCP
client — see `docs/embedding.md` for those and for what changes once this
is published to PyPI (`uvx test-trust mcp`, no local path needed at all).

After that it's invisible day to day: the agent calls `get_test_trust`
itself, mid-task, the same way it already calls its file-read or bash
tools — you don't call it directly.

## Or run it on every PR, no agent required

`.github/actions/test-trust-pr` scores every changed, supported file in a PR
and posts (or updates, on later pushes) one comment — flags any function
whose existing tests wouldn't actually catch a regression there. Useful to
any team doing code review, whether or not anyone's using an AI agent.
`.github/workflows/test-trust.yml` is the working example; verified end to
end (diff detection, scoring, both the "all clear" and "flagged" comment
renderings) against real commits in a real repo — only the live GitHub API
call itself is untested, since that needs an actual PR to verify against.

## Developing

```
uv sync
uv run pytest tests/            # this project's own unit test suite
uv run test-trust check <repo> <file>   # exercise it against real code
```

`tests/` covers the deterministic logic (diff scoping, trust-score
aggregation, language auto-detection, each mutation engine's report
adapter) with crafted fixtures — fast, no external mutation-testing tools
required, and this suite is what actually runs in CI
(`.github/workflows/ci.yml`). It doesn't replace the real-repo validation
above — that was a manual, one-time exercise done during development (real
external clones, real engines installed), not something CI re-runs on every
push. Worth automating later; not done yet.

MIT licensed.