test-trust

Tells AI coding agents (Claude Code, Codex, OpenHands, Cursor, or a local model — agent-agnostic via MCP, the Model Context Protocol) not just which tests cover a change, but whether those tests can actually be trusted to catch a regression there.

"CI is green" and "this change is safe" are treated as the same thing by every agent today. They aren't. A test that covers a function isn't the same as a test that would actually fail if that function broke — over-mocked tests, snapshot tests that just re-record whatever the new output is, and assertion-free tests all show green while catching nothing.

How

1. Mutation testing (wraps mature existing engines, not reimplemented — mutmut for Python, StrykerJS for JS/TS, gremlins for Go, cargo-mutants for Rust): deliberately introduce small bugs into the changed code, check whether the covering tests actually fail. Test selection per mutant comes from each engine's own coverage data, not reimplemented here — mutmut and Stryker (where a supported runner: jest/mocha/vitest) select only the specific tests that cover each mutant; gremlins and cargo-mutants skip mutants with zero coverage but re-run the full relevant test suite for every mutant that has any, a coarser (and slower on large repos) but still correct mechanism.

2. Diff scoping (ours): maps a git diff's changed lines to the enclosing function via each language's own parsing, so both the mutation run and the trust score are scoped to what actually changed, not a whole file/repo.

3. Fusion (the new part, not built anywhere else — verified): combine the above into one trust score per changed function, exposed as an MCP tool any agent can call before treating a green test run as evidence of safety.

See examples/weak-test-fixture/ (Python), examples/weak-test-fixture-js/ (JS), examples/weak-test-fixture-go/ (Go), examples/weak-test-fixture-rust/ (Rust), examples/plug-and-play-fixture/ (zero config at all), and examples/multi-file-fixture-js/ (a function that imports a sibling module, proving mutation-scoping doesn't break cross-file imports) for working, live demonstrations: a function covered by a test that passes today but wouldn't catch a real bug.

Related MCP server: sumo-qa

Validated against real repos, not just fixtures

Every fixture above is a toy. Before trusting the concept, this was also run zero-config against real, external, unmodified repositories in all four languages: psf/requests (Python, 37 real functions scored, e.g. resolve_proxies correctly flagged at trust 0.0 — genuinely zero tests reference it), kind-of (JS, ~50M weekly downloads, isArray and isRegexp flagged despite all 36 tests passing), dustin/go-humanize (Go), and chronotope/humantime (Rust, a genuine multi-module crate — confirms mutation-scoping doesn't break on real cross-module code, not just our own fixture). That process caught and fixed several real bugs in three of the four — including one where a failed mutation run was silently reported as a false 100% trust score, and a scaling bug where a request scoped to one file silently mutated an entire real package instead. Rust's real-repo check didn't surface a new bug, consistent with cargo-mutants -f being the cleanest, most directly-supported scoping mechanism of the four. See docs/architecture.md for the full account.

External tools this wraps (install once, per language you use)

• Python: mutmut — installed automatically as a dependency of this project.

• JS/TS: @stryker-mutator/core — fetched automatically via npx on first use.

• Go: gremlins — go install github.com/go-gremlins/gremlins/cmd/gremlins@latest

• Rust: cargo-mutants — cargo install cargo-mutants

Try it

Not yet published to PyPI. Every command below assumes a local clone of this repo, run from inside it (uv run --directory <path> ... from outside it, as in the agent-embedding example further down). Once published, all of that collapses to a plain pip install test-trust / uvx test-trust, runnable from anywhere — no clone, no path to remember, no uv run --directory. That single change is the last step between "works, with setup" and "actually easy to adopt"; nothing about how the tool behaves changes, only how you get it.

Requires Python 3.11+ and uv. Per-language mutation engines (see above) are only needed for the languages you actually use — the Python example below needs nothing beyond uv sync.

uv sync
uv run test-trust check examples/plug-and-play-fixture inventory.py

No setup file, no manual mutation-testing run — this one command auto-detects the language, auto-writes config, auto-runs mutation testing scoped to that file using the repo's own real test command, and prints the trust score.

To run against a different file or repo:

uv run test-trust check <path-to-repo> <source-file> [--changed-file F] [--base-ref REF] [--threshold T]

Changing the threshold (default 0.5) — what counts as adequately tested. Raise it (e.g. 0.8) to flag anything short of near-total mutation coverage; lower it to only flag the worst gaps. Same parameter, three surfaces:

• CLI: --threshold 0.8, as shown above.

• MCP tool: threshold is a get_test_trust argument, e.g. get_test_trust(repo_path=..., source_file=..., threshold=0.8). You don't call this yourself — the agent does — so setting it means telling the agent to (in your prompt, or as a standing instruction in CLAUDE.md: "call get_test_trust with threshold=0.8"). Left unset, it uses 0.5.

• GitHub Action: the low-trust-threshold input in your workflow file:

  - uses: ./.github/actions/test-trust-pr
    with:
      github-token: ${{ secrets.GITHUB_TOKEN }}
      low-trust-threshold: "0.8"

Embed it in an agent

One-time setup, registered globally rather than per-project — since repo_path is a parameter on every call rather than fixed at startup, one registration serves every project you open afterward, not just the one it was set up in. For Claude Code, add to ~/.claude.json (user-level, not a project's .mcp.json):

{
  "mcpServers": {
    "test-trust": {
      "command": "uv",
      "args": ["run", "--directory", "/absolute/path/to/test-trust", "test-trust", "mcp"]
    }
  }
}

Same command/args shape works for Codex, Cursor, or any other MCP client — see docs/embedding.md for those and for what changes once this is published to PyPI (uvx test-trust mcp, no local path needed at all).

After that it's invisible day to day: the agent calls get_test_trust itself, mid-task, the same way it already calls its file-read or bash tools — you don't call it directly.

Or run it on every PR, no agent required

.github/actions/test-trust-pr scores every changed, supported file in a PR and posts (or updates, on later pushes) one comment — flags any function whose existing tests wouldn't actually catch a regression there. Useful to any team doing code review, whether or not anyone's using an AI agent. .github/workflows/test-trust.yml is the working example; verified end to end (diff detection, scoring, both the "all clear" and "flagged" comment renderings) against real commits in a real repo — only the live GitHub API call itself is untested, since that needs an actual PR to verify against.

Developing

uv sync
uv run pytest tests/            # this project's own unit test suite
uv run test-trust check <repo> <file>   # exercise it against real code

tests/ covers the deterministic logic (diff scoping, trust-score aggregation, language auto-detection, each mutation engine's report adapter) with crafted fixtures — fast, no external mutation-testing tools required, and this suite is what actually runs in CI (.github/workflows/ci.yml). It doesn't replace the real-repo validation above — that was a manual, one-time exercise done during development (real external clones, real engines installed), not something CI re-runs on every push. Worth automating later; not done yet.

MIT licensed.