Prove your tests

Your tests pass. Do they catch bugs? Prove it.

Measured on the example in this repo (reproduce with bash examples/shop/run_demo.sh /tmp/r):

Try it in 60 seconds

pipx install git+https://github.com/quartercharts-com/prove-your-tests
git clone https://github.com/quartercharts-com/prove-your-tests && cd prove-your-tests
PYT=prove-your-tests bash examples/shop/run_demo.sh /tmp/r && cat /tmp/r/weak-tests-vs-code-changes.md

The first report shows 15 SURVIVED plants: each one is a line no test checks.

Related MCP server: TestHeal

Use it with your AI

Then ask: "prove the tests for refunds". The AI writes plants without reading your tests; the verdict comes from a script.

Why you can trust it

• It never leaves your code changed. It refuses uncommitted work, journals every file before it touches it, and restores after a crash, Ctrl-C, a kill or a closed terminal; recover handles kill -9. Each case is a test in tests/test_prover.py (class Safety).

• No AI decides PASS. The verdict is a plain script. A timeout or a flaky test is NOT-RUN, never a kill.

• Keys stay yours. The optional headless plant writer reads an API key only from an environment variable, never stores, prints or logs it (tested).

• It proves itself. 91 planted bugs in the runner itself, written by a separate writer that never read the self-test, run by CI.

Where it helps

Illustrations of the kind of bug it exposes, not measured results. Longer examples: docs/use-cases.md.

How it compares

Use mutmut, Stryker or PIT for broad, fast mutation scores over a whole codebase. Use prove-your-tests where a specific test or guard must be proven: money, accounts, access, and code an AI wrote.

How it works

A green test only proves the test ran. This skill plants small deliberate bugs in your code, one at a time. For each one it runs your tests, which must go red. It then puts the file back byte for byte and runs the tests again, which must go green. Any planted bug your tests miss is a hole, and the skill shows you where it is.

Pre-release (0.1.0). MIT licensed. Works with any AI agent, or with none: the verdict is always a plain script.

What it checks

• Behavior: if someone changes a line (>= to >, 50 to 51, drops a call), does a test fail?

• Guards: if someone adds code that sneaks around a rule ("only refund() may move money"), through an alias, getattr, a stored function, an env flag, a second module, an error path, does the guard test notice?

The rules that make the proof honest

1. The plants are written by someone who never reads the tests. A separate plant writer (an agent) sees only the code, a scan of how the code is really called, and a catalog of 24 bug forms. Plants chosen by the person who wrote the tests share the tests' blind spots.

2. At least 25 plants, and every catalog form covered (or marked not applicable, with a reason).

3. PASS only when everything is proven: every plant ran and turned the tests red, every file came back byte-identical, and the tests were green again afterwards. Anything not run is NOT-RUN, never PASS.

4. The runner is a script, not a model. No model reads test output to decide a verdict.

5. A plant is never left behind. The runner refuses uncommitted work, journals every file before it changes it, and puts files back after a crash, Ctrl-C, a kill or a closed terminal (recover after kill -9).

6. A hang or a flaky test is never a kill. Timeouts and tests that fail without a plant are NOT-RUN.

The idea in one example

def is_adult(age):
    return age >= 18

def test_adult():
    assert is_adult(30)

The test is green. The skill plants age > 18, and the test is still green: the plant SURVIVED. Nothing checks the line itself, so an 18-year-old could be turned away and no test would notice. The skill reports the hole, and you add the test that was missing:

def test_the_line():
    assert is_adult(18)
    assert not is_adult(17)

Run it again on the same plants, and the plant is now KILLED.

Worked example: a tiny shop

examples/shop/shop.py has refunds and a money ledger. The rule: money leaves only through refund(), and a customer may refund at most 50. There are three sets of tests:

The effect guard did not pass the first time. The skill found 3 real holes, and each became a test:

• MAX_SELF_REFUND = 50 changed to 51 went unnoticed. The test read the code's own constant, so it moved with the change. Fix: write 50 and 51 into the test.

• amount <= 0 changed to amount <= 1 went unnoticed. No test refunded exactly 1. Fix: test the smallest legal refund.

• A customer refund of 100 went through when the limit compared numbers as text. As text, "100" < "50". The test only tried 51, which happens to compare correctly. Fix: try 51, 99, 100, 499 and 1000.

Run it yourself (it works on a copy, never on your files; needs python3 and git):

bash examples/shop/run_demo.sh /tmp/shop-reports
cat /tmp/shop-reports/effect-guard-vs-escapes.md

Install

Pick one. Nothing is on a package registry yet; every option installs from this repository.

Then, in Claude Code, ask to "prove the tests for refunds" or type /prove-your-tests. On the command line:

prove-your-tests plant --repo . --out plants.json src/refunds.py      # or have any AI write plants.json
prove-your-tests check-plants plants.json --repo .
prove-your-tests prove --repo . --plants plants.json --test-cmd "python3 -m pytest -q tests/test_refunds.py" --out .prove-out/refunds

MCP clients that take a JSON config (Claude Desktop, Cursor .cursor/mcp.json, and others):

{ "mcpServers": { "prove-your-tests": { "command": "prove-your-tests", "args": ["mcp"] } } }

Not built yet: Homebrew and npm packages.

Works with

• Run here: the command line with no model at all (plants given as a file), Claude Code (skill, plant-writer agent, plugin manifests validated), the MCP server (stdio, as a script), the headless plant writer against a local fake of the Anthropic and OpenAI-compatible APIs.

• Format checked, not run: OpenAI Codex and other AGENTS.md readers, Gemini CLI, Cursor rules, MCP configs for Claude Desktop and Cursor, the GitHub Action on GitHub, real calls to OpenAI, Anthropic, Moonshot (Kimi), DeepSeek or OpenRouter. ChatGPT connects only to remote MCP servers, so the local server does not plug into it directly.

• Tested on: macOS with Python 3.12. Declared for Python 3.9 or newer on macOS and Linux (CI). Windows is not tested.

Which model for which step

The headless writer for CI takes any provider: prove-your-tests write-plants --provider anthropic|openai --model <name> [--base-url <OpenAI-compatible URL>] .... It reads the key only from an environment variable (ANTHROPIC_API_KEY, OPENAI_API_KEY, or --key-env), never prints it, and writes nothing unless the model's answer passes check-plants and every edit applies.

Settings

skill/prove-your-tests/config.default.json holds the defaults. Copy it to .prove-your-tests.json in your repo to change them.

• Plant writer: the strongest model, high effort. On a small plan, a cheaper model can write the simple, repetitive plants.

• Runner: no model.

• Proof: at least 25 plants, 3 green baseline runs, a control run after every plant, a 300 second timeout per run.

The skill proves itself too

tests/test_prover.py checks the runner's own promises in 60 cases: verdicts, exit codes, stale plants, created files, dirty trees, flaky tests, timeouts, apply failures, signals, a real kill -9 and recovery, shards. tests/test_adapters.py checks the CLI, the MCP server, the headless writer and the adapter files in 18 cases. tests/plants/prover.json holds 91 plants that would each make the runner lie, for example "count a timeout as a kill", "skip the byte check" or "accept 24 plants when 25 are required", written by a separate plant writer that never read the self-test. CI runs the self-test against all 91 in 8 shards (.github/workflows/ci.yml).

Files

Badge

Proved your tests? Add the badge to your README. It is a plain static image from shields.io: nothing is hosted or tracked by us, and it links here.

[![tests proved with prove-your-tests](https://img.shields.io/badge/tests-proved%20with%20prove--your--tests-2ea44f)](https://github.com/quartercharts-com/prove-your-tests)

License

MIT, see LICENSE. Security reports: see SECURITY.md. Contributing: see CONTRIBUTING.md.

Made by QuarterCharts.