Socrates

Not on PyPI yet — install from GitHub (below). The package name will be socrates-oss: the bare socrates on PyPI is an unrelated project. The command is socrates.

English · Português

Know thyself; prove thyself. Socrates refuses a claim without proof — the claims your tests make, and the claims your code makes.

A codebase is full of assertions nobody checked. A test that's green but asserts nothing. A public function the README promises but no test ever touches. Both look done. Socrates does the Socratic thing — the elenchus, cross-examination — to each:

• Your tests claim to test. → socrates mutate breaks your code on purpose, one small change at a time, and checks a test notices. A break no test catches (mutation testing) proves the test was empty.

• Your project claims to do things. → socrates claims finds the public capabilities that no test so much as names — a promise with no proof.

Σωκράτης: "the unexamined life is not worth living." Nor the unexamined test suite.

Built for the loop, not the post-mortem

Agents love to write a test, watch it go green, and declare "tested" — when the test asserts nothing. That placebo is the single most expensive lie in agentic coding. Socrates isn't a report you run on a finished suite — it's the check an agent runs the moment it writes a test: mutate breaks the code the test covers and confirms the test notices. If it doesn't bite, the agent knows now — before it moves on and before it says "done" — and rewrites it into a real test.

Primary use — the agent's self-check (MCP). Run Socrates as an MCP server inside your coding agent's loop: after generating a test the agent asks "does the test I just wrote actually catch a bug?" and can't claim "tested" while the mutation score is a floor of survivors. (Also a CLI and CI gate.)

Who it helps most: weaker, cheaper, autonomous models. A frontier model writes fewer placebo tests. Socrates catches the moments a model thinks it tested but wrote an empty green — widest on cheap models running long, on their own, with nobody reading the assertions. It runs your real suite, so it costs a little; use it at the checkpoint where the agent would otherwise declare victory. On a top model it's a spot-check; on a cheap autonomous one it's what stops a suite of placebos.

One of three. Siblings: Platão ("did you actually finish?" — completeness & placebo detection) and Basanos ("is this button real?" — UI wiring). Platão asks is this test shaped like a placebo? statically; Socrates proves whether it bites, and whether your API is backed. They run standalone.

Related MCP server: Rework

30 seconds

# in the same environment as your project and its tests (Socrates runs your pytest)
pip install "socrates-oss @ git+https://github.com/Owxessus/socrates"

Installed on its own instead (pipx, uvx), tell it which interpreter has your project: socrates mutate … --python .venv/bin/python (Windows: .venv\Scripts\python.exe).

Ability 1 — do your tests bite? (mutation testing)

socrates mutate src/calc.py --tests tests/test_calc.py

Socrates - calc.py
  mutants: 12 - killed: 9 - survived: 3 - score: 75%

  survivors (mutations your tests did NOT catch):
    line 14   Add -> Sub
    line 14   return <expr> -> return None
    line 27   Lt -> GtE

  Each survivor is a behavior no test checks - add an assertion that would fail on it.

Ability 2 — does your code do what it claims? (capability-proof)

socrates claims .

Socrates - .: 2 public capabilities no test names (claimed, unproven):

    src/exporter.py:40   export_pdf
    src/exporter.py:58   export_csv

  Each is a promise nothing proves - add a test that exercises it, or make it private.

Both hand you concrete, actionable gaps: the exact line, and the exact thing nothing proves.

Why this exists

Two of the most common ways work looks done and isn't — and coding agents produce both constantly:

The placebo test. Coverage lies: "100% covered" means every line ran, not that any test would fail if the line were wrong.

def test_add():
    add(2, 3)        # runs add() — 100% coverage!
    assert True      # …and asserts nothing. A placebo.

Mutation testing is the only technique that catches it: change + to -, and if the test still passes, it never tested addition. socrates mutate does that for every mutable spot.

The unproven capability. A public function is a promise — "this project can do X." A promise no test names is just an assertion. socrates claims finds every public/exported symbol that no test mentions — the API you shipped and never proved.

The honest caveats (release-quality means saying these)

• mutate is dynamic — unlike Platão/Basanos (static), it runs your test suite, once per mutant. That's inherent to mutation testing; it's the price of a real answer. So scope it tight: point it at the file you just wrote and the tests that cover it — the agent-first workflow, prove the test you just wrote bites, in seconds — not a nightly full-suite crawl.

• mutate edits the file in place while it runs — each mutant is written over the target, and the original bytes are put back at the end. Until then the original is kept in <file>.socrates-backup; if a run is killed, the next run restores it (and refuses, touching nothing, if you edited the file in between). Don't edit the file while it runs.

• claims is MEDIUM by design — "no test names this symbol" is a strong hint, not a proof of zero coverage (a symbol can be exercised indirectly). It's lenient on purpose: any mention in any test counts as proof, so it flags only the truly-unmentioned. Point it at a directory that contains both your code and your tests (usually the repo root); it respects your package's __all__ as the advertised API.

Usage modes

pip install "socrates-oss[mcp] @ git+https://github.com/Owxessus/socrates"
socrates mcp        # stdio server; point your agent at it

A typical MCP client entry (Claude Code, Cursor and others use this shape):

{ "mcpServers": { "socrates": { "command": "socrates", "args": ["mcp"] } } }

If the server runs outside your project's environment, the agent passes python (your project's interpreter) to socrates_prove_tests.

The gates and the MCP tools are the point: an agent (or CI) writes code, a test, and a public API, then must prove the test catches a bug and the API is tested before the work counts as finished.

The mutation operators

Curated for high signal and few equivalent mutants (changes that don't alter behavior, which would look like false survivors):

Contributing — the rigid gate

See CONTRIBUTING.md. Every mutation operator and every capability check must ship with:

1. Deterministic — pure AST/pattern, no network.

2. prove_effect — a fixture where it fires on the real defect (a mutant that survives a weak test; a public capability no test names).

3. negative_control — a fixture where it stays silent (a strong test kills the mutant; a tested capability isn't flagged) — no equivalent-mutant / indirect-coverage noise.

4. Declared severity.

CI runs the proofs on every PR. No proof, no merge. A tool that proves your tests bite cannot itself ship checks that don't.

Proven, not asserted

Anti-placebo is a rule this project holds itself to. Its mutation operators and capability checks are tested with prove_effect + negative_control pairs — a fixture with the defect they must catch, and an honest twin they must stay silent on — and CI runs that suite on every PR. And it passes its own audit — Socrates proves its own public API is test-backed (claims . --fail-on-claims is green).

It was then tuned by running it over a range of open-source projects. Where it showed a test that didn't bite, good; where it flagged something that was fine, that pattern became a fix with a regression test. False positives still happen — when you hit one, an issue with the snippet is the most useful contribution there is.

Where this came from

Socrates was extracted from Athena, a personal AI agent project, where the same two questions are asked of the code the agent writes for itself: does the test it just wrote catch a bug, and is every capability it announces backed by a test? It is released on its own because placebo tests and unproven APIs show up wherever agents write code. It has no dependency on the rest of Athena.

License

MIT, plus the CONTRIBUTING.md proof gate.