socrates
by Owxessus
README.md
# Socrates
<img src="assets/banner.png" alt="Socrates — mutation testing and capability proof for agent-written code">
[](https://github.com/Owxessus/socrates/actions/workflows/ci.yml) [](LICENSE) [](pyproject.toml)
> **Not on PyPI yet** — install from GitHub (below). The package name will be **`socrates-oss`**: the bare `socrates` on PyPI is an unrelated project. The command is `socrates`.
**English · [Português](README.pt-BR.md)**
**Know thyself; prove thyself. Socrates refuses a claim without proof — the claims your *tests* make, and the claims your *code* makes.**
A codebase is full of assertions nobody checked. A test that's green but asserts nothing. A public function the README promises but no test ever touches. Both *look* done. Socrates does the Socratic thing — the *elenchus*, cross-examination — to each:
- **Your tests claim to test.** → **`socrates mutate`** breaks your code on purpose, one small change at a time, and checks a test notices. A break no test catches (*mutation testing*) proves the test was empty.
- **Your project claims to do things.** → **`socrates claims`** finds the public capabilities that no test so much as names — a promise with no proof.
Σωκράτης: *"the unexamined life is not worth living."* Nor the unexamined test suite.
## Built for the loop, not the post-mortem
Agents love to write a test, watch it go green, and declare "tested" — when the test asserts nothing. That placebo is the single most expensive lie in agentic coding. Socrates isn't a report you run on a finished suite — it's the check an agent runs **the moment it writes a test**: `mutate` breaks the code the test covers and confirms the test **notices**. If it doesn't bite, the agent knows *now* — before it moves on and before it says "done" — and rewrites it into a real test.
**Primary use — the agent's self-check (MCP).** Run Socrates as an **MCP server** inside your coding agent's loop: after generating a test the agent asks *"does the test I just wrote actually catch a bug?"* and can't claim "tested" while the mutation score is a floor of survivors. (Also a CLI and CI gate.)
**Who it helps most: weaker, cheaper, autonomous models.** A frontier model writes fewer placebo tests. Socrates catches the moments a model **thinks** it tested but wrote an empty green — widest on **cheap models running long, on their own, with nobody reading the assertions.** It runs your real suite, so it costs a little; use it at the checkpoint where the agent would otherwise declare victory. On a top model it's a spot-check; on a cheap autonomous one it's what stops a suite of placebos.
> **One of three.** Siblings: **[Platão](https://github.com/Owxessus/platao)** ("did you actually finish?" — completeness & placebo detection) and **[Basanos](https://github.com/Owxessus/basanos)** ("is this button real?" — UI wiring). Platão asks *is this test shaped like a placebo?* statically; Socrates **proves** whether it bites, and whether your API is backed. They run standalone.
---
## 30 seconds
<img src="assets/demo.svg" alt="Socrates — an illustrated run: the command and the kind of report it prints" width="640">
```bash
# in the same environment as your project and its tests (Socrates runs your pytest)
pip install "socrates-oss @ git+https://github.com/Owxessus/socrates"
```
Installed on its own instead (pipx, uvx), tell it which interpreter has your project: `socrates mutate … --python .venv/bin/python` (Windows: `.venv\Scripts\python.exe`).
### Ability 1 — do your tests bite? (mutation testing)
```bash
socrates mutate src/calc.py --tests tests/test_calc.py
```
```
Socrates - calc.py
mutants: 12 - killed: 9 - survived: 3 - score: 75%
survivors (mutations your tests did NOT catch):
line 14 Add -> Sub
line 14 return <expr> -> return None
line 27 Lt -> GtE
Each survivor is a behavior no test checks - add an assertion that would fail on it.
```
### Ability 2 — does your code do what it claims? (capability-proof)
```bash
socrates claims .
```
```
Socrates - .: 2 public capabilities no test names (claimed, unproven):
src/exporter.py:40 export_pdf
src/exporter.py:58 export_csv
Each is a promise nothing proves - add a test that exercises it, or make it private.
```
Both hand you **concrete, actionable** gaps: the exact line, and the exact thing nothing proves.
---
## Why this exists
Two of the most common ways work *looks* done and isn't — and coding agents produce both constantly:
**The placebo test.** Coverage lies: "100% covered" means every line *ran*, not that any test would *fail* if the line were wrong.
```python
def test_add():
add(2, 3) # runs add() — 100% coverage!
assert True # …and asserts nothing. A placebo.
```
Mutation testing is the only technique that catches it: change `+` to `-`, and if the test still passes, it never tested addition. `socrates mutate` does that for every mutable spot.
**The unproven capability.** A public function is a promise — "this project can do X." A promise no test names is just an assertion. `socrates claims` finds every public/exported symbol that no test mentions — the API you shipped and never proved.
---
## The honest caveats (release-quality means saying these)
- **`mutate` is dynamic** — unlike Platão/Basanos (static), it *runs your test suite*, once per mutant. That's inherent to mutation testing; it's the price of a real answer. So **scope it tight**: point it at the file you just wrote and the tests that cover it — the agent-first workflow, *prove the test you just wrote bites, in seconds* — not a nightly full-suite crawl.
- **`mutate` edits the file in place while it runs** — each mutant is written over the target, and the original bytes are put back at the end. Until then the original is kept in `<file>.socrates-backup`; if a run is killed, the next run restores it (and refuses, touching nothing, if you edited the file in between). Don't edit the file while it runs.
- **`claims` is MEDIUM by design** — "no test *names* this symbol" is a strong hint, not a proof of zero coverage (a symbol can be exercised indirectly). It's lenient on purpose: *any* mention in *any* test counts as proof, so it flags only the truly-unmentioned. Point it at a directory that contains **both** your code and your tests (usually the repo root); it respects your package's `__all__` as the advertised API.
---
## Usage modes
| Mode | Command / setup |
|---|---|
| **CLI** | `socrates mutate <file> --tests <path> [--python <interpreter>]` · `socrates claims <path>` |
| **CI gate** | `socrates mutate … --fail-under 0.9` · `socrates claims . --fail-on-claims` |
| **MCP server** | `socrates mcp` — exposes `socrates_prove_tests` and `socrates_prove_claims` for an agent to prove its own work |
```bash
pip install "socrates-oss[mcp] @ git+https://github.com/Owxessus/socrates"
socrates mcp # stdio server; point your agent at it
```
A typical MCP client entry (Claude Code, Cursor and others use this shape):
```json
{ "mcpServers": { "socrates": { "command": "socrates", "args": ["mcp"] } } }
```
If the server runs outside your project's environment, the agent passes `python` (your project's interpreter) to `socrates_prove_tests`.
The gates and the MCP tools are the point: an agent (or CI) writes code, a test, and a public API, then must **prove the test catches a bug and the API is tested** before the work counts as finished.
---
## The mutation operators
Curated for high signal and few *equivalent mutants* (changes that don't alter behavior, which would look like false survivors):
| Operator | Example |
|---|---|
| Comparison / boundary | `==`↔`!=`, `<`→`>=`, `>`→`<=` |
| Boolean logic | `and`↔`or` |
| Arithmetic | `+`↔`-`, `*`↔`/` |
| Augmented assignment | `+=`↔`-=` |
| Constants | `True`↔`False`, `n`→`n+1` |
| Negation | `not x`→`x` |
| Return value | `return <expr>`→`return None` |
---
## Contributing — the rigid gate
See [CONTRIBUTING.md](CONTRIBUTING.md). Every mutation operator and every capability check must ship with:
1. **Deterministic** — pure AST/pattern, no network.
2. **`prove_effect`** — a fixture where it fires on the real defect (a mutant that survives a weak test; a public capability no test names).
3. **`negative_control`** — a fixture where it stays silent (a strong test kills the mutant; a tested capability isn't flagged) — no equivalent-mutant / indirect-coverage noise.
4. **Declared severity.**
CI runs the proofs on every PR. No proof, no merge. A tool that proves your tests bite cannot itself ship checks that don't.
---
## Proven, not asserted
Anti-placebo is a rule this project holds *itself* to. Its mutation operators and capability checks are tested with **`prove_effect` + `negative_control` pairs** — a fixture with the defect they must catch, and an honest twin they must stay silent on — and CI runs that suite on every PR. And it passes its own audit — Socrates **proves its own public API is test-backed** (`claims . --fail-on-claims` is green).
It was then tuned by running it over a range of open-source projects. Where it showed a test that didn't bite, good; where it flagged something that was fine, that pattern became a fix with a regression test. False positives still happen — when you hit one, an issue with the snippet is the most useful contribution there is.
## Where this came from
Socrates was extracted from **Athena**, a personal AI agent project, where the same two questions are asked of the code the agent writes for itself: does the test it just wrote catch a bug, and is every capability it announces backed by a test? It is released on its own because placebo tests and unproven APIs show up wherever agents write code. It has no dependency on the rest of Athena.
## License
MIT, plus the [CONTRIBUTING.md](CONTRIBUTING.md) proof gate.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues