# vacuous-tests-mcp

An MCP server that finds tests which pass no matter what the code does.

A vacuous test is worse than a missing one. A missing test is visibly missing. A vacuous test
sits in the suite, runs green, counts in the total, and gets quoted as evidence that a
behaviour is covered — while checking nothing. It is coverage that is invisibly absent, and it
survives exactly the situations tests exist to catch.

They are easy to write by accident and hard to spot by reading, because a vacuous test and a
real one often look nearly identical.

## The case this was built from

A Rust file, 6,764 lines, in a project whose test suite was already under regular manual audit.
Two tests in it:

```rust
// flagged
const SRC: &str = include_str!("lib.rs");
assert!(SRC.contains("pub sas_verified: bool,"), "the field must be exposed");
```

```rust
// not flagged
const SRC: &str = include_str!("lib.rs");
let prod = SRC.split_once("\n#[cfg(test)]\nmod tests {").expect("test module").0;
assert!(prod.contains("InviterSecrets::create_for_group("), "production never calls it");
```

Both embed the file's own source. The first asserts on `SRC` directly — and `SRC` contains the
test itself, including the very string being searched for. It is true by construction: delete
the production code it claims to guard and it stays green. The second cuts the test module off
first and asserts on the production slice only, so it fails when the production code changes.
That one is a real gate.

Scanning that file reports **one** finding, at the right line, and leaves the other four
`include_str!` sites alone. The scanner follows the binding rather than pattern-matching on
`include_str!`, which is what separates the two cases.

## Rules

| Rule | Severity | What it catches |
| --- | --- | --- |
| `self-referential-source` | high | The test embeds its own source and asserts a literal appears in it. The literal is in the assertion, so it can never fail. |
| `no-assertions` | high | No assertion of any kind. Only a panic or throw can fail the test, so wrong-but-quiet behaviour passes. |
| `tautological-assertion` | high | `assert!(true)`, `assert_eq!(x, x)`, `expect(true).toBe(true)` — holds regardless of the code. |
| `empty-body` | high | Nothing in the body to fail. |
| `skipped-test` | info | `#[ignore]`, `it.skip`, `@pytest.mark.skip`. Runs green because it does not run. |

## Languages

| Language | Method | Accuracy |
| --- | --- | --- |
| Python | `ast` from the standard library | Exact |
| Rust | brace-matching scanner over the source text | Heuristic |
| JavaScript / TypeScript | brace-matching scanner over the source text | Heuristic |

The Rust and JS scanners mask string literals and comments before matching, so a `{` inside a
string or a commented-out assertion cannot mislead them. They are tuned to miss a case rather
than invent one: a false positive costs more than a false negative here, because the first
wrong answer teaches people to ignore the output.

Treat every finding as a question to check, not a verdict. Each one names a file and a line, so
confirming it takes seconds.

## Install

Not on PyPI yet — install from source:

```bash
git clone https://github.com/zegroged/vacuous-tests-mcp
cd vacuous-tests-mcp
pip install .
```

That puts a `vacuous-tests-mcp` command on your PATH.

## Use it from an MCP client

Add to your client's MCP configuration:

```json
{
  "mcpServers": {
    "vacuous-tests": {
      "command": "vacuous-tests-mcp"
    }
  }
}
```

For Claude Code:

```bash
claude mcp add vacuous-tests -- vacuous-tests-mcp
```

Then ask it to scan something:

> Scan ./src for tests that can't fail.

## Tools

**`scan_tests(path, include_skipped=True, max_findings=100)`**
Walk a file or directory and report tests that cannot fail. Build and dependency directories
(`target`, `node_modules`, `.venv`, …) are skipped. Findings come back highest severity first,
each with a path, line, test name, rule and snippet.

**`list_rules()`**
Every rule with a description, so a model can decide what to ask for.

**`explain_rule(rule)`**
What one rule detects and how the finding is usually resolved.

The server only reads. It does not write files, does not execute the code it scans, and does
not look outside the path it was given.

## Development

```bash
pip install -e ".[dev]"
pytest
```

The suite covers each rule in each language, and — more importantly — checks that a normal test
sitting next to a vacuous one is *not* reported. There is also an end-to-end test that starts
the server as a subprocess and drives it through a real MCP handshake, so the protocol layer is
covered rather than assumed.

## Limitations

- Rust and JS/TS detection is textual, not a parse. Macro-generated tests, unusual formatting
  and heavily nested closures can be missed.
- `no-assertions` does not know that a test may exist purely to prove something does not panic.
  Such a test is a true positive by the rule and a false positive by intent; say so with an
  explicit assertion and it goes quiet.
- Only the languages in the table above are scanned. Other files are ignored, not guessed at.

## License

MIT