vacuous-tests-mcp

An MCP server that finds tests which pass no matter what the code does.

A vacuous test is worse than a missing one. A missing test is visibly missing. A vacuous test sits in the suite, runs green, counts in the total, and gets quoted as evidence that a behaviour is covered — while checking nothing. It is coverage that is invisibly absent, and it survives exactly the situations tests exist to catch.

They are easy to write by accident and hard to spot by reading, because a vacuous test and a real one often look nearly identical.

The case this was built from

A Rust file, 6,764 lines, in a project whose test suite was already under regular manual audit. Two tests in it:

// flagged
const SRC: &str = include_str!("lib.rs");
assert!(SRC.contains("pub sas_verified: bool,"), "the field must be exposed");

// not flagged
const SRC: &str = include_str!("lib.rs");
let prod = SRC.split_once("\n#[cfg(test)]\nmod tests {").expect("test module").0;
assert!(prod.contains("InviterSecrets::create_for_group("), "production never calls it");

Both embed the file's own source. The first asserts on SRC directly — and SRC contains the test itself, including the very string being searched for. It is true by construction: delete the production code it claims to guard and it stays green. The second cuts the test module off first and asserts on the production slice only, so it fails when the production code changes. That one is a real gate.

Scanning that file reports one finding, at the right line, and leaves the other four include_str! sites alone. The scanner follows the binding rather than pattern-matching on include_str!, which is what separates the two cases.

Related MCP server: openhonest-umbra

Rules

Languages

The Rust and JS scanners mask string literals and comments before matching, so a { inside a string or a commented-out assertion cannot mislead them. They are tuned to miss a case rather than invent one: a false positive costs more than a false negative here, because the first wrong answer teaches people to ignore the output.

Treat every finding as a question to check, not a verdict. Each one names a file and a line, so confirming it takes seconds.

Install

Not on PyPI yet — install from source:

git clone https://github.com/zegroged/vacuous-tests-mcp
cd vacuous-tests-mcp
pip install .

That puts a vacuous-tests-mcp command on your PATH.

Use it from an MCP client

Add to your client's MCP configuration:

{
  "mcpServers": {
    "vacuous-tests": {
      "command": "vacuous-tests-mcp"
    }
  }
}

For Claude Code:

claude mcp add vacuous-tests -- vacuous-tests-mcp

Then ask it to scan something:

Scan ./src for tests that can't fail.

Tools

scan_tests(path, include_skipped=True, max_findings=100) Walk a file or directory and report tests that cannot fail. Build and dependency directories (target, node_modules, .venv, …) are skipped. Findings come back highest severity first, each with a path, line, test name, rule and snippet.

list_rules() Every rule with a description, so a model can decide what to ask for.

explain_rule(rule) What one rule detects and how the finding is usually resolved.

The server only reads. It does not write files, does not execute the code it scans, and does not look outside the path it was given.

Development

pip install -e ".[dev]"
pytest

The suite covers each rule in each language, and — more importantly — checks that a normal test sitting next to a vacuous one is not reported. There is also an end-to-end test that starts the server as a subprocess and drives it through a real MCP handshake, so the protocol layer is covered rather than assumed.

Limitations

• Rust and JS/TS detection is textual, not a parse. Macro-generated tests, unusual formatting and heavily nested closures can be missed.

• no-assertions does not know that a test may exist purely to prove something does not panic. Such a test is a true positive by the rule and a false positive by intent; say so with an explicit assertion and it goes quiet.

• Only the languages in the table above are scanned. Other files are ignored, not guessed at.

License

MIT