Spec Evidence Lab

Español ·

A Python CLI that checks required test IDs against JUnit results and reviews structured change proposals. It identifies missing, failed or skipped tests and proposals that do not match the specification’s hash, requirements, tests or allowed paths.

The specification declares stable IDs, expected tests, allowed paths and non-goals. A passing result describes consistency with those declarations. The reviewer must still check where the test report came from and whether the tests verify the intended behavior. A valid proposal remains subject to human review and executes no changes. The CLI uses the standard library; an optional MCP adapter exposes the same checks over stdio.

Run the working example

Requires Python 3.11 or newer. From this directory:

python3 -m unittest discover -s tests -v
python3 tools/run_demo_tests.py --output artifacts/demo.xml
python3 -m spec_evidence check --spec examples/offline_catalog/spec.json --junit artifacts/demo.xml
python3 -m spec_evidence review --spec examples/offline_catalog/spec.json --proposal examples/offline_catalog/proposal.json

The bundled catalog tests really execute before their JUnit report is checked. They cover preserving cached data, a first-load failure, recovery, repeated reservations and unknown items. The sample proposal is a synthetic example of structured planning input, not a claim that an external model generated it.

To see a rejected proposal:

python3 -m spec_evidence review --spec examples/offline_catalog/spec.json --proposal examples/offline_catalog/proposal-stale.json

That command returns exit code 1 and needs_revision. digest --spec ... computes the canonical specification hash when a reviewed contract changes.

Related MCP server: deepbom

What is checked

Inputs are bounded to 2 MiB. JSON uses UTF-8, at most 64 nesting levels, finite numbers and integers up to 128 digits. XML DTD/entity declarations are rejected. The supported JUnit shape uses testsuite/testsuites without a namespace and testcase classname plus name. The checker computes outcomes from cases and checks declared totals for contradictions. It does not execute commands from JSON, inspect a repository automatically or modify files.

Exit codes: 0 for a consistent result, 1 for failed acceptance or a proposal needing revision, 2 for invalid input.

Language and project preferences

The default JSON report stays unchanged. To add an optional human summary, use --language en|es or a bounded JSON preferences file containing only language. An explicit flag overrides the file:

python3 -m spec_evidence check --spec examples/offline_catalog/spec.json --junit artifacts/demo.xml --config examples/preferences.es.json
python3 -m spec_evidence review --spec examples/offline_catalog/spec.json --proposal examples/offline_catalog/proposal.json --config examples/preferences.es.json --language en

The added presentation object contains the language and summary. Machine keys, status values, test identities, exit codes and the specification hash do not change. Unknown configuration fields, duplicate keys, unsupported languages and oversized input fail validation. Diagnostics and MCP responses retain their machine contract; the language preference only changes the presentation layer.

Human review in the terminal

Use --format text for a compact result with criterion/test IDs, issue codes and the specification hash. --language en|es selects the summary; issue codes stay stable, while proposal explanations tell you how to revise each field. Terminal controls in supplied identifiers are escaped. The default remains JSON for automation, and MCP is unchanged.

python3 -m spec_evidence check --spec examples/offline_catalog/spec.json --junit artifacts/demo.xml --format text
python3 -m spec_evidence review --spec examples/offline_catalog/spec.json --proposal examples/offline_catalog/proposal-stale.json --format text --language es
python3 -m spec_evidence check --help

Text output keeps the same exit codes: 0 for a consistent result, 1 for failed evidence/revision and 2 for invalid input. A passing text summary still requires review of evidence provenance and test adequacy. See presentation decisions.

Optional MCP server

The official Python MCP SDK is pinned in the optional dependency group:

python3 -m venv .venv
.venv/bin/pip install -e '.[mcp]'
.venv/bin/python tools/run_demo_tests.py --output artifacts/demo.xml
.venv/bin/python tools/check_mcp.py

The smoke check starts a real server subprocess, negotiates MCP, lists tools, checks structured output, submits valid evidence, rejects a stale plan and observes an invalid-contract tool error. It uses the official SDK client; individual IDE/assistant integrations have not been certified by this check.

Server entry point: .venv/bin/python -m spec_evidence.mcp_server. Tools:

• check_acceptance(spec_json, junit_xml)

• review_ai_proposal(spec_json, proposal_json)

Both tools accept contents, not filesystem paths. They have read-only, idempotent annotations, use no model API or credentials and expose no filesystem or command-execution tool. They do not listen on a network port.

Read the implementation

• contracts.py: contract validation and canonical hash.

• evidence.py: acceptance-to-test traceability.

• proposals.py: structured planning review.

• mcp_server.py: optional protocol adapter.

• tests/: positive and negative behavior checks.

• examples/offline_catalog/: original fixture, contract and proposals.

Current checks: 44 validator/configuration/presentation tests, five example tests and four real MCP protocol checks. Verification records the commands, machine-output comparisons, failure cases and hosted results. The optional adapter uses the official MCP Python SDK, pinned to 2.3.0.

License

Original source and documentation: MIT, copyright 2026 Manuel Naval. Third-party dependencies keep their own licenses.