🧾 mcp-proof

Ship an MCP server with a receipt.

Audit an MCP server from the wire. Get conformance, security, regression — and effect — evidence in one reproducible, offline-verifiable delivery report.

v0.8 extends conformance below the response boundary — a tool can declare readOnlyHint: true, return a perfectly ordinary response, and still mint a credential that outlives the grant that authorized it. An out-of-band observer reads the real external effects; a probe exercises the authority the tools create. → the research lane

stdio + Streamable HTTP · 2026-07-28 + legacy eras · HTML / JSON / JUnit / SARIF

English · 简体中文 · 日本語 · 한국어 · Français

Browse the live reports → · Effect-aware evaluation →

A real audit of the official MCP filesystem server: 27 conformance checks, the MSSS compliance table, 34 regression replays — ship-ready, one advisory finding.

🚀 Quick start

pip install git+https://github.com/YuCPbit/mcp-proof
mcp-proof run python my_server.py --fixtures fixtures/ --record-if-missing --out report.html

Auditing a running HTTP server instead? mcp-proof run --url http://localhost:8000/mcp --out report.html

Exit codes are the gate: 0 — every MUST check passed, no blocking security findings (advisories may remain), no behavioural drift. 1 — the audit completed and the server failed it. 2 — the audit did not complete (missing baseline, internal auditor error) and proves nothing about the server, in either direction.

mcp-proof plan python my_server.py                             # what would auto-baselining call, and why
mcp-proof record python my_server.py --fixtures fixtures/      # freeze the behavioural contract
mcp-proof replay --fixtures fixtures/ -- python my_server.py   # fail on any drift
mcp-proof inspect python my_server.py --out baseline.json      # freeze the contract surface
mcp-proof diff baseline.json current.json                      # BREAKING / ADDITIVE / METADATA, exit 1 on breaking
mcp-proof verify report.json                                   # recheck the report's internal fingerprints offline
mcp-proof effects --sqlite state.db -- python my_server.py     # declared vs observed external effects (v0.8)
mcp-proof effects --fs-root data/ -- python my_server.py       # same lane, directory-backed servers

See the difference in 60 seconds with the built-in demo pair — a clean server and one with nine planted violations:

mcp-proof run python demo/good_server.py --fixtures demo/fixtures-good --out report-good.html   # → SHIP-READY
mcp-proof run python demo/bad_server.py --out report-bad.html                                    # → 5 MUST failures, 3 security findings

Related MCP server: MCProbe

🔬 The four lanes

Every lane feeds one report — and the report ends with a prioritized fix list, so it doubles as a remediation plan.

✅ Validation

An audit tool has to earn more trust than the thing it audits. What stands behind every release:

• 160 tests, including an adversarial suite that attacks the auditor itself: violations hidden on page 2 of paginated listings, tampered fixtures and manifests, hash-stripping downgrade attempts, reports with edited verdict banners, drift classes that used to slip through, invalid synthesized baselines — and, for the effect lane, a read-only-annotated tool that mints a credential detectable only out-of-band, a persistent object that must not be classified as authority-bearing, a no-probe audit whose authority verdicts must degrade to unknown/SKIP instead of passing, and a create+delete call that must not hide its delete behind the headline effect.

• CI on Linux, macOS and Windows × Python 3.11 / 3.12 / 3.13, plus a packaging job that builds the wheel, installs it fresh, and runs a real audit against a real server before anything ships — and an experiments job that re-runs E1–E3 from scratch and fails unless the outputs are byte-identical to the committed results: the reproducibility claim is CI-enforced, not asserted.

• Exercised against servers we did not build: three case-study audits of published third-party MCP servers (both official reference servers and a community one — three store types, with and without annotations), with committed evidence — see the effect lane section below.

• Cross-validated against the official v2 SDK in both directions: the official client adopts mcp-proof's hand-rolled modern test server via server/discover, and mcp-proof runs fully green against official v2 SDK servers on both transports (scripts/crosscheck_modern_server.py).

• Fail-closed by design: a broken pagination walk, a tampered or unverifiable fixture, a missing baseline or an internal auditor error each stop the audit loudly — and every command answers with the same taxonomy: exit 2 and one stable line, never a traceback, never a silently smaller audit, and never evidence against the target.

• Offline-verifiable reports: mcp-proof verify report.json recomputes both fingerprints from the report's own fields; the document fingerprint covers everything a reader sees — verdict banner, audit status, summary counters, the MSSS table, next steps — so any post-audit edit breaks it. It is an internal-consistency proof, not a signature (attestation is on the roadmap).

✨ What's under the hood

• 🔍 Wire-level protocol checks across every surface, every page, both eras — mcp-proof speaks raw JSON-RPC to your server and auto-detects its era: 32 checks for the 2026-07-28 modern era (server/discover, _meta envelope enforcement, resultType, ttlMs/cacheScope on every cacheable result, -32022 version rejection, HTTP routing-header enforcement) and 27 for the initialize-handshake era — exact error codes, schema validity, structured output, stdout hygiene, pagination safety on all three list surfaces, dedicated resources & prompts lanes, and verified negative probes: TOOL-07 sends inputs that provably violate the declared inputSchema (a schema-valid baseline with exactly one field mutated) and warns when the server answers them normally — and treats a hang as its own finding, never as rejection. One pagination collector feeds every lane, so a tool hidden on page 2 is audited exactly like a tool on page 1.

• 🛡️ Security audit tied to a public standard — 6 deterministic checks (tool-description poisoning, invisible/bidi characters, leaked credentials, unconstrained injection surfaces, advertised shell execution) over every advertised tool on every page, with a schema walker that sees through $ref/allOf/nesting/array items — config.shell.command cannot hide one level down. Each check maps to canonical control IDs of the MCP Server Security Standard's 24-entry control matrix (23 fully documented controls plus the MCP-DEPLOY-04 future-control placeholder), rendered as a compliance table whose verdicts never outrun their evidence: full direct proof says met, clean-but-indirect evidence says partial, and a control the checks cannot see says manual review.

• 🧪 Effect checks that read the world, not the response (v0.8) — with an observation channel configured (--sqlite for a SQLite-backed server, --fs-root for a directory-backed one), the effect lane snapshots external state before and after every call and diffs it into per-object create/update/delete effects, attributed to the call that caused them (per-target ops, so a move can't hide its delete behind its create). Four checks compare that against the declared annotations, following the spec's semantics exactly — defaults are pessimistic, so an absent hint is never flagged, only an explicit claim the observed effect falsifies: EFF-01 (a readOnlyHint: true tool caused no observed write), EFF-02 (no observed delete under an explicit destructiveHint: false), EFF-03 (an idempotentHint tool's repeated identical call was a no-op), EFF-06 (a created credential's continued effectiveness still depends on the grant that authorized it — established by revoking each candidate dependency out-of-band and re-exercising the object, then restoring it). Every field of every effect record carries how it was known — declared, observed, probed, or unknown — and a dimension with no channel SKIPs rather than passing.

• 📼 A regression suite your client keeps — and that verifies itself before it judges anyone — records in either protocol era; golden fixtures freeze the server's behaviour with SHA-256 provenance, including every content type (binary payloads as digests, so a swapped image can never replay as OK). Before replaying, an integrity gate recomputes every contract hash and the manifest fingerprint: a missing, tampered, duplicated or stale fixture aborts the replay instead of being silently skipped — deleting a fixture's stored hash counts as tampering, not as an older schema, and baselines that predate contract hashing are refused unless --allow-legacy-fixtures explicitly opts in. Replay grades every drift (BREAKING / VALUE / COSMETIC / LATENCY) — any structured or JSON value change is at least VALUE, a flipped "approved"→"denied" can never pass as cosmetic — and preserves stateful call order (sequence-numbered fixtures, order-sensitive fingerprint). A baseline is never created implicitly: run fails closed when fixtures are missing unless you opt in with --record-if-missing.

• 📄 A report for humans and machines — self-contained HTML with sticky navigation, per-check anchors (report.html#SEC-03), attention/passed filters, an evidence-scope card and a collapsible MSSS matrix; --pdf for print. The same versioned model ships as --json (schema v3), --junit for any CI, and --sarif for the GitHub Security tab. The effect lane renders its own evidence page: declared annotations beside the observed effect, the response a response-only auditor would have read, the objects that resulted, and the probe's authority/dependency verdict, each value tagged with how it was known.

• 🔁 Reproducible by design — zero LLM calls, zero API keys. Two fingerprints, honestly separated: behavior_sha256 is computed from server behaviour alone (check verdicts, replay verdicts, protocol facts — never timestamps, latency, the launch command or the auditor's version), so identical server behaviour fingerprints identically on any machine; run_hash freezes the whole report document — evidence, verdict banner, audit status, summaries, MSSS table — minus only the volatile timestamp block. mcp-proof verify rechecks both offline: an internal-consistency proof that any post-audit edit breaks, not a signature. Acceptance is verification, not trust.

• 🧯 Conservative call planning, and a trust correction (v0.8) — auto-baselining classifies tools by a conservative name/description heuristic, and mcp-proof plan shows exactly what would be called and on what basis before anything touches production. As of v0.8, MCP annotations may only add caution: destructiveHint: true still forces a skip, but an unverified readOnlyHint: true no longer rescues a mutating-looking tool into the auto-call set — the spec says clients MUST treat annotations as untrusted, and the effect lane exists precisely because a "read-only" tool can mint credentials. --include-destructive and --edge-cases opt into more.

• 📋 A contract diff for CI — mcp-proof inspect freezes the served surface (capabilities + tools + resources + prompts, fully paginated, absent-vs-empty recorded) into a fingerprinted manifest — and refuses to write one at all if any pagination walk cannot be completed, because half a surface frozen as "the baseline" makes every later diff against the missing half invisible. Volatile wire metadata is removed by location, never by key name, so a schema property that happens to be called ttlMs or nextCursor stays part of the contract. mcp-proof diff classifies every change as BREAKING / ADDITIVE / METADATA and exits non-zero on breaking ones — schema tightening, enum narrowing, required-flips, removed output fields and weakened safety annotations all count.

📊 Real audits, real reports

🧪 The effect-aware research lane (v0.8)

The three delivery lanes stop at the wire: their notion of behaviour is the response byte stream. The effect lane extends the same declared-vs-observed method one level deeper. Its parts, concretely:

• Observer (effects/observe.py): snapshots external state — a SQLite store read directly from the file, or a directory tree — before and after each tool call, and diffs the snapshots into per-object create/update/delete deltas. It never asks the tools what changed, so an effect is seen whether or not the response mentions it.

• Probe (effects/probes.py): attempts to use a created object as a credential against the service's real authorization rule. "Authority-bearing" is then an observed outcome (the object authorized an action), not a guess from a field name; "still effective" is the probe succeeding now, not the object still being listed.

• Lineage, kept as three separate fields: created_via (which call produced the object — observed), authorized_by (the grant the session ran under — declared), and depends_on (what its continued effectiveness actually requires — established by revoking each candidate out-of-band, re-exercising, and restoring). The distinction is the point: an API key authorized_by a grant whose depends_on does not include that grant survives the grant's revocation.

• Testbed (testbed/): a deterministic SQLite-backed MCP server with ordinary persistent objects (notes) and credential objects (API keys, webhooks, share links), a one-bit grant, lifecycle tools, and mutation flags that plant exactly one annotation lie each — modelled on documented incident patterns (a read path that mints authority; revocation that doesn't cascade). Ground truth is read out-of-band by testbed/saas_oracle.py, never through the audited MCP surface.

Three experiments run against it (python experiments/run_all.py — deterministic, and CI re-runs the pipeline to enforce byte-identical outputs). Each does a different job — E1 is the detection experiment, E2 is construct validation (the persistence/authority distinction is operationally real), E3 is a lifecycle measurement (existence ≠ current effectiveness). The numbers are detection behaviour on planted inconsistencies in this controlled setting — not production prevalence:

The two results that motivate the lane: in E1, the three lies whose effects never appear in the response (silent-keymint, shadow-webhook, phantom-write) are caught only by the state diff — a response-level auditor is structurally blind to them, which is why all-lane recall for the baselines is 0.333. In E3's grant_revoked scenario, the key created under the grant remains effective after the grant is revoked (the testbed's auth rule checks the key's own row, not the grant — the same shape as documented OAuth-app persistence incidents); the grant-status proxy reports it neutralized, which is the one dangerous false-ineffective in the table — authorized_by read as if it were depends_on.

And off the testbed: three case studies run the same instrument against published third-party servers — the official memory server (JSONL store, fully annotated), the official filesystem server (jailed directory, stock observer, fully annotated) and a community SQLite server (no annotations at all). Every declaration that could be checked held; where nothing was declared the checks SKIPped rather than inventing a verdict; and the filesystem run exposed a real blind spot in EFF-02's first implementation (a create+delete call whose headline effect hid the delete), now fixed and pinned by a test. The probe half stayed honestly unknown throughout — none of these services mints an exercisable credential — so probe-backed authority results remain testbed-validated. Evidence: case studies on the evaluation site.

Methodology, oracle design, baselines, related work and limitations: docs/effect-aware-conformance.md · results with raw evidence: evaluation site · reproduction: experiments/README.md.

🧭 How this relates to the official conformance suite

The MCP project maintains modelcontextprotocol/conformance — scenario tests that verify protocol behaviour for servers and clients, including auth flows. If you need a protocol-correctness baseline, run it; mcp-proof's conformance lane covers overlapping ground from its own wire-level probes.

mcp-proof exists for the half the official suite doesn't do: delivery evidence. A fingerprinted, offline-verifiable report a client can keep; MSSS security mapping; golden behavioural regression with a fail-closed integrity gate; contract snapshot/diff as a CI gate; SARIF/JUnit artifacts; and the effect-conformance research lane. Use the official suite to prove the protocol; use mcp-proof to prove the delivery — they compose, and cross-validating against the official suite is on the roadmap.

📡 Protocol support

Works with servers in any language — mcp-proof talks to the process (or URL), not to your codebase.

⚙️ CI in one step

- uses: YuCPbit/mcp-proof@v0.8.1
  with:
    server-command: python my_server.py
    fixtures: fixtures/

The job fails unless the server is ship-ready, and leaves mcp-proof-report.html / .json / .junit.xml / .sarif behind for upload. Prefer raw commands? mcp-proof run … --junit r.xml --sarif r.sarif plus mcp-proof diff is the same gate.

🏗️ Build on the audit-clean template

Building a server rather than auditing one? templates/server-starter/ is a fastmcp server that passes this audit out of the box — constrained input schemas, proper error semantics, structured output, every practice annotated with the check ID it satisfies. Copy, implement your tools, audit, ship with the report.

🖥️ Platforms

🗺️ Roadmap

Release history lives in CHANGELOG.md.

🔍 Limitations

mcp-proof proves what can be proven deterministically, and says which is which:

• Security checks cover the observable protocol and metadata surface. MSSS controls that need deployment, source or process evidence are always reported as manual review — never as passed.

• Authorization flows are out of scope for the delivery report: OAuth handshakes are not audited (the official conformance suite covers auth scenarios). The effect lane reasons about authority-bearing objects a tool creates, on a controlled testbed with an out-of-band observer — it does not audit a production OAuth deployment.

• The effect lane is a measurement instrument, not a black-box lane. It needs an observation channel (--sqlite, --fs-root); effects to systems it cannot observe are reported unknown/SKIP, never assumed absent. Its quantitative results are detection behaviour on a synthetic testbed — the case studies show the instrument working on third-party servers, but they validate the observation half only: probe-backed authority/effectiveness remains testbed-validated. See docs/effect-aware-conformance.md §8.

• Auto-baselining classifies tools by a conservative name/description heuristic; as of v0.8 an unverified readOnlyHint no longer overrides it. Review the skip list in the fixtures manifest before trusting a baseline recorded against production.

• Semantic correctness (does the answer mean the right thing?) is outside the deterministic core by design.

📄 License

MIT — the taxonomy in the MSSS compliance section follows the MCP Server Security Standard (CC BY-SA 4.0).