MCP Failure Lab
README.md
# MCP Failure Lab
[](https://scorecard.dev/viewer/?uri=github.com/cloudlabworks/mcp-failure-lab)
[](https://www.bestpractices.dev/projects/16205)
[](https://doi.org/10.5281/zenodo.23720942)
A deliberately faulty MCP server and a bounded reference-client harness.
v0.1.0, MIT. Built on the official MCP TypeScript SDK **2.3.1**, not a replacement protocol implementation.
## Try it
Requires Docker. The image uses Node 24 and pinned npm dependencies.
```sh
docker build -t mcp-failure-lab:local .
docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges mcp-failure-lab:local npm test
docker run --rm --network none --read-only --cap-drop ALL --security-opt no-new-privileges mcp-failure-lab:local node runner.mjs
```
The last command prints JSON results and OpenAI-message-shaped traces from seven deterministic runs.
No language model participates. Timings are observed elapsed milliseconds, not benchmarks.
The test command exits nonzero on a regression; the demo command emits observations, not a grading verdict.
For local development: `npm ci --ignore-scripts`, then `npm test`.
From the published image (Official MCP Registry: `io.github.cloudlabworks/mcp-failure-lab`):
```sh
docker run -i --rm -e LAB_SCENARIO=tool-error ghcr.io/cloudlabworks/mcp-failure-lab:0.1.2
```
### Verifying a release
From v0.1.2, each release has build provenance signed through Sigstore by the `publish-mcp` workflow. With the GitHub CLI:
```sh
gh attestation verify oci://ghcr.io/cloudlabworks/mcp-failure-lab:0.1.2 --repo cloudlabworks/mcp-failure-lab
gh attestation verify mcp-failure-lab-0.1.2.tar.gz --repo cloudlabworks/mcp-failure-lab
```
The second command checks the source tarball from the GitHub Release. To verify offline, add `--bundle mcp-failure-lab-0.1.2.tar.gz.intoto.jsonl` (the provenance file attached to the same release).
## Faults
| LAB_SCENARIO | Injected behavior | Reference-client expectation |
| --- | --- | --- |
| healthy | Normal tool response | One success |
| tool-error | Valid result with isError=true | Stop at configured attempt budget |
| transient-error | First LAB_FAILURES calls fail | Recover if the attempt budget permits |
| delay | Wait LAB_DELAY_MS before tool response | Succeed within deadline or time out |
| malformed | Valid JSON-RPC envelope, invalid text-block value | Reject the tool result |
| disconnect | Exit during a tool response | Report closed connection, not success |
Only the advertised `lab_probe` tool is affected. Initialization and discovery use the official SDK.
The transport wrapper corrupts **result shape**, not JSON framing; truncated frames, bad IDs, HTTP failures and authentication faults are not implemented.
Configuration is read at process startup. Scenarios reset per process.
`LAB_DELAY_MS`: integer 0–10000 (default 1000).
`LAB_FAILURES`: integer 0–10 (default 2).
Values must be plain decimal digits; an unset or empty variable uses the default, and anything else (`0x10`, `1e3`, ` 7`) stops the server with exit code 2.
The reference client retries only explicit tool errors, at most five attempts; protocol errors, timeouts and disconnects stop immediately.
The harness's tool-call deadline is 10–10000 ms; SDK connection/discovery deadlines are separate.
Retries intentionally have no backoff so the generated fixture exposes repeated calls; this is not recommended production retry policy.
## Attach your own MCP client
Configure a **test** MCP connection to run this command, with stdin/stdout attached:
```sh
docker run --rm -i --network none --read-only --cap-drop ALL --security-opt no-new-privileges -e LAB_SCENARIO=transient-error -e LAB_FAILURES=2 mcp-failure-lab:local
```
Call `lab_probe` with `{"message":"probe"}`.
There are no real downstream services, credentials, file-writing tools or listening network ports.
Client integrations beyond the included reference client are **unverified**.
## Feed a failure into Agent Trace Clinic
From this repository, with Clinic checked out next to it:
```sh
mkdir -p artifacts
docker run --rm --network none --read-only mcp-failure-lab:local node runner.mjs > artifacts/run.json
node --input-type=module -e 'import fs from "node:fs"; const r=JSON.parse(fs.readFileSync("artifacts/run.json","utf8")); console.log(JSON.stringify(r.results.find(x=>x.scenario==="tool-error").trace));' > artifacts/tool-error.json
node ../agent-trace-clinic/cli.mjs artifacts/tool-error.json --html > artifacts/clinic-report.html
```
Clinic should identify tool_loop, retry_storm and unhandled_tool_error on this fixture.
These are heuristic leads: deliberate bounded retries can also trigger them. No token counts are invented.
## Relationship to upstream
[MCP conformance](https://github.com/modelcontextprotocol/conformance) is the upstream authority for protocol tests.
This project contributes a small fault surface for recovery experiments; it does **not** claim conformance certification.
The official framework currently documents HTTP-based client/server scenario runners. This first release is stdio-only;
an upstream scenario adapter and a Streamable HTTP target are next, not shipped capabilities.
Sources checked 2026-10-14:
- [SDK v2 documentation](https://ts.sdk.modelcontextprotocol.io/v2/)
- [SDK source and examples](https://github.com/modelcontextprotocol/typescript-sdk)
- [Conformance framework](https://github.com/modelcontextprotocol/conformance)
## Maintenance contract
One transport, one tool, six fault modes, one reference client. Pin SDK upgrades and rerun every fault test.
Owner: Cloud Lab Works / Waku. Gherkin review has not been assigned or completed.
Before public release: source/dependency review, an independent test pass, and publication approval.
Before cloud execution: bounded job budget, teardown, and measured runtime. No cloud account is required for this release.
No telemetry or remote reporting. Generated reports stay in ignored `artifacts/`.
## Agent Reliability Lab
[Open the public demo](https://huggingface.co/spaces/cloudlabworks/agent-reliability-lab) or read the [experiment guide](lab/README.md). Four controlled before/after cases connect the official MCP reference client to deterministic Clinic inspection. The public sample is recorded local evidence, not a live cloud execution. [Cloud adapters and runbook](deploy/README.md) are included.
## Citation
If you use this project, please cite the accompanying preprint:
[MCP Failure Lab: Reproducible Tool-Failure Fixtures and Cross-Platform Traces for Testing Agent Reliability](https://doi.org/10.5281/zenodo.23720942). DOI: [10.5281/zenodo.23720942](https://doi.org/10.5281/zenodo.23720942).
See the [paper page](https://cloudlabworks.dev/papers/mcp-failure-lab) for the abstract and PDF.
```bibtex
@techreport{alvord2026mcp,
author = {Alvord, Alex},
title = {{MCP Failure Lab: Reproducible Tool-Failure Fixtures and Cross-Platform Traces for Testing Agent Reliability}},
institution = {CloudLab Works},
publisher = {Zenodo},
year = {2026},
month = oct,
type = {Preprint},
doi = {10.5281/zenodo.23720942},
url = {https://doi.org/10.5281/zenodo.23720942}
}
```
TDQS
C2.5/5.0
Scored across 1 tool
Disambiguation5/5
There is only one tool, so there is no possibility of confusion or overlap. The single tool's purpose is clearly stated as a synthetic failure-lab probe.
Naming Consistency5/5
With a single tool name (lab_probe), there is no inconsistency to assess. The snake_case naming is clear and readable.
Tool Count1/5
A single trivial probe tool is an extreme mismatch for an MCP server, even one framed as a failure lab. There is no room for multiple probes, configuration, or lifecycle operations.
Completeness2/5
The apparent domain is a synthetic failure lab, which typically requires multiple probes, failure-mode selection, or reset capabilities. A single generic probe leaves significant gaps for any realistic lab use.
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues