mcp-server-and-agent

A hand-written MCP server at the JSON-RPC level, four agent topologies over it, and the failure rate of each measured under a controlled fault model. The agent brain is a deterministic simulation, not a live LLM — a scripted policy executes plans over the real MCP server while a parameterised fault process perturbs its choices at stated rates. That is the instrument, not a compromise: a topology's failure rate measured against a live model is confounded by that model's nondeterminism, and you cannot tell whether a difference came from the topology or from the sampler. Everything below is a failure rate of a topology under a fault model, never of any real model.

This repo answers one question:

What is each agent topology's failure rate, and does supervisor actually beat single-agent?

The answer

Supervisor beats single-agent, by less than it costs, and it is dominated by two simpler designs.

600 runs per topology (6 tasks x 100 trials), every topology facing the identical fault stream for a given (task, trial) — a paired comparison, not four independent samples.

• Supervisor's 3.5-point gain is real, not noise: paired bootstrap delta -0.035 [-0.062, -0.008] at 95%, interval excludes zero.

• It costs 1.73x the tokens. pipeline and reflexive both beat it on failure rate and on tokens. Paying 1.7x for the third-best outcome is the finding.

• The ranking reflexive < pipeline < supervisor < single holds across a 0.5x–2.0x sweep of the fault rates.

• The gaps narrow as faults rise. Topology is a second-order effect; tool reliability is the first-order one.

Full table, confidence intervals and sensitivity sweep: results/topologies.md. Every number is generated by scripts/generate_results.py, which asserts its own claims and exits non-zero if they break.

Related MCP server: agentloop

The more useful answer

Under the blended fault model above the topologies look similar. Turn one fault up at a time and they are radically different — and which topology helps depends entirely on which fault you have:

Two of six are fixed by topology, both by the same property — isolation of resources per unit of work, not supervision as such. The other four are fixed in the tool layer. Note that supervisor and pipeline are worse than single on error cascades: a fresh worker context discards the error history that would have told it not to repeat the call. Isolation contains cascades and also amputates learning.

The judgement artifact, with reproduction seeds, real traces and a mitigation per mode: docs/failure-taxonomy.md.

Scope

Deliberately narrow. In scope: the MCP protocol surface, four topologies, six failure modes, and the token cost of each. Out of scope: anything that does not help answer the question above.

Quickstart

uv sync --extra dev
uv run pytest -q                          # 90 tests
uv run python scripts/generate_results.py # regenerates results/
uv run python scripts/find_failure_seeds.py

Run the MCP server against any client speaking stdio:

uv run python -m mcp_server_and_agent.server

What is in here

The MCP server is written against the spec, not on an SDK

initialize / tools/list / tools/call / resources/read / prompts/get / ping, with correct -32700 / -32600 / -32601 / -32602 / -32603 error objects. The reason is not purity: an SDK hides exactly the seams this repo measures. Protocol conformance is one of the few things in an agent stack that is exactly testable — a malformed request has one correct error code — so tests/test_protocol_conformance.py covers the cases a happy-path implementation gets wrong: notifications getting no reply, "id": null being a request rather than a notification, batch rejection, and the boundary below.

A tool that does not exist is -32602. A tool that exists and fails is a successful response carrying isError. The first is a bug in the client; the second is feedback the agent can act on. Collapsing them means the agent either retries unfixable calls forever or gives up on recoverable ones.

Four topologies

• single — one agent, the whole tool list, the whole plan. Baseline.

• supervisor — a supervisor dispatches each step to a fresh worker. Each dispatch pays a handoff tax because the worker starts cold.

• pipeline — a fixed discover → fetch → aggregate chain. No routing decision to get wrong; no re-planner when a stage fails. Context threads forward.

• reflexive — single agent plus exactly one reflection-and-retry pass. Chosen as the fourth because it isolates what supervisor confounds: whether a second attempt is worth more than a second agent. Both cost extra tokens; under this fault model only one adds a capability.

Provenance

Every number in this README comes from a committed script.

• Date: 2026-08-25

• Hardware: 24-core CPU, 32 GB RAM, no GPU

• Model: simulated-scripted-policy — no LLM involved

• Seed: 20260825

• Reproduce: python scripts/generate_results.py

• Raw artifact: results/topologies-raw.md (gitignored — per-task detail)

• Committed artifact: results/topologies.md

CI regenerates results/ and fails on git diff --exit-code, so a hand-edited number breaks the build. That gate only means something because the experiment is deterministic — same seed, same failure rates — which tests/test_topologies.py asserts in both directions.

Limitations

• No LLM was involved in any measurement. The agent brain is a scripted policy and the faults are drawn from a distribution I chose. These are failure rates of topologies under a controlled fault model, not of any real model in any real deployment. What transfers is the shape of the result — which mitigation attacks which mechanism — not the values.

• The fault rates are uncalibrated. Nothing here establishes that a real agent misselects a tool 16% of the time. Calibrating them requires the live runs this repo deliberately does not do, and until someone does that, the absolute rates are arbitrary and only the comparisons are meaningful.

• Token counts are synthetic: a fixed charge per step plus len(text)//4 for observations. Consistent across topologies so the ratios hold; the dollar figures forecast nobody's bill.

• Six tasks, one server, one task shape. All six are search-then-fetch-then-aggregate. The case a supervisor is supposed to win — genuinely parallel, separable subtasks — is not represented here, and these numbers are not evidence against it. This is the single biggest thing the repo does not establish.

• reflexive gets one retry the others do not. Part of its advantage is a second draw from the fault distribution rather than reflection as such.

• No LangGraph. The brief named it; the agent loop here is hand-written in ~200 lines because the failure modes under study are properties of the loop, and a framework would have made the step cap, context accounting and cascade detector someone else's implementation details.

• Detection is measured; recovery mostly is not. Apart from the rollback path, this establishes that failures are caught, not that a system built on these mitigations completes more tasks.

Built on

• llm-client-kit v0.1.0 — CostLedger for token and spend accounting.

• llm-eval-harness v0.1.0 — stats.paired_delta_ci and is_reportable for the confidence intervals, types.RunMeta for the provenance block.

License

MIT — see LICENSE.