Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@multi-source-mcp-benchmark run the ledger-drift scenario and grade my answer"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A reproducible benchmark for evaluating AI agents on multi-source evidence investigation, not on retrieving a single fact, but on reconciling several disagreeing sources, ruling out a plausible wrong explanation, and producing a conclusion backed by specific evidence.
I built this after noticing how easy it is to get a false signal from an agent eval: give it one clean data source and a well-posed question, and almost any capable model looks good, because there's nothing to reconcile. The scenarios here are built the other way around. Each one starts from a real decision someone has to make (which number goes in the close packet, whether a performance claim survives scrutiny, what actually caused an outage) and gives the agent exactly the evidence a person would have, no more, including the parts that disagree with each other.
Three scenarios, each a full, runnable environment: a Docker image, a set of MCP tools the agent can use to investigate, a written brief, a verified reference answer, and an independent set of scoring claims.
Scenario | Domain | What the agent has to untangle |
Marketplace payments | A payout ledger and a payment processor's settlement export disagree on what four sellers should be paid, for four unrelated reasons: a duplicate webhook delivery, a stray unmatched record, a payout that settles one day past the batch cutoff, and a fee-tier change that only got applied to some of the affected payments. | |
Distributed systems performance | A team wants to ship a new caching strategy fleet-wide based on a memo with two impressive numbers. Both numbers come from load test runs that were never actually comparable to the baseline. | |
Production incident response | A circuit breaker gets reconfigured right before a traffic surge, and the resulting outage looks, on the surface, exactly like the customers' own systems being down. The evidence, including a real git history of the config change, says otherwise. |
Every discrepancy in every scenario is something the agent has to notice on its own; nothing in the evidence states a conclusion. See docs/design-notes.md for the rules I held myself to while building these, and docs/architecture.md for how the harness itself is put together.
Related MCP server: MEVA Health AI MCP Server
pip install -e ".[dev]"
cd scenarios/ledger-drift/environment
docker compose up -d --build
# list what the filesystem MCP server actually exposes
python ../../../tools/mcp_probe.py list-tools --config ../mcp-servers.json --server filesystem
# check a claims file and a candidate answer are well-formed, no API key required
python ../../../tools/grade.py --claims ../evaluation/claims.yaml --answer /path/to/answer.txt --dry-runPoint any MCP-capable agent at a scenario's mcp-servers.json and hand it brief.md. Grade its answer for real by dropping --dry-run and setting MSMB_API_BASE / MSMB_API_KEY / MSMB_MODEL.
pytestThe suite validates every scenario's structure, config, and evidence, and runs grade.py --dry-run against each one's reference answer, the same checks CI runs on every push.
MIT, see LICENSE.
This server cannot be deployed
Reproducible benchmarks and reliability evidence for agent tools.
Tamper-evident proof creation and verification for AI agents via MCP, A2A, and REST.
MCP server for building and testing AI agents with multi-model experimentation and insights.
One MCP tool for verified AI-agent outcomes with success-only charging.