Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@multi-source-mcp-benchmark run the ledger-drift scenario and grade my answer"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A reproducible benchmark for evaluating AI agents on multi-source evidence investigation, not on retrieving a single fact, but on reconciling several disagreeing sources, ruling out a plausible wrong explanation, and producing a conclusion backed by specific evidence.
I built this after noticing how easy it is to get a false signal from an agent eval: give it one clean data source and a well-posed question, and almost any capable model looks good, because there's nothing to reconcile. The scenarios here are built the other way around. Each one starts from a real decision someone has to make (which number goes in the close packet, whether a performance claim survives scrutiny, what actually caused an outage) and gives the agent exactly the evidence a person would have, no more, including the parts that disagree with each other.
Three scenarios, each a full, runnable environment: a Docker image, a set of MCP tools the agent can use to investigate, a written brief, a verified reference answer, and an independent set of scoring claims.
Scenario | Domain | What the agent has to untangle |
Marketplace payments | A payout ledger and a payment processor's settlement export disagree on what four sellers should be paid, for four unrelated reasons: a duplicate webhook delivery, a stray unmatched record, a payout that settles one day past the batch cutoff, and a fee-tier change that only got applied to some of the affected payments. | |
Distributed systems performance | A team wants to ship a new caching strategy fleet-wide based on a memo with two impressive numbers. Both numbers come from load test runs that were never actually comparable to the baseline. | |
Production incident response | A circuit breaker gets reconfigured right before a traffic surge, and the resulting outage looks, on the surface, exactly like the customers' own systems being down. The evidence, including a real git history of the config change, says otherwise. |
Every discrepancy in every scenario is something the agent has to notice on its own; nothing in the evidence states a conclusion. See docs/design-notes.md for the rules I held myself to while building these, and docs/architecture.md for how the harness itself is put together.
Related MCP server: MEVA Health AI MCP Server
pip install -e ".[dev]"
cd scenarios/ledger-drift/environment
docker compose up -d --build
# list what the filesystem MCP server actually exposes
python ../../../tools/mcp_probe.py list-tools --config ../mcp-servers.json --server filesystem
# check a claims file and a candidate answer are well-formed, no API key required
python ../../../tools/grade.py --claims ../evaluation/claims.yaml --answer /path/to/answer.txt --dry-runPoint any MCP-capable agent at a scenario's mcp-servers.json and hand it brief.md. Grade its answer for real by dropping --dry-run and setting MSMB_API_BASE / MSMB_API_KEY / MSMB_MODEL.
pytestThe suite validates every scenario's structure, config, and evidence, and runs grade.py --dry-run against each one's reference answer, the same checks CI runs on every push.
MIT, see LICENSE.
Create a new directory or ensure a directory exists. Can create multiple nested directories in one operation. If the directory already exists, this operation will succeed silently. Perfect for setting up directory structures for projects or ensuring required paths exist. Only works within allowed directories.
Inputs: path (required). Returns: content
Get a recursive tree view of files and directories as a JSON structure. Each entry includes 'name', 'type' (file/directory), and 'children' for directories. Files have no children array, while directories always have a children array (which may be empty). The output is formatted with 2-space indentation for readability. Only works within allowed directories.
Inputs: path (required), excludePatterns. Returns: content
Make line-based edits to a text file. Each edit replaces exact line sequences with new content. Returns a git-style diff showing the changes made. Only works within allowed directories.
Inputs: path (required), edits (required), dryRun. Returns: content
Retrieve detailed metadata about a file or directory. Returns comprehensive information including size, creation time, last modified time, permissions, and type. This tool is perfect for understanding file characteristics without reading the actual content. Only works within allowed directories.
Inputs: path (required). Returns: content
Returns the list of directories that this server is allowed to access. Subdirectories within these allowed directories are also accessible. Use this to understand which directories and their nested paths are available before trying to access files.
Inputs: no named parameters. Returns: content
Get a detailed listing of all files and directories in a specified path. Results clearly distinguish between files and directories with [FILE] and [DIR] prefixes. This tool is essential for understanding directory structure and finding specific files within a directory. Only works within allowed directories.
Inputs: path (required). Returns: content
Get a detailed listing of all files and directories in a specified path, including sizes. Results clearly distinguish between files and directories with [FILE] and [DIR] prefixes. This tool is useful for understanding directory structure and finding specific files within a directory. Only works within allowed directories.
Inputs: path (required), sortBy. Returns: content
Move or rename files and directories. Can move files between directories and rename them in a single operation. If the destination exists, the operation will fail. Works across different directories and can be used for simple renaming within the same directory. Both source and destination must be within allowed directories.
Inputs: source (required), destination (required). Returns: content
Read the complete contents of a file as text. DEPRECATED: Use read_text_file instead.
Inputs: path (required), head, tail. Returns: content
Read a file and return it as a base64-encoded content block with its MIME type. Image and audio files are returned as image/audio content; any other file type is returned as an embedded resource. Only works within allowed directories.
Inputs: path (required). Returns: content
Read the contents of multiple files simultaneously. This is more efficient than reading files one by one when you need to analyze or compare multiple files. Each file's content is returned with its path as a reference. Failed reads for individual files won't stop the entire operation. Only works within allowed directories.
Inputs: paths (required). Returns: content
Read the complete contents of a file from the file system as text. Handles various text encodings and provides detailed error messages if the file cannot be read. Use this tool when you need to examine the contents of a single file. Use the 'head' parameter to read only the first N lines of a file, or the 'tail' parameter to read only the last N lines of a file. Operates on the file as text regardless of extension. Only works within allowed directories.
Inputs: path (required), head, tail. Returns: content
Recursively search for files and directories matching a pattern. The patterns should be glob-style patterns that match paths relative to the working directory. Use pattern like '.ext' to match files in current directory, and '**/.ext' to match files in all subdirectories. Returns full paths to all matching items. Great for finding files when you don't know their exact location. Only searches within allowed directories.
Inputs: path (required), pattern (required), excludePatterns. Returns: content
Create a new file or completely overwrite an existing file with new content. Use with caution as it will overwrite existing files without warning. Handles text content with proper encoding. Only works within allowed directories.
Inputs: path (required), content (required). Returns: content
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 14 tools
Most tools target distinct operations (read vs write vs move vs search vs metadata). However, read_file is explicitly deprecated in favor of read_text_file, creating a confusing overlap, and list_directory vs list_directory_with_sizes differ only by an optional detail.
The set is largely consistent verb_noun (read_text_file, write_file, move_file, search_files, get_file_info). Minor deviation with directory_tree, which is noun-only, and the legacy read_file naming.
14 tools is well-scoped for a filesystem server, with each tool earning its place across read, write, navigate, and metadata operations.
Core read/write/edit/move/search/metadata are covered, but there is no delete or remove operation, a notable gap for full filesystem lifecycle coverage. No copy operation either, though move partially covers that.
Tamper-evident proof creation and verification for AI agents via MCP, A2A, and REST.
MCP server for building and testing AI agents with multi-model experimentation and insights.
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
Independent AI-agent reviews: trust checks, evidence scorecards, incident registry, recommendations.
