mcp-eval
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
MCP Agent Evaluation Harness
A compact reference project for evaluating coding agents that use Model Context Protocol tools. The harness gives an agent controlled repository access, runs repeatable checks, compares repeated executions, and returns a structured verification report.
What the project demonstrates
An MCP server with repository listing, reading, searching, and verification tools
Root-confined file access with traversal protection
Deterministic command execution with fixed locale, timezone, and Python hash seed
Required and forbidden code-pattern checks
Repeat-run comparison for nondeterminism detection
Golden reference material for a reproducible bug-fixing task
Standard-library unit tests and a continuous integration workflow
Related MCP server: mcp-software-engineering-challenge
Architecture
MCP client or coding agent
|
v
Repository MCP tools
|
v
SafeWorkspace boundary
|
v
DeterministicVerifier
|
v
Structured VerificationReportThe core verifier has no third-party dependency. The MCP adapter uses the official Python SDK v2.
Quick start
Run the unit tests.
python -m unittest discover -s tests -vRun the included evaluation task.
PYTHONPATH=src python -m mcp_eval_harness.cli verify \
examples/rounding-bug/task.json \
examples/rounding-bug/workspaceInstall the MCP adapter and start the stdio server.
python -m pip install -e ".[mcp]"
mcp-eval serveMCP tools
Tool | Purpose |
| Return a stable, sorted file list under the configured root |
| Read one UTF-8 file while blocking path traversal |
| Find literal text with deterministic ordering and result limits |
| Run the task specification and return a structured report |
Task specification
Each task is JSON and declares the files, code requirements, verification command, timeout, and repeat count.
{
"task_id": "decimal-rounding-fix",
"required_files": ["calculator.py", "test_calculator.py"],
"required_patterns": {"calculator.py": ["Decimal", "ROUND_HALF_UP"]},
"forbidden_patterns": {"calculator.py": ["round("]},
"command": ["python", "-m", "unittest", "discover", "-s", ".", "-p", "test_*.py"],
"timeout_seconds": 10,
"repeat_count": 2
}Security boundary
The path checks protect the configured repository root. The command verifier is designed for trusted local task definitions. Run untrusted candidate code inside a container or another operating-system sandbox.
Author
Mohammed Ibrahim Sadiq
License
MIT
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
Paid deterministic data-quality and execution-verification tools for AI agents.
Repository evidence for agents before they adopt dependencies, enter codebases, compare, or merge.
Code intelligence platform for AI agents. 20 tools for architecture, security & impact analysis.
QA platform for agents: coverage signals, in-repo test plans, verified tests and release governance.
Related MCP Servers
- FlicenseNot gradedqualityBmaintenanceEnables reproducible evaluation of AI coding agents by exposing repository inspection, code editing, test running, and deterministic verification through MCP tools.-
- FlicenseNot gradedqualityBmaintenanceEnables AI agents to safely inspect, edit, and test code within a bounded repository environment to solve software engineering tasks and verify fixes.-
- FlicenseNot gradedqualityCmaintenanceEnables deterministic, sandboxed evaluation and RL training of software engineering agents via MCP, providing repository and terminal tools, failure classification, and live dashboards.-
- AlicenseNot gradedqualityBmaintenanceProvides bounded MCP tools for AI agents to inspect, write, verify, and diff files in a sandboxed repository, requiring human approval and logging an audit trail.MIT