A
licenseC
qualityC
maintenanceEnables acceptance gates for AI coding-agent runs by recording evidence, running deterministic validation, applying a quality gate, and rendering auditable outcomes.
7
Apache 2.0
mcp-eval
MCP Agent Evaluation Harness
一个精简的参考项目，用于评估使用 Model Context Protocol 工具的编码智能体。该评测框架为智能体提供受控的仓库访问，运行可重复执行的检查，对比多次执行的结果，并返回结构化的验证报告。
项目演示内容
一个提供仓库列表、读取、搜索和验证工具的 MCP 服务器
限制在根目录内的文件访问，并具备路径穿越防护
在固定 locale、时区和 Python 哈希种子下执行确定性命令
必需与禁止的代码模式检查
重复运行对比，用于检测不确定性
为可复现的 bug 修复任务提供黄金参考材料
标准库单元测试和持续集成工作流
Related MCP server: phionyx-pipeline-mcp
架构
MCP client or coding agent
|
v
Repository MCP tools
|
v
SafeWorkspace boundary
|
v
DeterministicVerifier
|
v
Structured VerificationReport核心验证器没有任何第三方依赖。MCP 适配器使用官方 Python SDK v2。
快速开始
运行单元测试。
python -m unittest discover -s tests -v运行内置的评估任务。
PYTHONPATH=src python -m mcp_eval_harness.cli verify \
examples/rounding-bug/task.json \
examples/rounding-bug/workspace安装 MCP 适配器并启动 stdio 服务器。
python -m pip install -e ".[mcp]"
mcp-eval serveMCP 工具
工具 | 用途 |
| 在配置的根目录下返回稳定且有序的文件列表 |
| 读取单个 UTF-8 文件并阻止路径穿越 |
| 按确定性顺序和固定结果数量查找字面文本 |
| 运行任务规范并返回结构化验证报告 |
任务规范
每个任务都是一个 JSON，声明了文件、代码要求、验证命令、超时时间和重复次数。
{
"task_id": "decimal-rounding-fix",
"required_files": ["calculator.py", "test_calculator.py"],
"required_patterns": {"calculator.py": ["Decimal", "ROUND_HALF_UP"]},
"forbidden_patterns": {"calculator.py": ["round("]},
"command": ["python", "-m", "unittest", "discover", "-s", ".", "-p", "test_*.py"],
"timeout_seconds": 10,
"repeat_count": 2
}安全边界
路径检查保护配置的仓库根目录。命令验证器是为受信任的本地任务定义而设计的。请在容器或其他操作系统沙箱中运行不可信的候选代码。
作者
Mohammed Ibrahim Sadiq
许可证
MIT
This server cannot be installed
Maintenance
–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Servers
- AlicenseAqualityAmaintenanceEnables verification of AI coding agent self-reports against git diff truth and a deterministic gate, producing pass/regenerate/reject directives to ensure claimed work matches actual changes.6AGPL 3.0
- FlicenseNot gradedqualityDmaintenanceProvides AI coding agents with dependency analysis, impact detection, and build verification tools.
- AlicenseNot gradedqualityAmaintenanceEnables AI agents to scan code for security and quality issues and receive machine-readable reports with suggested fixes and verification criteria.892MIT
Related MCP Connectors
Deterministic AI code review, with an audit record. Governance inside the agent loop.
Cross-agent artifact workspace with provenance across Claude Code, Codex, Cursor, LangGraph.
Evaluate, benchmark, and simulate AI agents on the VerifyAX agent-evaluation platform.
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/Mohammedibrahim111/mcp-agent-evaluation-harness'
If you have feedback or need assistance with the MCP directory API, please join our Discord server