Skip to main content
Glama

MCP Agent Evaluation Harness

A compact reference project for evaluating coding agents that use Model Context Protocol tools. The harness gives an agent controlled repository access, runs repeatable checks, compares repeated executions, and returns a structured verification report.

What the project demonstrates

  • An MCP server with repository listing, reading, searching, and verification tools

  • Root-confined file access with traversal protection

  • Deterministic command execution with fixed locale, timezone, and Python hash seed

  • Required and forbidden code-pattern checks

  • Repeat-run comparison for nondeterminism detection

  • Golden reference material for a reproducible bug-fixing task

  • Standard-library unit tests and a continuous integration workflow

Related MCP server: mcp-software-engineering-challenge

Architecture

MCP client or coding agent
          |
          v
Repository MCP tools
          |
          v
SafeWorkspace boundary
          |
          v
DeterministicVerifier
          |
          v
Structured VerificationReport

The core verifier has no third-party dependency. The MCP adapter uses the official Python SDK v2.

Quick start

Run the unit tests.

python -m unittest discover -s tests -v

Run the included evaluation task.

PYTHONPATH=src python -m mcp_eval_harness.cli verify \
  examples/rounding-bug/task.json \
  examples/rounding-bug/workspace

Install the MCP adapter and start the stdio server.

python -m pip install -e ".[mcp]"
mcp-eval serve

MCP tools

Tool

Purpose

list_repository_files

Return a stable, sorted file list under the configured root

read_repository_file

Read one UTF-8 file while blocking path traversal

search_repository

Find literal text with deterministic ordering and result limits

verify_solution

Run the task specification and return a structured report

Task specification

Each task is JSON and declares the files, code requirements, verification command, timeout, and repeat count.

{
  "task_id": "decimal-rounding-fix",
  "required_files": ["calculator.py", "test_calculator.py"],
  "required_patterns": {"calculator.py": ["Decimal", "ROUND_HALF_UP"]},
  "forbidden_patterns": {"calculator.py": ["round("]},
  "command": ["python", "-m", "unittest", "discover", "-s", ".", "-p", "test_*.py"],
  "timeout_seconds": 10,
  "repeat_count": 2
}

Security boundary

The path checks protect the configured repository root. The command verifier is designed for trusted local task definitions. Run untrusted candidate code inside a container or another operating-system sandbox.

Author

Mohammed Ibrahim Sadiq

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers