Skip to main content
Glama

CodeForgeX: Deterministic AI-Agent Evaluation Environment & MCP Software Engineering Harness

Python 3.10+ MCP SDK v2 Tests Passing Benchmark Score Docker Hardened License: MIT

CodeForgeX is an enterprise-grade, deterministic AI-agent software engineering evaluation environment and tool-calling execution harness. Built on the official Model Context Protocol (MCP Python SDK v2), it provides an isolated, uncheatable sandbox where AI agents explore repositories, reproduce failures, formulate hypotheses, apply unified diff patches, and verify solutions against public and hidden test suites.

Unlike subjective "LLM-as-a-judge" grading, CodeForgeX employs deterministic, multi-criteria verification with cryptographic test tampering detection, regression guards, and wall-clock execution limits.


Architecture Topology

flowchart TD
    subgraph ControlPlane["CLI & Orchestration (scripts/run_evaluation.py)"]
        CLI["CLI Runner"] --> Runner["EvaluationRunner"]
        CLI --> Loop["AgentExecutionLoop"]
    end

    subgraph AgentLayer["Autonomous Agent & Planning (src/agent/)"]
        Loop --> Planner["SystematicSWEPlanner / LLMPlanner"]
        Planner -->|AgentAction| Loop
        Loop -->|call_tool| Client["MCPClient (Async stdio)"]
    end

    subgraph ProtocolBoundary["Model Context Protocol Boundary"]
        Client <==>|Anonymous OS Pipes (stdio JSON-RPC)| Server["MCPServer (Subprocess)"]
    end

    subgraph ToolSurface["Sandboxed Tool Surface (src/mcp_server/)"]
        Server --> T1["list_files"]
        Server --> T2["read_file"]
        Server --> T3["search_code"]
        Server --> T4["apply_patch"]
        Server --> T5["get_git_diff"]
        Server --> T6["run_tests"]
        Server --> T7["get_test_output"]
        Server --> T8["get_repository_status"]
    end

    subgraph SecurityBoundary["Security & Anti-Cheat Sandbox (src/mcp_server/security/)"]
        T1 & T2 & T3 & T4 & T5 & T6 & T7 & T8 --> Sandbox["SecurityPolicy Engine"]
        Sandbox --> Guard1["Path Traversal Containment"]
        Sandbox --> Guard2["Anti-Cheat (Hidden Tests Isolated)"]
        Sandbox --> Guard3["Command Whitelisting & Injection Defense"]
    end

    subgraph TaskWorkspaces["Ephemeral Sandboxes (tasks/)"]
        Guard1 & Guard2 & Guard3 --> TargetRepo["Ephemeral Workspace (.git)"]
    end

    subgraph Evaluator["Deterministic Evaluator Engine (src/evaluator/)"]
        TargetRepo --> Verifier["TaskVerifier"]
        Verifier --> ShaCheck["SHA-256 Digest Tamper Check"]
        Verifier --> PublicRun["Public Test Execution"]
        Verifier --> HiddenRun["Privileged Hidden Test Suite"]
        Verifier --> DiffAnalysis["Git Diff & Line Metrics"]
        ShaCheck & PublicRun & HiddenRun & DiffAnalysis --> Scorer["ScoringEngine (100 Pt Model)"]
        Scorer --> Artifacts["Telemetry Artifacts (results/eval_*.json, .md)"]
    end

Related MCP server: Coding Tools MCP

Key System Capabilities

  • Official Model Context Protocol (SDK v2): Real-time tool discovery and execution over standard I/O (stdio), eliminating TCP port exhaustion and network race conditions.

  • Multi-Provider Schema Reflection: Dynamic tool schema conversion supporting OpenAI function calling, Anthropic Claude, and Google Gemini function declarations.

  • Multi-Dimensional Scoring (0 - 100 Points):

    • Task Completion (40 pts)

    • Hidden Verification Tests (25 pts)

    • Public Baseline Tests (15 pts)

    • Regression Safety (10 pts)

    • Tool Efficiency & Token Conservation (5 pts)

    • Patch Conciseness & Quality (5 pts)

  • Anti-Cheat & Anti-Tampering Protection: Dual-layer defense with cryptographic SHA-256 test file digest verification. Modifying or deleting test assertions forces an immediate 0.0 / 100.0 disqualification score.

  • Systematic SWE Reasoning Workflow: Enforces Test-Driven Software Engineering: EXPLORE $\rightarrow$ REPRODUCE $\rightarrow$ ANALYZE $\rightarrow$ PATCH $\rightarrow$ VERIFY $\rightarrow$ FINISH.

  • Fault-Tolerant Circuit Breakers: Active consecutive-failure guards prevent runaway token expenditure and model hallucination loops.

  • Hardened Docker Isolation: Non-root user execution (uid=1000), cap_drop: [ALL], no-new-privileges:true, and RAM-backed in-memory tmpfs mounts.

  • Unified Benchmark Dashboard: Terminal dashboard with JSON and Markdown artifact generation.


Benchmark Suite Catalog

CodeForgeX includes 6 diverse benchmark tasks spanning core software engineering modalities:

Task ID

Category

Difficulty

Problem Domain

Baseline Defect

Golden Score

bug_fix_001

bug_fix

Easy

E-Commerce Pricing Engine

Flat rate subtraction instead of % calculation; no bound checks

100.0 / 100.0

bug_fix_002

bug_fix

Medium

Concurrent LRU Cache with TTL

Expired nodes not purged on access; evicts MRU instead of LRU

98.0 / 100.0

feature_001

feature

Medium

Thread-Safe Token Bucket Limiter

Class raises NotImplementedError across all methods

98.0 / 100.0

refactor_001

refactor

Medium

Request Dispatcher to Strategy

Monolithic if/elif/else router; lacks BaseRequestHandler registry

98.0 / 100.0

perf_001

performance

Medium

Log Stream Deduplication ($O(N^2) \rightarrow O(N)$)

$O(N \times W)$ quadratic nested search takes > 1.5s and times out

100.0 / 100.0

algo_001

algorithm

Medium

Topological Build Dependency Sorter

Lacks topological sort and Tarjan/DFS cycle path detection

100.0 / 100.0


Quickstart Guide

1. Installation & Environment Setup

# Clone the repository
git clone https://github.com/kirubesh/CodeForgeX.git
cd CodeForgeX

# Install virtual environment and dependencies using uv or pip
pip install -e .

2. Run the Full Test Suite (91 Tests)

pytest -v

3. Run Benchmark Evaluations via CLI

# Run the entire benchmark suite in golden reference mode
python scripts/run_evaluation.py --all

# Run a specific task in autonomous agent mode (MCP stdio tool-calling loop)
python scripts/run_evaluation.py --task bug_fix_001 --mode agent-systematic

# Filter benchmarks by category or difficulty
python scripts/run_evaluation.py --category algorithm
python scripts/run_evaluation.py --difficulty medium

4. Containerized Execution with Docker

# Build the security-hardened container
docker build -t codeforge-x:latest .

# Run full evaluation across all tasks inside Docker
docker run --rm codeforge-x:latest --evaluate-all

# Or run using Docker Compose with cgroups limits and tmpfs in-memory sandboxes
docker compose up codeforge-eval

Technical Documentation & Interview Defenses


License

MIT License. See LICENSE for details.

Available Tools

8 tools
apply_patchA

Atomically apply a unified diff patch to repository files.

Args: patch: The unified diff content (e.g., standard 'diff --git' or '--- / +++' format). file_path: Optional relative target path if targeting a single file.

Returns: JSON string with success status, list of changed files, and status message.

ParametersJSON Schema
NameRequiredDescriptionDefault
patchYes
file_pathNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full disclosure burden. It discloses atomicity, accepted patch formats ('diff --git' or '--- / +++'), and the JSON return contract. It does not warn about destructive side effects or conflict behavior, but the core behavior is clearly and honestly stated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and well-structured: a one-sentence summary followed by two argument explanations and a returns line. There is no redundant phrasing, and the most important facts are front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter mutation tool with an output schema, the description covers the input format, optional path targeting, atomicity, and return structure. It is slightly thin on safety context such as whether changes are reversible or how conflicts are handled, but an agent has enough information to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema provides only bare type names, so schema description coverage is 0%. The description compensates by explaining the patch parameter as unified diff content with format examples and clarifying file_path as optional, relative, and for targeting a single file. This adds real meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description leads with a specific verb and object: 'Atomically apply a unified diff patch to repository files.' This clearly differentiates it from sibling tools like get_git_diff, which generate diffs, and read_file/search_code, which are read-only. The purpose is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied: an agent should call this when it has a unified diff to apply to repository files. However, there is no explicit when-to-use versus alternatives guidance, such as 'use get_git_diff to inspect changes instead,' so the agent must infer the decision from sibling names.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_git_diffA

Retrieve the deterministic Git diff of changes made in the repository.

Args: path: Optional relative file or directory path to scope the diff. cached: If True, inspects staged changes (git diff --cached) (default: False).

Returns: JSON string containing diff string, has_changes boolean, and changed_files list.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathNo
cachedNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the behavioral burden. It discloses staged-change behavior via cached, mentions determinism, and specifies the JSON return fields. However, it does not explicitly state the default scope as unstaged changes only or that untracked files are excluded.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the main purpose and uses compact Args/Returns sections. Every sentence adds meaningful information, with no filler or unnecessary repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with two optional parameters and no annotations, the description covers the operation, inputs, and return shape. It is slightly incomplete because it does not clarify the default diff scope or mention edge cases like no changes or invalid paths, and it does not guide selection among siblings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description fully documents both parameters beyond the schema. Path is explained as a relative file/directory scope, and cached is explained as switching to staged changes with a default. This fully compensates for the sparse schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description identifies a specific action ('Retrieve') and resource ('deterministic Git diff'), and clarifies optional path and cached scoping. It is clearly distinct from siblings like get_repository_status or list_files. The purpose is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use the tool through its name and purpose, but it does not explicitly state when to choose this over alternatives. There is no mention of get_repository_status or other sibling tools, and no exclusions are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_repository_statusA

Inspect the current Git repository status (modified, staged, untracked files).

Returns: JSON string with branch, clean flag, and lists of modified/staged/untracked files.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the behavioral transparency burden. It communicates a read-only intent through 'Inspect' and explicitly describes the returned JSON structure (branch, clean flag, file lists). It does not mention error behavior or explicitly state 'does not modify files,' but for a simple status-inspection tool this is adequate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is only two sentences, with the primary purpose front-loaded and the return format in a compact second sentence. There is no filler, repetition, or unnecessary detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a parameterless, read-only status tool, the description covers what it does and what it returns, and an output schema is present to formalize the response. Missing error-condition details are a minor gap given the tool's simplicity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so there is no parameter schema ambiguity to resolve; the baseline 4 applies. The description appropriately omits parameter details because none exist.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Inspect') and resource ('current Git repository status'), with a parenthetical listing modified/staged/untracked files. It clearly states what the tool does and the status focus implicitly distinguishes it from get_git_diff, though it does not explicitly name or contrast that sibling.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The usage context is implied: use this when you need the current repository status such as modified, staged, or untracked files. However, there is no explicit guidance on when to choose this over get_git_diff or other siblings, so the agent must infer the appropriate selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_outputA

Retrieve test execution output and logs from a previous test run.

Args: run_id: Specific test run identifier (empty string defaults to most recent run). full: If True, returns full untruncated stdout and stderr (default: False).

Returns: JSON string containing logs, exit code, duration, and pass/fail counts.

ParametersJSON Schema
NameRequiredDescriptionDefault
fullNo
run_idNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations available, the description provides useful behavioral detail beyond the schema: the 'full' flag controls truncation of stdout/stderr, and the return value is a JSON string with specific fields. The verb 'retrieve' also communicates that the operation is read-only, covering the main safety question. It stops short of describing error behavior for invalid run_id, but that is a minor gap for a retrieval tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a one-sentence summary followed by clearly labeled Args and Returns sections. Every sentence carries necessary information, with defaults and return details front-loaded and no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

This is a simple two-parameter tool with defaults for both parameters and an output schema already present, so the description does not need to explain return structure, yet it does. The Args section fully explains each parameter and the Returns section states the payload shape. No critical information is missing for an agent to select and invoke this tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has no descriptions (0% coverage), and the description fully compensates by explaining both parameters. It defines run_id's default behavior (empty string = most recent run) and full's effect on output truncation, adding substantial meaning beyond the bare types and defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific action ('Retrieve test execution output and logs') and a resource ('from a previous test run'). It implicitly distinguishes itself from sibling run_tests by focusing on a previous run rather than executing tests, but it never names the alternative or states that it does not trigger a run.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'from a previous test run' implies the tool is for inspecting completed runs, and the parameter descriptions clarify defaults, but there is no explicit 'when to use this vs. alternatives' guidance. No exclusions or alternative tools are mentioned, leaving usage somewhat implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_filesA

List files and directories in the repository.

Args: directory: Relative subfolder inside the repository (empty string for root). recursive: Whether to list recursively through child directories (default: True). max_depth: Maximum directory recursion depth (default: 10).

Returns: JSON string containing total_entries and a structured list of files with sizes.

ParametersJSON Schema
NameRequiredDescriptionDefault
directoryNo
max_depthNo
recursiveNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the behavioral disclosure burden. It clearly states the return type (JSON string), the top-level return fields (total_entries and a structured list), and that files include sizes. It also clarifies recursion defaults. It could mention error behavior or edge cases, but it is transparent about the core behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with a concise purpose statement and then organized into compact Args and Returns sections. Every sentence adds necessary information, and there is no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple three-parameter listing tool with an output schema available, the description is complete: it covers purpose, all parameters with defaults, and the return contract. Nothing critical is missing for an agent to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% description coverage, so the description must compensate, and it does. It explains that directory is a relative subfolder with empty string meaning root, that recursive controls traversal into child directories, and that max_depth limits recursion depth. This adds real meaning beyond the bare property names and defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear verb-resource pair: 'List files and directories in the repository.' This is specific and naturally distinguishes the tool from siblings like read_file (file content access) and search_code (querying code).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides solid invocation details: the directory argument is relative to the repository root, recursion defaults to true, and max_depth defaults to 10. However, it never explicitly states when to prefer this tool over alternatives or when not to use it, leaving the routing decision mostly implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_fileA

Read text content from a file inside the repository with line windowing.

Args: path: Relative path to the file inside the repository. start_line: 1-indexed starting line number (default: 1). end_line: 1-indexed ending line number (default: None, reads up to 1000 lines).

Returns: JSON string containing path, line numbers, content, and truncation status.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYes
end_lineNo
start_lineNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations available, the description carries the behavioral transparency burden and does a solid job: it states the default 1000-line cap, the 1-indexed line behavior, and the JSON return shape including truncation status. It does not mention error handling for missing files or invalid line ranges, but for a read operation this is a minor gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and well-structured, with a front-loaded purpose sentence followed by a clear Args section and a Returns section. No sentence is wasted, and every line earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's low complexity, no annotations, and the presence of an output schema, the description covers everything an agent needs: purpose, parameter semantics, defaults, line limit, and return contents. It is complete enough for reliable selection and invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description fully compensates by explaining each parameter: path is a relative repository path, start_line is 1-indexed with default 1, and end_line is 1-indexed with default None and a 1000-line limit. This adds real meaning beyond the bare schema types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb and resource: 'Read text content from a file inside the repository with line windowing.' It is unambiguous and distinct from siblings like list_files or search_code, though it does not explicitly name or contrast those alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended use is implied by the tool's name and first sentence, and the line-windowing parameters clarify how to read a specific range of lines. However, the description does not explicitly say when to prefer this over search_code, list_files, or get_git_diff, nor does it mention any exclusion conditions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_testsA

Execute automated pytest tests within the repository sandbox.

Args: test_target: Optional relative test file or test node (e.g. 'tests/test_math.py::test_add'). timeout_seconds: Maximum time allowed before terminating process (default: 30s).

Returns: JSON string with exit code, passed/failed counts, duration, and output summary.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_targetNo
timeout_secondsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description bears the full burden of explaining behavior. It discloses timeout termination and return structure, which is useful, but it does not mention potential side effects of running tests, resource usage, or whether the sandbox fully isolates the process.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact, front-loaded with the core purpose, and uses a clear Args/Returns structure. Every sentence contributes useful information without redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with two optional parameters, the description covers behavior, parameters, and return format. The main gap is the lack of guidance about when to use this instead of get_test_output, but the presence of an output schema and clear parameter docs make it mostly complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must explain the parameters itself. It fully does: test_target is described as a relative file or node with a concrete example, and timeout_seconds includes its default and meaning. This adds substantial meaning beyond the raw schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Execute'), a well-defined resource ('automated pytest tests'), and a context ('repository sandbox'). This clearly distinguishes it from siblings like get_test_output, which retrieves results rather than running tests.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description tells what the tool does but gives no explicit guidance on when to choose it over alternatives such as get_test_output or apply_patch. There is no mention of when not to use it, prerequisites, or how it relates to sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_codeA

Search for literal text or regular expressions across repository files.

Args: query: The substring or regex to search for. path: Optional relative directory or file to constrain search scope. is_regex: Whether query is a regular expression (default: False). case_sensitive: Whether match should be case-sensitive (default: False). max_results: Maximum matching lines to return (default: 50).

Returns: JSON string with matched lines, line numbers, and file paths.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathNo
queryYes
is_regexNo
max_resultsNo
case_sensitiveNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden. It discloses matching modes, defaults, result limits, and return format, which is solid, but it does not mention regex flavor, whether the search respects ignore files, or behavior on binary files. These are non-trivial but not critical gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with a one-sentence purpose, followed by a terse Args list and a clear Returns line. Every sentence earns its place; some redundancy with schema defaults exists but is minor and supports quick scanning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a five-parameter tool with no annotations, the description covers the operation, parameters, and return shape well, and an output schema is present. It is complete enough for correct invocation, though a note on regex flavor and path resolution would make it fully self-contained.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must and does compensate by explaining each parameter's purpose and default value in the Args section. It adds meaning beyond bare schema titles, though it leaves small nuances like path semantics and regex syntax details implicit.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Search') and resource ('repository files') and clearly distinguishes from siblings by emphasizing content search with literal or regex matching. It also explains the tool's core distinction from read_file or list_files without needing to name them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description makes clear when to use the tool: when you need to find matching lines across repository files, optionally constrained by a path. It omits explicit 'when not to use' guidance, but the context is unambiguous enough that an agent can select it correctly.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 8 tool updatesv0.1.0
    • First observedapply_patch
    • First observedget_git_diff
    • First observedget_repository_status
    • First observedget_test_output
    • First observedlist_files
    • First observedread_file
    • First observedrun_tests
    • First observedsearch_code

TDQS

A4.2/5.0

Scored across 8 tools

Disambiguation5/5

Each tool targets a distinct concern: file listing, status inspection, content reading, search, test execution, test output retrieval, patching, and diff viewing. The only related pair is run_tests/get_test_output, but their roles are clearly sequential rather than interchangeable.

Naming Consistency4/5

Most tools follow a clear verb_noun pattern like list_files, read_file, search_code, run_tests, and apply_patch. The get_* variants are slightly different in style but still predictable and consistent with common Git-related naming.

Tool Count5/5

Eight tools is a well-scoped size for a repository coding assistant. Each tool covers a necessary operation without redundancy or bloat, and the set is small enough for an agent to navigate easily.

Completeness4/5

The toolset covers the core inspect-modify-test loop well: reading, searching, patching, diffing, and running tests. Missing commit/branch management tools are a minor gap for full repository lifecycle coverage, but agents can still complete most coding workflows.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • F
    license
    Not graded
    quality
    D
    maintenance
    Enables AI agents to write and execute Python code in an isolated sandbox that can orchestrate multiple MCP tool calls, reducing context window bloat and improving efficiency for complex workflows.
    23
    -
  • A
    license
    Not graded
    quality
    A
    maintenance
    Empower any MCP-compatible AI Agent(MCP Client) with engineering-grade capabilities to understand, modify, run, and deliver real-world code repositories.
    993
    Apache 2.0
  • A
    license
    A
    quality
    C
    maintenance
    Runs AI-generated code in secure Firecracker microVMs with opt-in network policy enforcement, PII scanning, prompt injection defense, and audit logging. Exposes MCP tools for running commands, managing files, and the full sandbox lifecycle.
    7
    52
    1
    Apache 2.0
  • A
    license
    Not graded
    quality
    B
    maintenance
    Gives any MCP-compatible AI chat or agent a safe, model-neutral coding runtime with file read/search, structured multi-file patches, command execution, interactive sessions, and git operations, all confined to a single workspace and gated by permission modes.
    Apache 2.0