CodeForgeX
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@CodeForgeXReproduce the failing test, fix the bug, and verify all tests pass"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
CodeForgeX: Deterministic AI-Agent Evaluation Environment & MCP Software Engineering Harness
CodeForgeX is an enterprise-grade, deterministic AI-agent software engineering evaluation environment and tool-calling execution harness. Built on the official Model Context Protocol (MCP Python SDK v2), it provides an isolated, uncheatable sandbox where AI agents explore repositories, reproduce failures, formulate hypotheses, apply unified diff patches, and verify solutions against public and hidden test suites.
Unlike subjective "LLM-as-a-judge" grading, CodeForgeX employs deterministic, multi-criteria verification with cryptographic test tampering detection, regression guards, and wall-clock execution limits.
Architecture Topology
flowchart TD
subgraph ControlPlane["CLI & Orchestration (scripts/run_evaluation.py)"]
CLI["CLI Runner"] --> Runner["EvaluationRunner"]
CLI --> Loop["AgentExecutionLoop"]
end
subgraph AgentLayer["Autonomous Agent & Planning (src/agent/)"]
Loop --> Planner["SystematicSWEPlanner / LLMPlanner"]
Planner -->|AgentAction| Loop
Loop -->|call_tool| Client["MCPClient (Async stdio)"]
end
subgraph ProtocolBoundary["Model Context Protocol Boundary"]
Client <==>|Anonymous OS Pipes (stdio JSON-RPC)| Server["MCPServer (Subprocess)"]
end
subgraph ToolSurface["Sandboxed Tool Surface (src/mcp_server/)"]
Server --> T1["list_files"]
Server --> T2["read_file"]
Server --> T3["search_code"]
Server --> T4["apply_patch"]
Server --> T5["get_git_diff"]
Server --> T6["run_tests"]
Server --> T7["get_test_output"]
Server --> T8["get_repository_status"]
end
subgraph SecurityBoundary["Security & Anti-Cheat Sandbox (src/mcp_server/security/)"]
T1 & T2 & T3 & T4 & T5 & T6 & T7 & T8 --> Sandbox["SecurityPolicy Engine"]
Sandbox --> Guard1["Path Traversal Containment"]
Sandbox --> Guard2["Anti-Cheat (Hidden Tests Isolated)"]
Sandbox --> Guard3["Command Whitelisting & Injection Defense"]
end
subgraph TaskWorkspaces["Ephemeral Sandboxes (tasks/)"]
Guard1 & Guard2 & Guard3 --> TargetRepo["Ephemeral Workspace (.git)"]
end
subgraph Evaluator["Deterministic Evaluator Engine (src/evaluator/)"]
TargetRepo --> Verifier["TaskVerifier"]
Verifier --> ShaCheck["SHA-256 Digest Tamper Check"]
Verifier --> PublicRun["Public Test Execution"]
Verifier --> HiddenRun["Privileged Hidden Test Suite"]
Verifier --> DiffAnalysis["Git Diff & Line Metrics"]
ShaCheck & PublicRun & HiddenRun & DiffAnalysis --> Scorer["ScoringEngine (100 Pt Model)"]
Scorer --> Artifacts["Telemetry Artifacts (results/eval_*.json, .md)"]
endRelated MCP server: Coding Tools MCP
Key System Capabilities
Official Model Context Protocol (SDK v2): Real-time tool discovery and execution over standard I/O (
stdio), eliminating TCP port exhaustion and network race conditions.Multi-Provider Schema Reflection: Dynamic tool schema conversion supporting OpenAI function calling, Anthropic Claude, and Google Gemini function declarations.
Multi-Dimensional Scoring (0 - 100 Points):
Task Completion (40 pts)
Hidden Verification Tests (25 pts)
Public Baseline Tests (15 pts)
Regression Safety (10 pts)
Tool Efficiency & Token Conservation (5 pts)
Patch Conciseness & Quality (5 pts)
Anti-Cheat & Anti-Tampering Protection: Dual-layer defense with cryptographic SHA-256 test file digest verification. Modifying or deleting test assertions forces an immediate
0.0 / 100.0disqualification score.Systematic SWE Reasoning Workflow: Enforces Test-Driven Software Engineering:
EXPLORE$\rightarrow$REPRODUCE$\rightarrow$ANALYZE$\rightarrow$PATCH$\rightarrow$VERIFY$\rightarrow$FINISH.Fault-Tolerant Circuit Breakers: Active consecutive-failure guards prevent runaway token expenditure and model hallucination loops.
Hardened Docker Isolation: Non-root user execution (
uid=1000),cap_drop: [ALL],no-new-privileges:true, and RAM-backed in-memorytmpfsmounts.Unified Benchmark Dashboard: Terminal dashboard with JSON and Markdown artifact generation.
Benchmark Suite Catalog
CodeForgeX includes 6 diverse benchmark tasks spanning core software engineering modalities:
Task ID | Category | Difficulty | Problem Domain | Baseline Defect | Golden Score |
|
| Easy | E-Commerce Pricing Engine | Flat rate subtraction instead of % calculation; no bound checks |
|
|
| Medium | Concurrent LRU Cache with TTL | Expired nodes not purged on access; evicts MRU instead of LRU |
|
|
| Medium | Thread-Safe Token Bucket Limiter | Class raises |
|
|
| Medium | Request Dispatcher to Strategy | Monolithic |
|
|
| Medium | Log Stream Deduplication ($O(N^2) \rightarrow O(N)$) | $O(N \times W)$ quadratic nested search takes > 1.5s and times out |
|
|
| Medium | Topological Build Dependency Sorter | Lacks topological sort and Tarjan/DFS cycle path detection |
|
Quickstart Guide
1. Installation & Environment Setup
# Clone the repository
git clone https://github.com/kirubesh/CodeForgeX.git
cd CodeForgeX
# Install virtual environment and dependencies using uv or pip
pip install -e .2. Run the Full Test Suite (91 Tests)
pytest -v3. Run Benchmark Evaluations via CLI
# Run the entire benchmark suite in golden reference mode
python scripts/run_evaluation.py --all
# Run a specific task in autonomous agent mode (MCP stdio tool-calling loop)
python scripts/run_evaluation.py --task bug_fix_001 --mode agent-systematic
# Filter benchmarks by category or difficulty
python scripts/run_evaluation.py --category algorithm
python scripts/run_evaluation.py --difficulty medium4. Containerized Execution with Docker
# Build the security-hardened container
docker build -t codeforge-x:latest .
# Run full evaluation across all tasks inside Docker
docker run --rm codeforge-x:latest --evaluate-all
# Or run using Docker Compose with cgroups limits and tmpfs in-memory sandboxes
docker compose up codeforge-evalTechnical Documentation & Interview Defenses
System Architecture Specification: Deep-dive into MCP server tools, security sandboxing, scoring math, and agent loop lifecycle.
Security & Threat Modeling: Comprehensive threat matrix covering path traversal, shell injection, privilege elevation, and anti-cheat guards.
Docker Reproducibility Guide: Container security specs, capability dropping, and cgroup resource quotas.
Staff-Level Technical Interview Defense Guide: Complete question-and-answer handbook defending every architectural trade-off and systems engineering choice.
License
MIT License. See LICENSE for details.
Available Tools
8 toolsapply_patchA
Atomically apply a unified diff patch to repository files.
Args: patch: The unified diff content (e.g., standard 'diff --git' or '--- / +++' format). file_path: Optional relative target path if targeting a single file.
Returns: JSON string with success status, list of changed files, and status message.
| Name | Required | Description | Default |
|---|---|---|---|
| patch | Yes | ||
| file_path | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full disclosure burden. It discloses atomicity, accepted patch formats ('diff --git' or '--- / +++'), and the JSON return contract. It does not warn about destructive side effects or conflict behavior, but the core behavior is clearly and honestly stated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured: a one-sentence summary followed by two argument explanations and a returns line. There is no redundant phrasing, and the most important facts are front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter mutation tool with an output schema, the description covers the input format, optional path targeting, atomicity, and return structure. It is slightly thin on safety context such as whether changes are reversible or how conflicts are handled, but an agent has enough information to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides only bare type names, so schema description coverage is 0%. The description compensates by explaining the patch parameter as unified diff content with format examples and clarifying file_path as optional, relative, and for targeting a single file. This adds real meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description leads with a specific verb and object: 'Atomically apply a unified diff patch to repository files.' This clearly differentiates it from sibling tools like get_git_diff, which generate diffs, and read_file/search_code, which are read-only. The purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied: an agent should call this when it has a unified diff to apply to repository files. However, there is no explicit when-to-use versus alternatives guidance, such as 'use get_git_diff to inspect changes instead,' so the agent must infer the decision from sibling names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_git_diffA
Retrieve the deterministic Git diff of changes made in the repository.
Args: path: Optional relative file or directory path to scope the diff. cached: If True, inspects staged changes (git diff --cached) (default: False).
Returns: JSON string containing diff string, has_changes boolean, and changed_files list.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | ||
| cached | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the behavioral burden. It discloses staged-change behavior via cached, mentions determinism, and specifies the JSON return fields. However, it does not explicitly state the default scope as unstaged changes only or that untracked files are excluded.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the main purpose and uses compact Args/Returns sections. Every sentence adds meaningful information, with no filler or unnecessary repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with two optional parameters and no annotations, the description covers the operation, inputs, and return shape. It is slightly incomplete because it does not clarify the default diff scope or mention edge cases like no changes or invalid paths, and it does not guide selection among siblings.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description fully documents both parameters beyond the schema. Path is explained as a relative file/directory scope, and cached is explained as switching to staged changes with a default. This fully compensates for the sparse schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description identifies a specific action ('Retrieve') and resource ('deterministic Git diff'), and clarifies optional path and cached scoping. It is clearly distinct from siblings like get_repository_status or list_files. The purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool through its name and purpose, but it does not explicitly state when to choose this over alternatives. There is no mention of get_repository_status or other sibling tools, and no exclusions are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_repository_statusA
Inspect the current Git repository status (modified, staged, untracked files).
Returns: JSON string with branch, clean flag, and lists of modified/staged/untracked files.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral transparency burden. It communicates a read-only intent through 'Inspect' and explicitly describes the returned JSON structure (branch, clean flag, file lists). It does not mention error behavior or explicitly state 'does not modify files,' but for a simple status-inspection tool this is adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is only two sentences, with the primary purpose front-loaded and the return format in a compact second sentence. There is no filler, repetition, or unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a parameterless, read-only status tool, the description covers what it does and what it returns, and an output schema is present to formalize the response. Missing error-condition details are a minor gap given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there is no parameter schema ambiguity to resolve; the baseline 4 applies. The description appropriately omits parameter details because none exist.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Inspect') and resource ('current Git repository status'), with a parenthetical listing modified/staged/untracked files. It clearly states what the tool does and the status focus implicitly distinguishes it from get_git_diff, though it does not explicitly name or contrast that sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The usage context is implied: use this when you need the current repository status such as modified, staged, or untracked files. However, there is no explicit guidance on when to choose this over get_git_diff or other siblings, so the agent must infer the appropriate selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_test_outputA
Retrieve test execution output and logs from a previous test run.
Args: run_id: Specific test run identifier (empty string defaults to most recent run). full: If True, returns full untruncated stdout and stderr (default: False).
Returns: JSON string containing logs, exit code, duration, and pass/fail counts.
| Name | Required | Description | Default |
|---|---|---|---|
| full | No | ||
| run_id | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations available, the description provides useful behavioral detail beyond the schema: the 'full' flag controls truncation of stdout/stderr, and the return value is a JSON string with specific fields. The verb 'retrieve' also communicates that the operation is read-only, covering the main safety question. It stops short of describing error behavior for invalid run_id, but that is a minor gap for a retrieval tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a one-sentence summary followed by clearly labeled Args and Returns sections. Every sentence carries necessary information, with defaults and return details front-loaded and no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
This is a simple two-parameter tool with defaults for both parameters and an output schema already present, so the description does not need to explain return structure, yet it does. The Args section fully explains each parameter and the Returns section states the payload shape. No critical information is missing for an agent to select and invoke this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has no descriptions (0% coverage), and the description fully compensates by explaining both parameters. It defines run_id's default behavior (empty string = most recent run) and full's effect on output truncation, adding substantial meaning beyond the bare types and defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific action ('Retrieve test execution output and logs') and a resource ('from a previous test run'). It implicitly distinguishes itself from sibling run_tests by focusing on a previous run rather than executing tests, but it never names the alternative or states that it does not trigger a run.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'from a previous test run' implies the tool is for inspecting completed runs, and the parameter descriptions clarify defaults, but there is no explicit 'when to use this vs. alternatives' guidance. No exclusions or alternative tools are mentioned, leaving usage somewhat implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_filesA
List files and directories in the repository.
Args: directory: Relative subfolder inside the repository (empty string for root). recursive: Whether to list recursively through child directories (default: True). max_depth: Maximum directory recursion depth (default: 10).
Returns: JSON string containing total_entries and a structured list of files with sizes.
| Name | Required | Description | Default |
|---|---|---|---|
| directory | No | ||
| max_depth | No | ||
| recursive | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral disclosure burden. It clearly states the return type (JSON string), the top-level return fields (total_entries and a structured list), and that files include sizes. It also clarifies recursion defaults. It could mention error behavior or edge cases, but it is transparent about the core behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with a concise purpose statement and then organized into compact Args and Returns sections. Every sentence adds necessary information, and there is no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple three-parameter listing tool with an output schema available, the description is complete: it covers purpose, all parameters with defaults, and the return contract. Nothing critical is missing for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% description coverage, so the description must compensate, and it does. It explains that directory is a relative subfolder with empty string meaning root, that recursive controls traversal into child directories, and that max_depth limits recursion depth. This adds real meaning beyond the bare property names and defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear verb-resource pair: 'List files and directories in the repository.' This is specific and naturally distinguishes the tool from siblings like read_file (file content access) and search_code (querying code).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides solid invocation details: the directory argument is relative to the repository root, recursion defaults to true, and max_depth defaults to 10. However, it never explicitly states when to prefer this tool over alternatives or when not to use it, leaving the routing decision mostly implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
read_fileA
Read text content from a file inside the repository with line windowing.
Args: path: Relative path to the file inside the repository. start_line: 1-indexed starting line number (default: 1). end_line: 1-indexed ending line number (default: None, reads up to 1000 lines).
Returns: JSON string containing path, line numbers, content, and truncation status.
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | ||
| end_line | No | ||
| start_line | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations available, the description carries the behavioral transparency burden and does a solid job: it states the default 1000-line cap, the 1-indexed line behavior, and the JSON return shape including truncation status. It does not mention error handling for missing files or invalid line ranges, but for a read operation this is a minor gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured, with a front-loaded purpose sentence followed by a clear Args section and a Returns section. No sentence is wasted, and every line earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity, no annotations, and the presence of an output schema, the description covers everything an agent needs: purpose, parameter semantics, defaults, line limit, and return contents. It is complete enough for reliable selection and invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description fully compensates by explaining each parameter: path is a relative repository path, start_line is 1-indexed with default 1, and end_line is 1-indexed with default None and a 1000-line limit. This adds real meaning beyond the bare schema types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb and resource: 'Read text content from a file inside the repository with line windowing.' It is unambiguous and distinct from siblings like list_files or search_code, though it does not explicitly name or contrast those alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use is implied by the tool's name and first sentence, and the line-windowing parameters clarify how to read a specific range of lines. However, the description does not explicitly say when to prefer this over search_code, list_files, or get_git_diff, nor does it mention any exclusion conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_testsA
Execute automated pytest tests within the repository sandbox.
Args: test_target: Optional relative test file or test node (e.g. 'tests/test_math.py::test_add'). timeout_seconds: Maximum time allowed before terminating process (default: 30s).
Returns: JSON string with exit code, passed/failed counts, duration, and output summary.
| Name | Required | Description | Default |
|---|---|---|---|
| test_target | No | ||
| timeout_seconds | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description bears the full burden of explaining behavior. It discloses timeout termination and return structure, which is useful, but it does not mention potential side effects of running tests, resource usage, or whether the sandbox fully isolates the process.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact, front-loaded with the core purpose, and uses a clear Args/Returns structure. Every sentence contributes useful information without redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with two optional parameters, the description covers behavior, parameters, and return format. The main gap is the lack of guidance about when to use this instead of get_test_output, but the presence of an output schema and clear parameter docs make it mostly complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must explain the parameters itself. It fully does: test_target is described as a relative file or node with a concrete example, and timeout_seconds includes its default and meaning. This adds substantial meaning beyond the raw schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Execute'), a well-defined resource ('automated pytest tests'), and a context ('repository sandbox'). This clearly distinguishes it from siblings like get_test_output, which retrieves results rather than running tests.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description tells what the tool does but gives no explicit guidance on when to choose it over alternatives such as get_test_output or apply_patch. There is no mention of when not to use it, prerequisites, or how it relates to sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_codeA
Search for literal text or regular expressions across repository files.
Args: query: The substring or regex to search for. path: Optional relative directory or file to constrain search scope. is_regex: Whether query is a regular expression (default: False). case_sensitive: Whether match should be case-sensitive (default: False). max_results: Maximum matching lines to return (default: 50).
Returns: JSON string with matched lines, line numbers, and file paths.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | ||
| query | Yes | ||
| is_regex | No | ||
| max_results | No | ||
| case_sensitive | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral burden. It discloses matching modes, defaults, result limits, and return format, which is solid, but it does not mention regex flavor, whether the search respects ignore files, or behavior on binary files. These are non-trivial but not critical gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with a one-sentence purpose, followed by a terse Args list and a clear Returns line. Every sentence earns its place; some redundancy with schema defaults exists but is minor and supports quick scanning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a five-parameter tool with no annotations, the description covers the operation, parameters, and return shape well, and an output schema is present. It is complete enough for correct invocation, though a note on regex flavor and path resolution would make it fully self-contained.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must and does compensate by explaining each parameter's purpose and default value in the Args section. It adds meaning beyond bare schema titles, though it leaves small nuances like path semantics and regex syntax details implicit.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Search') and resource ('repository files') and clearly distinguishes from siblings by emphasizing content search with literal or regex matching. It also explains the tool's core distinction from read_file or list_files without needing to name them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description makes clear when to use the tool: when you need to find matching lines across repository files, optionally constrained by a path. It omits explicit 'when not to use' guidance, but the context is unambiguous enough that an agent can select it correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
8 tool updates
v0.1.0- First observed
apply_patch - First observed
get_git_diff - First observed
get_repository_status - First observed
get_test_output - First observed
list_files - First observed
read_file - First observed
run_tests - First observed
search_code
TDQS
Scored across 8 tools
Each tool targets a distinct concern: file listing, status inspection, content reading, search, test execution, test output retrieval, patching, and diff viewing. The only related pair is run_tests/get_test_output, but their roles are clearly sequential rather than interchangeable.
Most tools follow a clear verb_noun pattern like list_files, read_file, search_code, run_tests, and apply_patch. The get_* variants are slightly different in style but still predictable and consistent with common Git-related naming.
Eight tools is a well-scoped size for a repository coding assistant. Each tool covers a necessary operation without redundancy or bloat, and the set is small enough for an agent to navigate easily.
The toolset covers the core inspect-modify-test loop well: reading, searching, patching, diffing, and running tests. Missing commit/branch management tools are a minor gap for full repository lifecycle coverage, but agents can still complete most coding workflows.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
AI-native git hosting — repos, PRs, issues, CI gates, and AI code review over MCP (60 tools).
MCP server for building and testing AI agents with multi-model experimentation and insights.
Governed app access for AI agents: 1,000+ apps & 12,000+ tools via Code Mode MCP.
One MCP tool for verified AI-agent outcomes with success-only charging.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceEnables AI agents to write and execute Python code in an isolated sandbox that can orchestrate multiple MCP tool calls, reducing context window bloat and improving efficiency for complex workflows.23-
- AlicenseNot gradedqualityAmaintenanceEmpower any MCP-compatible AI Agent(MCP Client) with engineering-grade capabilities to understand, modify, run, and deliver real-world code repositories.993Apache 2.0

declaw-mcp-serverofficial
AlicenseAqualityCmaintenanceRuns AI-generated code in secure Firecracker microVMs with opt-in network policy enforcement, PII scanning, prompt injection defense, and audit logging. Exposes MCP tools for running commands, managing files, and the full sandbox lifecycle.7521Apache 2.0- AlicenseNot gradedqualityBmaintenanceGives any MCP-compatible AI chat or agent a safe, model-neutral coding runtime with file read/search, structured multi-file patches, command execution, interactive sessions, and git operations, all confined to a single workspace and gated by permission modes.Apache 2.0