Self-Correcting Coding Agent

An autonomous coding agent that takes a task ("fix this bug," "make this failing test pass"), plans it, rewrites the affected file, runs the real test suite in an isolated Docker sandbox, and — if it fails — reads the actual error and retries. Exposed as an MCP server so external AI clients (like Claude Desktop) can call it as a tool directly.

Results

Benchmarked against a 4-task suite covering distinct bug types (off-by-one error, wrong comparison operator, missing return statement, incorrect operator):

Full run logs: eval/results/v2_fullfile.json

Note: this is a small, controlled benchmark meant to prove the architecture works end-to-end, not a claim of real-world SWE-bench-level performance. Next step is testing against a larger, messier benchmark.

Related MCP server: deepseek-subagent-mcp

Architecture

                 ┌─────────────┐
   task  ──────▶ │   Planner   │  breaks task into steps
                 └──────┬──────┘
                        ▼
                 ┌─────────────┐
                 │    Coder    │  rewrites the full corrected file
                 └──────┬──────┘  (not a diff — see "Design decisions")
                        ▼
                 ┌─────────────┐
                 │  Reviewer   │  approves or rejects the change
                 └──────┬──────┘  before it's ever written to disk
                        ▼
                 ┌──────────────┐
                 │   Sandbox    │  Docker, no network, memory-capped
                 │ (real tests) │  runs the actual test suite
                 └──────┬───────┘
                        │ pass ──▶ done
                        │ fail ──▶ real error fed back to Coder, retry
                        ▼
                  (up to N iterations)

All of the above is also exposed as MCP tools (run_task, run_tests), so any MCP-compatible client — Claude Desktop, Claude Code, etc. — can drive the agent directly instead of a human running a script.

Design decisions worth knowing

Full-file rewrite instead of diff-based patching. The first version had the Coder emit a unified diff, applied via git apply. In practice, the model's diff output was inconsistent enough (missing +++ headers, malformed context lines, Windows CRLF mismatches) that diff application became the most fragile part of the system. Switching the Coder to return the complete corrected file — with a diff computed afterward via difflib, purely for logging and review — eliminated an entire class of failures and is the main reason the eval success rate went from inconsistent to 100%.

Three separated LLM roles, not one prompt. A single prompt tends to rubber-stamp its own output. Splitting Planner / Coder / Reviewer into distinct calls with narrow responsibilities makes the Reviewer step actually catch bad changes, and makes failures debuggable — you can tell exactly which stage broke.

Network-isolated sandbox with a pre-baked image. The sandbox runs with network_disabled=True and a memory cap, since it's executing LLM-generated code. Rather than allowing runtime pip install (which would require network access, defeating the isolation), a custom Docker image (sandbox.Dockerfile) has test dependencies pre-installed.

Tech stack

• Python — core agent logic

• Google Gemini API (google-genai) — Planner / Coder / Reviewer, behind a swappable LLMClient abstraction (originally built against Claude's API)

• Docker — isolated, network-disabled sandbox execution

• MCP (Model Context Protocol) — exposes the agent as tools for external AI clients

• pytest — target test framework for the benchmark tasks

Project structure

coding-agent/
├── src/
│   ├── agent/
│   │   ├── loop.py          # orchestration: planner → coder → sandbox → reviewer
│   │   ├── planner.py       # breaks a task into steps
│   │   ├── coder.py         # generates the full corrected file
│   │   ├── reviewer.py      # approves/rejects a change before it's written
│   │   ├── llm_client.py    # Gemini API wrapper with retry/backoff
│   │   └── prompts.py       # system prompts per role
│   ├── sandbox/
│   │   └── docker_runner.py # isolated, network-disabled test execution
│   └── mcp_server/
│       └── server.py        # exposes run_task / run_tests as MCP tools
├── eval/
│   ├── harness.py           # runs the agent across eval/tasks/*.json, scores results
│   ├── tasks/                # benchmark task definitions
│   └── results/              # eval run logs (success rate, iterations, tokens)
├── setup_eval_tasks.py       # (re)creates all benchmark task repos from scratch
├── sandbox.Dockerfile        # pre-baked sandbox image with test deps installed
└── requirements.txt

Setup

python -m venv venv
source venv/bin/activate       # Windows: venv\Scripts\Activate.ps1
pip install -r requirements.txt
cp .env.example .env            # add your GEMINI_API_KEY

docker build -t coding-agent-sandbox:latest -f sandbox.Dockerfile .

python setup_eval_tasks.py
python eval/harness.py --run-name my_run

Connect it to Claude Desktop (MCP)

Add to your Claude Desktop config (Settings → Developer → Edit config):

{
  "mcpServers": {
    "coding-agent": {
      "command": "/absolute/path/to/coding-agent/venv/Scripts/python.exe",
      "args": ["/absolute/path/to/coding-agent/src/mcp_server/server.py"]
    }
  }
}

Restart Claude Desktop, then ask it to use the coding-agent tool with a task_description, repo_path, test_command, and relevant_files.

What's not finished

• Benchmark is currently small (4 tasks) and synthetic — real-world bugs are messier and often span multiple files

• Single-file-edit heuristic (pick_target_file) doesn't yet support multi-file changes

• No cost-per-run tracking beyond raw token counts

• MCP server currently supports two tools (run_task, run_tests); more (e.g. get_diff, list_tasks) would make it more useful as a general-purpose tool for other agents

License

MIT