PatchLoop

Why

Coding agents are usually built on a framework, so the loop is a black box: how the agent recovers from errors, when it stops, what it costs and what it can touch are hidden. PatchLoop is the whole thing, small enough to read:

1. the agent loop is about 200 readable lines with no agent framework: budgets, retries, loop detection, context compaction and a trajectory log are all visible code;

2. model-written commands never touch the host: every tool call runs in a disposable container with no network, read-only tests and resource limits, and the boundary is tested;

3. every claim is measured on a benchmark built by injecting validated bugs into two real libraries, with confidence intervals and a paired significance test.

Everything runs locally for free: no API keys and no paid services.

Related MCP server: opencode-sandbox

Results

72 episodes (36 tasks × 2 toolsets), local qwen3:8b via Ollama with thinking off, on an Apple M5 laptop, 2026-10-23/24. 95% Wilson intervals in brackets. Full report: docs/EVAL_REPORT.md · annotated failures: docs/EVAL_ANALYSIS.md · raw data: results/.

What it shows

• The interface matters for a small model. On the same 36 tasks, 11 were solved only with the structured toolset and 2 only with bash (exact McNemar test, p = 0.022).

• Finding the bug isn't the bottleneck; committing to a fix is. Every episode reached the buggy file. With str_replace the model edited it in 20/36 episodes; with raw bash, in 10/36. Most failures end with an empty patch.

• Context resets hurt. Episodes that needed compaction resolved 4/22 versus 8/14 without (E1). One trajectory shows the model re-reading the same file after each reset. That's correlation, not proof.

• Thinking mode is impractical on a laptop GPU: with qwen3's thinking on, 0/5 tasks resolved, and every episode ran out of its 15-minute budget after 6–17 steps.

These are results for an 8B local model, not a frontier model: absolute rates aren't comparable to published agent benchmarks, and the toolset effect is model-dependent (strong models often do fine with bash alone).

How it works

flowchart LR
    T["issue + test command"] --> P["build request<br/>system + task + append-only history"]
    P --> M["model adapter<br/>Ollama · OpenAI-compatible · Anthropic · fake"]
    M -->|tool calls| X["Toolbox"]
    X -->|"docker exec, timeout"| S[("episode container<br/>no network · tests :ro")]
    S -->|"truncated observation<br/>(errors are data)"| P
    M -->|"submit / budget / loop / refusal"| G["git diff → grader<br/>in a fresh container"]

• One readable loop. agent/loop.py is a single for step in range(max_steps) with every exit going through one finish(). Stop reasons: submitted, step_limit, cost_limit, time_limit, model_refusal, loop_detected, error.

• Append-only history. The provider's raw assistant content is replayed byte-for-byte, which keeps prompt caches warm and thinking blocks valid. When the prompt nears its limit, the model writes a progress summary and the loop starts a fresh list; it never edits history.

• Failure modes handled in code. Transient API errors are retried with backoff and jitter. Tool errors and malformed calls go back to the model as observations. The same action 3 times in a row gets a nudge, and 5 times stops the episode. Every observation is truncated with an explicit marker.

• An interface designed for the model. str_replace requires a unique exact match. When it fails, it shows the most similar region of the file so the model can copy it instead of re-typing code from memory (ADR-012).

Line-by-line walkthrough: docs/learning/LOOP_WALKTHROUGH.md · diagrams: docs/ARCHITECTURE.md · design records: docs/DECISIONS.md.

Sandbox

All of it is tested against a real container in tests/integration/test_sandbox_security.py.

Benchmark

• Targets: toolz 1.2.0 and boltons 26.2.0 (both BSD-3), pinned by commit. Each full test suite runs in under 2 s (ADR-004).

• Strategy A, syntax-tree mutations: flip a comparison, off-by-one, negate a condition, swap arguments, remove a guard, change a constant, swap and/or. One site per bug, spliced into the original formatting.

• Validation: the original passes; the mutated repo fails specific tests (FAIL_TO_PASS) while the rest pass (PASS_TO_PASS), identically on two runs; duplicates and bugs breaking over 25% of the suite are rejected.

• Bug reports are written from failing test names and assertion output only, never from the diff. A leak check rejects any report that names the file, the line or the changed code.

• 36 tasks (18 per library). Strategy B (a model rewrites a function with one subtle bug) produced 6 more validated tasks with the local model, in bench/tasks_llm/. They aren't evaluated yet (ADR-013).

Quickstart

Requirements: Docker, uv, and Ollama for real-model runs.

git clone https://github.com/varad-oss/patchloop.git && cd patchloop
make setup && make images       # Python env + sandbox images (a few minutes, once)
make demo                       # a scripted model fixes a toy bug in the sandbox; no model needed

Run the real agent on the benchmark, for free:

make local-model                # pulls qwen3:8b (~5 GB) and builds the 16K-context variant
make eval-local RUN=E1          # 36 tasks, about 2.5 h on an Apple M5
make progress                   # live progress bar, current step and time left

patchloop commands below assume the environment is active (source .venv/bin/activate). Then patchloop report --run-id E1 writes a summary and results/E1/viewer.html, a page with every step of every episode. With an Anthropic key in .env, make eval-pilot RUN=E1 MODEL=claude-haiku-5-5 measures the cost on 3 tasks before anything larger runs.

Reproducing the numbers

MCP server

The same sandboxed tools are available to any MCP client, such as Claude Code or Claude Desktop, built on the official Python SDK v2. Each session gets its own container, created lazily and removed when idle:

claude mcp add patchloop -- "$(pwd)/.venv/bin/python" -m patchloop.mcp_server.server

Recorded demo: docs/MCP_DEMO.md shows Claude Code, with all built-in tools disabled, fixing the toy bug through the server in 8 turns, and being refused (Read-only file system) when it tries to edit a test. Config for Claude Desktop, a risk analysis (tool poisoning, over-broad permissions) and the reproducible demo command: docs/MCP.md.

Project structure

src/patchloop/
  agent/        loop.py (the loop) · models.py (adapters, pricing) · tools.py (6 tools)
                context.py (compaction) · prompts.py · trajectory.py (JSONL)
  sandbox/      docker_env.py (one locked-down container per episode) · images/ (Dockerfiles)
  benchgen/     mutators.py (strategy A) · llm_rewrite.py (strategy B) · validate.py · issues.py · build.py
  evals/        runner.py · grader.py · metrics.py · report.py · replay.py · viewer/
  mcp_server/   server.py
bench/tasks/    the 36 benchmark tasks      bench/tasks_llm/   strategy-B tasks
results/        committed episodes and trajectories behind every number
docs/           report, analysis, architecture, decisions, MCP, learning notes (M0-M7)
tests/          unit/ (offline) and integration/ (Docker)

Limitations

• One small local model. Results are for qwen3:8b, one trial per task; intervals are wide.

• Synthetic bugs. Mostly single-line mutations, easier to localize than real-world bugs.

• Contamination. Both libraries are surely in the model's training data. The bugs are new, but the original code is familiar.

• Template bug reports read like CI logs. A review of 10 found all reproducible and leak-free, but 6/10 titles name test functions rather than real APIs (review).

• Docker is the isolation boundary. A kernel or container-runtime escape would defeat it; a microVM (gVisor, Firecracker) would be stronger.

Roadmap

• Evaluate the strategy-B tasks, and grow them past 10 with a stronger model.

• Best-of-k with test-based selection, reported as pass@k and pass^k (implemented, not run).

• Repeat E1/E2 with a stronger model to test whether the interface effect shrinks.

• Real-bug tasks mined from the libraries' own history.

• More natural bug reports: titles from real API names, and LLM-written prose behind the same leak check.

Credits

Built by Varad Rajiv Pandare, with Claude Code as a pair programmer, from my specification and under my direction. Benchmark design, experiments and analysis are mine.

Ideas drawn from SWE-bench (FAIL_TO_PASS / PASS_TO_PASS grading), SWE-agent (agent-computer interface design), mini-SWE-agent (linear history, stateless bash) and SWE-smith (LLM-rewrite bug generation).

License

MIT