agent-testbench

A test bench your coding agent operates. Claude Code, Codex, Cursor or any agent gets a live kernel on real hardware - your machine, a Docker sandbox, any box you reach with ssh, a Modal GPU or a Colab VM - to run code and notebook cells, look at the values and plots, fix and rerun. Then it proves the result with a clean, checked run whose report names the code it ran and fingerprints its evidence. Every run is priced before it starts and stopped on the machine itself at its limit, inside a budget the agent cannot raise.

$ testbench loop start smoke --metric outputs/metrics.json:accuracy --budget-usd 0
BASELINE: accuracy 0.86, run 20261031-054550-smoke passed.

$ testbench loop try "hidden units 8 -> 64"        # the agent edited the code; testbench commits, runs, judges
KEPT try 2: accuracy 0.9689 (best was 0.8689, +0.1). Run 20261031-054609-smoke passed. Best is now 509b79c25.

$ testbench loop try "learning rate 0.05 -> 0.1"
REVERTED try 3: accuracy 0.9689 is not better than the best 0.9689. The branch is back at 509b79c25.

That is a real run: Claude Code, given only this plugin, took a model from 0.86 to 0.9756 accuracy in six tries, four kept and two reverted by testbench, in 78 seconds - docs/demo-loop.md.

testbench report --format html for the demo's fixed run - a real report, unedited (the tour).

Quick start

pip install "agent-testbench[all]"         # or: uv tool install "agent-testbench[all]"
testbench selftest                         # ten checks of the main promises on this machine, in under a minute
testbench demo                             # a small project, run for free here - see it work in a minute
testbench demo llm-app                     # or web-app, test-suite: your kind of project (testbench demo --list)
cd your-project && testbench init          # looks at the project and writes plans that already check something
testbench run <plan>                       # the report: what passed, what failed and why, what it cost

testbench init finds notebooks, a pytest suite, training scripts and web apps, and writes a plan for each - lint, run, the files they write, every test passing, a page with no console errors - with a FILL IN line for the number only you know ("accuracy >= 0.9"). Starting from a goal instead? testbench recipes lists ready ladders of plans - pytest, a Colab notebook, PyTorch training, a LoRA fine-tune, a web app, an LLM eval, a Docker sandbox, your own GPU box, a data pipeline - with each plan's worst case; testbench init --recipe pytorch-train writes one, and --add adds one to the testbench.yaml you have (docs/recipes.md). Not only Python: init turns npm test (jest, vitest, mocha, node --test, Playwright), a Go module or a Rust crate into a plan judged by the runner's own JUnit report, and the jest, vitest, go-test and cargo-test recipes also run the suite in a Docker container with no network.

Claude Code - the plugin adds the skills and a guard hook for spending and secrets:

claude plugin marketplace add sarptandoven/agent-testbench
claude plugin install agent-testbench@agent-testbench

Then ask in plain words: "get training working and prove it with the smoke plan", or "use testbench loop to raise accuracy, budget $2".

Cursor, VS Code, Codex, Gemini CLI: testbench connect cursor (or vscode, codex, gemini, claude; --user for every project, --list to see what is set up) registers the MCP server in the client's own config and writes the working rules into AGENTS.md; docs/clients.md says which guardrails hold outside Claude Code.

Any other MCP client (nothing to install with uv):

{ "mcpServers": { "agent-testbench": { "command": "uvx", "args": ["--from", "agent-testbench[all]", "testbench", "mcp"] } } }

In CI: uses: sarptandoven/agent-testbench@v1 with plan: smoke; comment: true puts the verdict and the numbers' change since the base branch on the pull request, and attest: true signs the report so testbench verify --attestation shows which workflow and commit made it (docs/github-action.md).

Your own agent or a notebook: from agent_testbench import Testbench runs plans and live sessions from Python, and agent_testbench.tools gives the same tools to the Anthropic or OpenAI API (tools.anthropic_tools(), tools.call(name, input)), inside the same budget, approvals and machine limits (docs/python-api.md, examples/agent-loop).

Any other agent reads the working rules testbench init writes into AGENTS.md.

Related MCP server: notebook-agent-mcp

What your agent can do with it

• Work live in a kernel that stays up: run code, .py files or a notebook's cells (Colab form fields set), get back what it printed, the last value, plots as images, and errors with the line that raised them. Install packages, run shell commands, move files, push local edits without losing variables.

• Prove it with a plan: notebooks, scripts or web UIs run end to end, checked against expectations - metric thresholds and tolerances, the project's own test suite, numbers in the log, files, video frames, a page's text, GPU use, a plot that should look like a known-good one. The report leads with what failed and says what changed since the last run.

• Rescue a notebook that no longer runs: testbench rescue nb.ipynb reads it and runs it top to bottom, then says which cell fails and why - a missing package and its pip name, cells that only worked in another order, a Colab-only path, a file downloaded by hand, a GPU this machine lacks - with a plan that proves it once fixed.

• Test LLM apps for free: mocks: [openai, anthropic] on a step swaps in strict mock SDKs - no key, no network, the same answer every time, and the SDK's own BadRequestError for a request the real API would refuse.

• Improve it with a loop: each idea is committed, proved by a run, and kept only if the number got better.

• Stay inside a budget: worst case priced before every launch, approvals above a threshold, machine-wide limits a project cannot raise, and a hard stop on the machine doing the work.

• Show it: reports as HTML pages with the evidence inline, JUnit for CI, history and trend lines, and proof records anyone can check with testbench verify.

Where it runs

Each is set per plan (backend:) or per session (--backend), so the same notebook can be proved for free locally, then on the target GPU for a few cents. Linux, macOS and Windows (Colab's CLI has no Windows build).

Live sessions

$ testbench session start gpu --backend modal --gpu T4 --idle-minutes 3 --max-minutes 8
gpu  ready on modal T4  up 0.0 min, idle 0.0 min (stops after 3 idle / 8 total)  ~$0.000 so far
Within budget: up to $0.16 (today so far $0.01 of $2.00).

$ testbench exec -f check_gpu.py
[check_gpu.py | ok | 3.3 s]
20 matmuls of 4096x4096 on Tesla T4: 0.67 s
=> torch.Size([4096, 4096])

$ testbench exec "float(b.mean()), torch.cuda.max_memory_allocated() // 2**20"      # variables persist
[code | ok | 0.2 s]
=> (1023.1211547851562, 200)

$ testbench session stop
gpu stopped after 1.4 min, about $0.017

Also: exec --cells nb.ipynb:1-4 --params '{"EPOCHS": 1}', files ls|get|put, session sync|install|shell|list. A call past --timeout is interrupted and the kernel keeps its variables; a session stops itself when idle.

Plans and expectations

plans:
  smoke:                               # cheapest proof first: here, free
    backend: local
    steps:
      - name: tests
        run: python -m pytest -q --junitxml=junit.xml
        expect:
          junit: {path: junit.xml, failures: 0, tests: ">= 10"}       # names each failing test
      - name: train quick
        notebook: train.ipynb
        params: {EPOCHS: 1}                                           # Colab form fields
        expect:
          json: {outputs/metrics.json: {accuracy: "~= 0.93 +- 0.02"}}
          metrics: {stdout: {"val_loss=([0-9.]+)": "< 0.3"}}          # a number printed to the log
          image: {outputs/confusion.png: {like: golden/confusion.png, max_diff: 0.05}}
  gpu-check:                           # a few minutes on the target GPU before a long run
    backend: modal
    gpu: A10
    steps:
      - name: train short
        notebook: train.ipynb
        max_minutes: 10                # also the hard stop, and what the worst case is priced on
        expect:
          gpu: {busy_pct: ">= 50"}     # an idle GPU fails the step

Steps are notebook: (an .ipynb or a percent-format .py; params: sets Colab form fields and papermill parameters - docs/notebooks.md), run: (a shell command) or web: (a page driven in headless Chromium: click, fill, wait, screenshot). mocks: [openai, anthropic] gives a step mock LLM SDKs, and mock_calls: {anthropic: {calls: "<= 5"}} bounds what the code asked for (docs/mocks.md). Every expectation and what its failure looks like: docs/expectations.md. Every field of testbench.yaml: docs/reference.md.

$ testbench run smoke                  # after a bad edit to the data cell (abridged)
# FAIL - plan `smoke` on local

Run `20261031-064120-smoke`, started 2026-10-31 06:41, 0.05 min, about $0.000. Stopped because step 2 (train quick) failed.

## FAIL: train quick

- **cell 2**: `AxisError`: axis 2 is out of bounds for array of dimension 2
- raised at cell line 10: X_train_mean = X.mean(axis=2) (raised in _methods.py:83, _count_reduce_items)

## Compared with the previous run `20261031-064113-smoke` (passed)

- Broke: train quick

Proof records

Each report names what it proves: the git commit it ran (and a hash of any uncommitted changes), the Python, platform and packages where it ran, and the sha256 of every evidence file, in a format published as a JSON Schema. testbench verify says whether the evidence still matches, and testbench diff A B says what changed between two runs - each checked number old -> new and whether that is better by its rule, checks that flipped, evidence that changed, the commits, packages that changed version. testbench watch smoke reruns a free plan whenever you save, one run at a time. report --format html writes one page with the plots inline, --format junit feeds CI test reports, testbench history <plan> shows how the checked numbers moved, and testbench ui writes static pages for every run with a trend line per number; testbench ui --serve shows them live at http://127.0.0.1:8770/ - what is running, its log as it grows, sessions, the loop, and today's spend against your limits - read-only and for this machine only (docs/dashboard.md). testbench export --to mlflow|wandb|csv|jsonl (or export: in testbench.yaml, after every run) puts runs where you already track experiments, with the evidence attached (docs/export.md).

Loops

testbench loop start smoke --metric outputs/metrics.json:accuracy --budget-usd 5 --max-hours 8
testbench loop try "lower the learning rate to 3e-4"     # commit, run, keep or revert
testbench loop status | stop

The loop works on a branch of its own. A try is kept only when its run passes and the metric beats the best so far; otherwise the branch goes back to the best commit, keeping edits made meanwhile. --metric is a number a json check reads (FILE:KEY) or any check's number (check:val_loss). One budget covers every run of the loop, approved once; each run is still checked against the per-run and daily limits. The report lists each kept change, what it moved and the run that proves it; merging is yours to decide.

Left to run, a loop stops itself: --target 0.95 (done), --patience 4 (four tries in a row not kept) or --max-tries 20 refuse the next try and say what to do instead. With notify: {webhook_env: TESTBENCH_WEBHOOK} in testbench.yaml you hear when runs and loops end, or when a launch waits for your yes - a Slack webhook or any JSON POST (docs/loops.md, docs/notify.md).

Guardrails

• Money: worst case = hardware price x time limit (+ start-up), checked before every run, session and loop. Over max_run_usd or a day's limit: refused. Over ask_above_usd: a person approves - in Claude Code the hook asks you, in MCP clients the server asks you directly (elicitation). ~/.testbench/limits.yaml caps every project on the machine; a project can lower those limits, never raise them, and they hold whatever was approved. Prices come from a built-in table that a fetched catalog can raise, never lower (testbench prices). Anything that is not a real number - a NaN budget, an infinite price, a broken ledger line - refuses the launch.

• Hard stops on the machine doing the work: Modal's timeouts, or kernelkit's watchdog on Colab VMs, ssh machines and in containers - so the bound holds with your laptop asleep. Ssh and docker runs are also stopped from this machine shortly after their limit, in case the work there stopped its watchdog.

• Secrets by name only, never on a command line; strict mocks stand in for paid APIs in tests.

• The guard hook (Claude Code) refuses raw cloud launches and commands that would print secrets, asks before an edit raises a budget, lowers a price or loosens a docker sandbox, turns every --approve into a question to you, and refuses edits to the ledgers. It is a safety net, not a sandbox (a script the agent writes can still pass --approve; the machine's limits still hold) - for running code nobody reviewed, use backend: docker with network: none.

MCP and agents

Tools: session_start, session_exec, session_files, session_sync, session_install, session_shell, session_list, session_stop, plans, run_plan, wait_run, get_report, list_runs, cancel_run, spend, lint_notebook, rescue_notebook, recipes, export_run, diff_runs, add_note, loop_start, loop_try, loop_status, loop_stop. They run side by side (stop a session while a long cell runs), return plots and screenshots as images, and pass on progress. testbench mcp --http serves streamable HTTP on 127.0.0.1. In Claude Code: claude mcp add agent-testbench -- testbench mcp, or the plugin, which adds the iterate and setup skills.

Commands

testbench help COMMAND shows a command's options and examples; exit codes are in docs/reference.md (0 passed, 1 failed, 2 refused or bad input, 3 still running).

Documentation

• docs/reference.md - every field of testbench.yaml, sessions, plans, steps, loops, proof records, prices, the ssh backend, machine limits, writing a backend plugin

• docs/expectations.md - every expectation, with what its failure looks like

• docs/recipes.md, docs/notebooks.md, docs/rescue.md, docs/mocks.md - starting points, what notebook steps accept, rescuing notebooks, mock LLM SDKs

• docs/docker.md, docs/kubernetes.md - the docker and kubernetes backends

• docs/loops.md, docs/notify.md, docs/export.md - loops, notifications, exports to MLflow and W&B

• docs/clients.md, docs/python-api.md - other coding agents, the Python API and LLM tool definitions

• docs/tour.md - a five-minute tour with real screenshots, from install to a proved fix

• docs/examples.md - the examples: an LLM app on a mock SDK, a web form in headless Chromium, a failing test suite for your agent to fix, a Colab-style notebook

• docs/why.md, docs/use-cases.md, docs/faq.md, docs/troubleshooting.md

• docs/github-action.md - the GitHub Action: inputs, outputs, what reviewers see

• docs/demo-loop.md, docs/demo-sandbox.md and docs/demo.md - real agent runs, with what they did

• examples/ - a Colab-style notebook with plans for every backend, and a web UI check

Install notes

• Modal: modal setup once (your account; pricing).

• Colab: uv tool install google-colab-cli, then log in once (colab new). Uses your compute units.

• ssh: ssh HOST true must work with keys; ssh: {host: HOST, usd_per_hour: 0} in testbench.yaml.

• Docker: a running Docker (or podman); docker: {network: none} for no network.

• Kubernetes: kubectl with a context for your cluster; kubernetes: {usd_per_hour: {cpu: 0, A100: 2.5}}.

• Trackers: pip install "agent-testbench[mlflow]" or [wandb] for exports.

• Web steps: playwright install chromium.

testbench doctor shows what this machine can run.

How it is tested

python -m pytest runs everything that needs no account - notebooks in real kernels, expectations, budgets and limits, the guard hook, the MCP server over stdio and HTTP, loops on real git repositories, the ssh backend through a stand-in for ssh, Docker in real containers, web steps in Chromium - on Linux and macOS with Python 3.10 and 3.12, and on Windows. tests/test_live.py runs the same things on Modal, Colab or a real ssh host and is opt-in. The demos in docs/ are real agent runs.

Limits

• Costs are estimates (time x price); testbench spend --modal reads Modal's actual bill; Colab unit rates vary by account (colab usage).

• It runs .ipynb files and live kernels; it does not drive the Colab or Modal Notebooks web pages.

• Colab VMs can be reclaimed after an hour or two even while busy; use Modal or ssh for long jobs.

• On Windows, interrupting a cell that is blocked inside a C call (time.sleep, a long native call) waits until that call returns; Python code is interrupted at once.

License

MIT