testrun-mcp — The Hard Way

Build an MCP server that lets an AI agent run your tests and understand the result. Start from subprocess.Popen and end with a server an agent can call over stdio. No test-runner framework hiding what happens, every line explained.

At the end you will have built a minimal version of testrun-mcp yourself, and you will understand the things most "run a command from an agent" tutorials skip: how to run untrusted code without leaking processes, how to shrink a 5,000-line log into five useful lines, and how to test a tool whose whole job is to survive misbehaving programs.

Why This Exists

An agent that fixes bugs runs the tests dozens of times. Handing it the raw output of pytest is the obvious first version, and it fails in predictable ways:

• the log is enormous, and the one line that matters is in the middle;

• a test hangs and the agent's tool call hangs with it;

• the test started a server, the run timed out, and the server is still there an hour later;

• an import error in one file looks like "the tests failed" and sends the agent to fix the wrong thing;

• a flaky test passes on the re-run and the agent decides the bug is fixed.

Every module below is one of those problems, solved and then proven against a real process.

Related MCP server: testmcp

Who This Is For

• Developers who have written a "hello world" MCP server and want to wrap something real

• Anyone who has been bitten by an orphaned child process after a timeout

• Anyone building a tool that runs code they did not write

Prerequisites: Python 3.10+ and comfort with subprocess. Nothing else: the projects under test are built on the fly in temporary directories.

python -m venv .venv && .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -r requirements.txt
python verify.py

Modules

Each module contains:

• README.md: the problem, the design decision, and a walk through the code

• main.py: runs the code from minirun/ against real processes and checks the results with assert

• break-it.md: challenges that break your own code, with hints

How the Code Is Laid Out

The server grows one file per module in minirun/:

minirun/procs.py    Module 01   run a command safely
minirun/junit.py    Module 02   JUnit XML -> test cases
minirun/trim.py     Module 03   tracebacks -> the frames that matter
minirun/runner.py   Module 02+04  run pytest, parse, remember failures, rerun
minirun/flaky.py    Module 05   N runs -> pattern and verdict
minirun/server.py   Module 06   the MCP tools

The module READMEs show the interesting parts of each file and explain the choices; the files are the finished versions, so a module's main.py may use a helper that a later module refined. Each main.py is self-contained apart from importing minirun and common.py (temporary projects, "is this pid dead").

Every Module Is Verified

python verify.py

This runs each module's main.py, which builds throwaway projects and runs real pytest, real processes and (in Module 06) a real MCP session over stdio against them. CI runs it on Linux and Windows on every push and weekly, so the code in this course cannot silently stop working.

Architecture Overview

 AI agent (Claude Code, ...)
        |  MCP over stdio: run_tests / rerun_failed / flaky_check     <- Module 06
        v
 +---------------------------------------------------------+
 | server.py   worker thread, errors -> ToolError            |
 |   runner.py   build argv, parse, remember failures        | <- Module 02, 04
 |     junit.py    XML -> cases with exact node ids          | <- Module 02
 |     trim.py     traceback -> relevant frames              | <- Module 03
 |     procs.py    Popen, timeout, tree kill, caps, env      | <- Module 01
 +---------------------------------------------------------+
        |  argv list, no shell, stdin closed
        v
   python -m pytest --junitxml=...        (the project's code runs here)

Key Design Principles

1. The agent can only act on what you show it. A 200-line traceback hides the one frame that matters. src/cart.py:27 in total is the whole answer.

2. Never trust the thing you run. It will hang, print a gigabyte, read stdin, start children, and call os._exit. Every one of those is a normal Tuesday for a test suite.

3. "Passed" must mean the tests ran. No tests collected, a broken conftest.py and a crashed interpreter are three different results, and none of them is a pass.

4. Errors are instructions. "pytest not found" is a dead end. "pytest is not installed for C:\proj\.venv\Scripts\python.exe; pass python=..." is the next step.

5. Prove the dangerous part. The tree kill is only real if a test shows the grandchild is dead. Module 01 and 07 do exactly that.

MIT licensed.