AI Test Pilot
by Drzymek92
README.md
# AI Test Pilot
> An LLM-driven test generator with a shared core and pluggable adapters. Point it at a Python
> module or a web page; it introspects the target, has an LLM propose test scenarios as
> schema-validated JSON, renders them into runnable tests, runs them, and triages the failures.
[](https://github.com/Drzymek92/ai-test-pilot/actions/workflows/ci.yml)
**Why it's different:** most "AI writes your tests" tools let the model emit test *code* directly —
which hallucinates imports, fabricates inputs, and asserts wrong things. AI Test Pilot takes the
opposite stance: the LLM only ever returns **structured, schema-validated JSON**; every line of
runnable code is rendered **deterministically** from that JSON. The LLM is used for the two genuinely
fuzzy steps only — *proposing* scenarios and *judging* ambiguous failures — and nothing else.
## Does it actually catch bugs?
Coverage is a weak proxy — a suite can execute every line and assert nothing useful. The real test of
a test generator is **kill rate**: generate a suite from *correct* code, then run it against a *buggy*
version and count how many bugs a test that passed on the correct code now catches (standard
mutation-testing semantics). The full reproducible eval ships in [`benchmark/`](benchmark/DETECTION.md).
| Corpus | Kill rate (95% Wilson CI) | Notes |
|---|---|---|
| **QuixBugs** (external, default config) | **0.80** [0.584–0.919] (n=20) | human-verified correct↔buggy pairs |
| **In-repo AST mutation** (full ablation) | **0.818** [0.523–0.949] (n=11) | controlled mutants; the "is it overengineered?" ablation substrate |
| **HumanEval held-out — standard tool (cosmic-ray)** | **0.923** [0.906–0.937] (n=1166) | re-measured with a standard mutation tool on code never seen during development |
**vs a search-based peer:** on the same QuixBugs targets and the same kill mechanic, **ours 0.80 vs
Pynguin's 0.30** (Pynguin at a modest 30s/SIMPLE budget — a floor; even the literature's stronger SBST
range, ~0.59–0.70, sits below 0.80). The edge is the **coverage-feedback loop** — after the first pass
the suite's uncovered lines are fed back to the LLM to reach them — the feedback mechanism plain LLM
test-generation lacks.
**Honest framing:** that held-out number was *first* measured with this repo's own lightweight mutation
operators (0.98) — a generalization check, **not** a PIT-style mutation score — so it's reframed and
**re-measured with a standard tool (0.923)** for a comparable figure; cosmic-ray doesn't exclude
equivalent mutants, so 0.923 is a conservative lower bound, and the harder, most representative number
remains QuixBugs **0.80**. Methodology, comparables vs the literature, and re-run procedure:
**[benchmark/DETECTION.md](benchmark/DETECTION.md)**.
```bash
python scripts/main.py detect --subset 20 # QuixBugs + in-repo mutation kill rate + feature ablation
```
## Demo — sample run
```text
$ python scripts/main.py --target path/to/rules/commission.py --selector compute_commission --golden
introspected 1 unit(s); resolved types: OrderView, LineItemView, RulesConfig, CommissionRules
generated 5 scenario(s)
golden mode: locked 5 characterization assertion(s)
run complete: 5/5 passed
✓ 5 passed · 0 failed · 0 error / 5 generated
tests: scripts/outputs/tests/test_commission_<ts>.py
report: scripts/outputs/reports/report_<ts>.md
```
A generated test constructs the real typed inputs and locks the computed result:
```python
def test_standard_commission():
"""Commission for a multi-item order."""
result = compute_commission(
order=OrderView(currency="PLN", status="DELIVERED", line_items=[
LineItemView(category="electronics", unit_amount=Decimal("100.00"), quantity=2)]),
config=RulesConfig())
assert repr(result) == ("CommissionBreakdown(currency='PLN', "
"items_commission=Decimal('4.00'), transaction_fee=Decimal('1.00'), "
"total_commission=Decimal('5.00'), rule_version='v1')")
```
For the `web_playwright` adapter, the same pipeline produces self-contained Playwright tests and an
idiomatic `.spec.ts` export. Three sample targets are included: `demo/signup.html` (simple form),
`demo/login_app/` (served — auth/`storage_state` + API interception), and `demo/ws_app/` (served —
WebSocket push/echo). The served demos are run with `--serve`:
```bash
# deep web: emits base_url + auth_state fixtures, page.route interception, async variant
python scripts/main.py --adapter web_playwright --target demo/login_app/index.html --serve
# websocket: emits page.route_web_socket mock (server push + echo) + expect_ws_message
python scripts/main.py --adapter web_playwright --target demo/ws_app/index.html --serve
```
## Features
- **Structured-output pipeline** — the LLM returns JSON validated against a Pydantic schema; every line
of test code is rendered deterministically from the validated objects, never written by the model.
- **Typed-input construction** — resolves parameter *types* from source (dataclass/Pydantic/`attrs`/
`NamedTuple`, nested) via `ast` **without importing the target**, so it tests real domain/OO code, not
just functions taking primitives.
- **Reproducible & fail-safe** — `temperature=0` + a scenario cache replay identical tests for an
unchanged target; the LLM call has timeout/retry and a documented **exit-code contract**, so it never
emits a half-generated suite.
- **Proven bug detection** — a `detect` command measures **mutation kill rate** (not just coverage),
with a feature ablation and a **coverage-feedback loop** plain LLM test-gen lacks.
- **Advanced Playwright (served mode)** — `base_url`/`auth_state` fixtures, `storage_state` reuse,
`page.route` network interception, and in-process WebSocket mocking — all from JSON the model emits.
- **MCP server + quality/cost gates** — callable from any MCP client; a quality-regression gate
(false-positive rate, test-smell density…) and opt-in budget caps that abort before overspending.
<details>
<summary><b>Full feature detail</b> (click to expand)</summary>
- **Structured-output pipeline** — the LLM returns JSON validated against a Pydantic schema (with a
one-shot repair retry); code is generated from the validated objects, never written by the model.
The value grammar's `$type`/`$call`/`$enum` symbols (and constructor argument names) are
**allow-listed against the types resolved from the target's own source**, so a crafted
docstring/source can't smuggle code tokens into a generated test that then gets executed.
- **Typed-input construction** — recursively resolves a function's parameter *types* from source
(dataclass + Pydantic + `attrs` + `NamedTuple`, nested, `Decimal`/`datetime`/`Enum`, defaults) via
`ast` — **without importing the target** — and builds real constructor calls. It even surfaces
Pydantic field constraints (`gt`/`le`/`min_length`, `Annotated[...]`) so the model picks
*valid* values instead of triggering a construction-time `ValidationError`. Lets it test
domain/OO code, not just functions taking primitives.
- **Reproducible generation** — generation runs at `temperature=0` and caches each scenario set keyed
by the prompt + resolved model version + temperature, so re-running an unchanged target replays the
*identical* tests at zero token cost; a model change invalidates the cache. `--no-cache` /
`--refresh-cache` override it.
- **Assertion strength from source** — the unit's own code is fed into generation (a bounded slice),
so assertions target *specific computed behaviour*, not just type/shape. `--golden` then locks the
real result; together they turn a draft into a regression guard.
- **Characterization (golden) mode** — runs each call and locks the assertion to the real
result, turning a generated test into a regression guard. Guarded against time-bombs: it
double-runs and keeps only reproducible results, skipping any clock/RNG-reading unit whose
time isn't pinned.
- **File & fixture inputs** — creates real temp files for file-processing functions, and can
optionally seed inputs from a companion [synthetic data factory](https://github.com/Drzymek92/synthetic-data-factory).
- **Failure triage** — a deterministic signal table classifies most failures for free
(`bad_scenario` / `env_issue` / a broken golden lock → `real_bug`); the LLM is called only for the
genuinely ambiguous ones.
- **Advanced Playwright (served web mode)** — fixtures (`base_url`, `auth_state`), authenticated
sessions via saved `storage_state`, network interception (`page.route`) to stub APIs
deterministically, in-process WebSocket mocking (`page.route_web_socket`, server-push + echo), and
an `async_playwright` variant. Each is just structured JSON the LLM emits — no Playwright code from
the model.
- **Self-tracking ledger + self-improving tuning** — every run is recorded to DuckDB; `accept`
backfills how many tests you kept. The tool then proposes the best prompt version *and* (in `auto`
mode) injects your previously-accepted scenarios for the same target as few-shot exemplars — closing
the loop with **zero extra LLM calls**.
- **Draft → suite workflow** — `discover` scans a project and prints ready-to-run commands per module;
`promote` strips a draft's boilerplate, rewrites golden locks into value assertions, and appends only
the non-duplicate tests into an existing suite. Both deterministic, zero-token.
- **MCP server** — exposes the engine as tools (`introspect`, `generate_tests`, `triage_failures`,
`run_metrics`, `accept_run`) so it's callable from any MCP client.
- **Fail-safe by contract** — the LLM call has a timeout + exponential-backoff retry and raises
rather than ever emit a half-generated suite; a per-test run cap bounds a hanging test; and the CLI
has a documented **exit-code contract** (`0` ok · `1` internal · `2` usage · `3` uninspectable
target · `4` LLM failure · `5` quality regression · `6` budget exceeded · `7` detection regression)
for scripting.
- **Quality regression gate** — `quality` runs a curated known-good target set and reports a metric
panel (coverage, pass-rate, false-positive rate, error rate, test-smell density, acceptance), gated
against a stored baseline so a generation/prompt change that regresses quality fails loudly.
- **Cost guardrails** — every run measures and records real token spend (a cache replay is free);
opt-in `[budget]` caps abort before overspending, and `sweep` "tests the diff" by generating only
for git-changed modules under a per-sweep cap.
- **Proven bug detection (not just coverage)** — a `detect` command measures **mutation kill rate**
(generate from correct code, re-run against buggy/mutant versions, count what's caught) over an
external corpus (QuixBugs) + in-repo mutants, with a feature ablation and a standard-tool
(cosmic-ray) cross-check on held-out code. A **coverage-feedback loop** then feeds the suite's
uncovered lines back to the LLM to target them — the feedback mechanism single-shot LLM test-gen
lacks. See [Does it actually catch bugs?](#does-it-actually-catch-bugs) and `benchmark/DETECTION.md`.
</details>
The same engine drives two target types through one **adapter seam**, so adding a new kind of
target is a single new file with zero changes to the core:
- **`python_pytest`** — points at a Python module, generates `pytest` tests.
- **`web_playwright`** — points at a web page, generates Playwright end-to-end tests (and exports
idiomatic TypeScript alongside the runnable Python).
## How it works
```mermaid
flowchart LR
T([target: module or URL]) --> I[1 · introspect<br/>ast / DOM — deterministic]
I --> G[2 · generate<br/>LLM → schema-validated JSON]
G --> M[3 · materialize<br/>render code — deterministic]
M --> R[4 · run<br/>pytest / Playwright]
R --> TR[5 · triage<br/>signals + LLM for ambiguous]
TR --> L[6 · record<br/>DuckDB ledger]
L -. 7 · propose tuning .-> G
```
Stages 1, 3, 4, 6 cost **zero tokens**. Stage 2 is one batched LLM call; stage 5 calls the LLM only
for failures the deterministic signal table can't classify. The core never imports an adapter
directly — only through a name registry — which is what keeps the two target types fully decoupled:
```mermaid
flowchart TD
C[shared core<br/>introspect · generate · materialize · run · triage · record] --> RG[registry]
RG --> A1[python_pytest adapter]
RG --> A2[web_playwright adapter]
A1 --> P[(pytest)]
A2 --> PW[(Playwright + TS export)]
```
## Tech Stack
- **Language:** Python 3.10+
- **Core:** `pydantic` (the schema spine), `jinja2` (pytest emission), `duckdb` (the run ledger)
- **LLM:** `langchain-openai` against any OpenAI-compatible gateway (`LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`)
- **Adapters:** `pytest` (python target runner), `playwright` (web — bundles its own driver, no Node needed)
- **Integration:** `mcp` (FastMCP) — exposes the engine over the Model Context Protocol
## Getting Started
### Prerequisites
- Python 3.10+ on PATH
### Installation
```bash
git clone https://github.com/Drzymek92/ai-test-pilot.git
cd ai-test-pilot
python -m venv .venv
.venv\Scripts\activate # Windows (source .venv/bin/activate on macOS/Linux)
pip install -e . # installs deps + the `ai-test-pilot` command
# (or `pip install -r requirements.txt` for deps only, without the console entry point)
cp config/.env.example config/.env # then fill in your LLM gateway values
# for the web adapter only:
python -m playwright install chromium
```
Installed this way you can call it as `ai-test-pilot ...` anywhere; the examples below use
`python scripts/main.py ...`, which is equivalent.
### Usage
```bash
# generate pytest tests for selected functions
python scripts/main.py --target path/to/module.py --selector func_a,func_b
# lock assertions to real results (characterization / regression mode)
python scripts/main.py --target path/to/module.py --golden
# also emit deterministic validator-rejection tests (a validator-gated pydantic type refuses bad input)
python scripts/main.py --target path/to/module.py --reject-tests
# generate Playwright tests for a web page
python scripts/main.py --adapter web_playwright --target path/to/page.html
# record how many proposed tests you kept (feeds tuning)
python scripts/main.py accept <run_id> --kept 4
# scan a project for testable targets (deterministic, no LLM)
python scripts/main.py discover path/to/project
# incremental: only the modules changed in git (regenerate tests for what moved, not the whole tree)
python scripts/main.py discover path/to/project --changed # working tree vs HEAD
python scripts/main.py discover path/to/project --since main # vs a ref / tag / commit
# clean a draft for the suite: strip boilerplate, rewrite golden locks, append non-duplicates
python scripts/main.py promote <run_id> --into tests/test_module.py
# force a fresh (uncached) generation, or regenerate + overwrite the cache
python scripts/main.py --target path/to/module.py --no-cache # bypass the scenario cache
python scripts/main.py --target path/to/module.py --refresh-cache # regenerate and re-store
# quality gate: run the curated set, print the metric panel, fail (exit 5) on regression
python scripts/main.py quality # gate vs baseline
python scripts/main.py quality --update-baseline # (re)set the baseline
# "test the diff": generate only for git-changed modules in a project, under a budget cap
python scripts/main.py sweep path/to/project --since main
```
Run as an **MCP server** (callable from any MCP client) — register it with:
```json
{ "command": "python", "args": ["/path/to/ai-test-pilot/scripts/mcp_server.py"] }
```
## Project Structure
```
scripts/
main.py # CLI + run_pipeline() (the one pipeline every interface reuses)
mcp_server.py # MCP server (FastMCP) exposing the engine as tools
cli.py # argument parsing + subcommand dispatch + the exit-code contract
pipeline.py # run_pipeline() — the one pipeline every interface reuses (typed RunRequest in)
core/ # adapter-agnostic engine: models, generate, materialize, runner, triage,
# ledger, tuning, context, fixtures, registry, discover, promote, report,
# feedback, cache, errors (exit codes), quality (gate), budget, detection
adapters/ # python_pytest · web_playwright (one file per target type)
prompts/ # scenario-generation prompts + the pytest Jinja template
config/ # ai_test_pilot.toml (defaults) + .env.example
benchmark/ # reproducible efficacy eval: mutation kill rate, ablation, cosmic-ray +
# Pynguin baselines, Wilson CIs, corpora loaders, DETECTION.md + artifacts
demo/ # signup.html · login_app/ (auth) · ws_app/ (websocket) — web adapter targets
tests/ # 204 unit tests
```
## Design notes
- **Determinism first.** Introspection, code emission, running, the triage signal table, and the
ledger are all plain code. The LLM is a tool for the two irreducibly fuzzy steps only.
- **Never imports the target.** Introspection is `ast`-only, so a target's heavy/optional
dependencies are never triggered to generate tests for it.
- **Human-in-the-loop.** Generated tests are *proposed* into `scripts/outputs/` — never written into
a target repository. Promoting them is a separate, explicit step.
Validated end-to-end across a typed business-rules engine, pure data-transformation helpers, a web
form, an authenticated app with API interception, and a WebSocket feed — producing runnable,
correctly-typed, regression-grade tests in each case.
> **CI note:** the published CI runs the full unit suite, which is browser-free by design — the
> `web_playwright` tests assert on the *generated test source*, not a live browser. The `--serve`
> demos are run locally (after `playwright install chromium`); CI doesn't download a browser.
## Limitations
The `python_pytest` adapter is a tested, quality-gated, cost-bounded tool I rely on for my own
pipelines — reproducible (temperature 0 + scenario cache), fail-safe (typed exit-code contract), and
guarded by a regression gate. Its honest scope is **trusted, single-machine** code: it executes the
tests it generates and sends introspected source to the configured LLM gateway, so it is not a
sandbox for untrusted targets. The `web_playwright` adapter remains a **demo** of advanced Playwright
technique. The input shapes the tool deliberately won't guess at, the non-determinism of the LLM
step, and the scale/safety/packaging boundaries are all written up in
**[LIMITATIONS.md](LIMITATIONS.md)** — knowing them is part of using it well.
## License
Licensed under the MIT License — see [LICENSE](LICENSE).