# testgen-eval

**An agent that tests another agent.** [`testgen`](https://github.com/nishant20/testgen)
generates test suites from requirements using Claude; `testgen-eval` calls it
over MCP, runs a deterministic gate plus an LLM-judge pass against a written
quality rubric, and reports back what's good, what's weak, and what's
outright broken.

```bash
npm run batch -- sample-requirements.json out/
open out/report.html
```

---

## Why this exists

This is the second half of a two-project pair built to practice things a
single CLI tool doesn't force you to learn: **MCP** as a real client/server
boundary, **spec-driven development** (writing down what "good output" means
*before* building the thing that judges it), and — the part most worth a
second look — a framework for **agents that evaluate other agents**. As LLM
agents show up more in test automation, "how do you know the agent's output
is actually trustworthy" becomes the real question, and this project is a
concrete answer.

`testgen-eval` doesn't reimplement `testgen`. It treats it as an external
service, reachable only through the MCP tool `generate_test_suite` — the
same constraint a real third-party client would have.

## How it works

```mermaid
flowchart TD
    R[Requirement + instructions] --> MCP[MCP client]
    R --> CLS[Classifier]
    MCP -->|generate_test_suite| TG[testgen HTTP API] --> TS[(TestSuite)]
    CLS -->|"security/NFR-sensitive?"| SENS[(sensitivity verdict)]
    TS --> GATE{Gate<br/>deterministic}
    GATE -->|fail| REJECT[Rejected - reported, not judged]
    GATE -->|pass| JUDGE[Judge - LLM, one call]
    SENS --> JUDGE
    JUDGE --> REPORT[Report<br/>markdown / json / html]
    REJECT --> REPORT
    REPORT --> SUMMARY[Batch summary<br/>pass rate, avg score, worst offenders]
```

Each requirement goes through the same five stages ([full breakdown in
ARCHITECTURE.md](ARCHITECTURE.md)):

1. **MCP client** spawns `testgen-eval`'s own MCP server and calls
   `generate_test_suite` — the only way this harness talks to `testgen`.
2. **Classifier** — a separate, small LLM call judging whether the
   requirement is security/NFR-sensitive (auth, payments, PII, rate
   limiting). Runs concurrently with generation; the result feeds the gate
   check below rather than being decided ad hoc.
3. **Gate** — plain TypeScript, no LLM. Rejects a suite only if it's
   genuinely unusable (empty IDs, cases with no steps, an empty feature
   name). A suite that passes the gate is *usable*; it isn't necessarily
   *good*.
4. **Judge** — one structured LLM call scoring a gate-passed suite against
   ten quality criteria (concreteness, realism, duplication, whether
   `instructions` were actually reflected, missing security coverage for
   sensitive requirements, scenario conflation, and more) — see
   [SPEC.md](SPEC.md) for the full rubric and the reasoning behind it.
5. **Report + batch summary** — every requirement gets its own report
   (markdown, JSON, or HTML); a full batch also gets one aggregate summary
   (gate pass rate, average score, worst offenders, and any pipeline
   failures called out separately from low scores).

## What "good" means

The rubric lives in [SPEC.md](SPEC.md), not in this README, because it's
meant to be read on its own and argued with. The short version: the gate
only protects against a test case being **non-actionable** — a tester
literally can't execute it. Everything about completeness, realism, or
polish is a *scored* quality signal, never a rejection reason.

## Setup

Requires `testgen` running locally — this harness calls it, it doesn't embed
it:

```bash
cd ../testgen
.venv\Scripts\activate      # macOS/Linux: source .venv/bin/activate
testgen serve                # stays up at http://127.0.0.1:8000
```

Then, in `testgen-eval`:

```bash
npm install
```

Both `testgen` and `testgen-eval` read `ANTHROPIC_API_KEY` from the
environment.

## Usage

### Run a batch through the full harness

```bash
npm run batch -- <requirements.json> [outDir]
```

`requirements.json` is a JSON array:

```json
[
  {
    "requirement": "Users must be able to reset their password via an emailed link.",
    "instructions": "focus on security and boundary cases"
  },
  { "requirement": "Users can toggle dark mode from the settings screen." }
]
```

(`sample-requirements.json` in this repo is a ready-to-run example.)

Each run writes three things to `outDir` (defaults to `out/`):

| File | What it is |
| --- | --- |
| `results.jsonl` | One line per requirement, appended as it completes — survives a crash mid-batch. |
| `batch-summary.{md,json}` | The aggregate: gate pass rate, average score, worst offenders, pipeline errors. |
| `report.html` | One self-contained page: stat strip, error banner, and a full scorecard per requirement. Open it in a browser. |

A pipeline failure on one requirement (a timeout, a malformed response)
doesn't abort the batch — it's isolated, logged into `results.jsonl` and the
error banner, and the rest of the batch keeps going.

### Try just the MCP server

The harness's own MCP server (`generate_test_suite`) can also be driven
directly, without the eval pipeline around it — useful for seeing the
protocol itself:

```bash
npx @modelcontextprotocol/inspector tsx src/index.ts
```

## Project structure

```
src/
  testgenClient.ts   testgen's HTTP API client
  server.ts           the MCP server and its one tool
  index.ts             MCP server entrypoint (stdio transport)
  harness/
    mcpClient.ts       MCP client - reuses one connection across a batch
    classifier.ts      requirement sensitivity classification
    gate.ts            deterministic structural checks
    judge.ts           LLM-judge quality scoring
    report.ts          per-requirement report (markdown / json / html)
    batch.ts           aggregate batch summary (markdown / json / html)
    htmlPage.ts         assembles the two html fragments into report.html
    htmlEscape.ts       XSS-safe escaping for LLM-sourced content
    run.ts              orchestrates everything, loops over a batch
```

## Development

```bash
npm test
```

Covers everything deterministic — `gate.ts`, `report.ts`'s and `batch.ts`'s
renderers — offline, no API calls. `classifier.ts`, `judge.ts`,
`mcpClient.ts`, and `run.ts` call real LLMs/APIs, so they're smoke-tested by
hand rather than covered by the automated suite.

## Roadmap

- Resume a partially-completed batch instead of always starting over.
- Model test *layer* (UI vs API) in `testgen` so type-coverage can become a
  real scored criterion instead of an informational one (see SPEC.md).
- Additional input sources beyond a hand-written JSON file.

## Related

- [`testgen`](https://github.com/nishant20/testgen) — the agent under test.
- [SPEC.md](SPEC.md) — what "good" means for a generated test suite.
- [ARCHITECTURE.md](ARCHITECTURE.md) — the harness's components and the
  decisions behind them.

## License

MIT