# batch-coverage

[中文说明](README.zh-CN.md)

Check that an agent processed every item in a batch. Freeze the real inputs before work, audit the outputs, and resume from the IDs that still need repair.

An agent can report “all 100 items are done” after producing 97 outputs. `batch-coverage` checks the frozen input set against the current output set and reports missing, duplicate, unexpected, or malformed results. Changed inputs invalidate the batch instead of silently changing what “complete” means.

Python 3.11+ · CLI and stdio MCP · SQLite · CPU only · no API key

## Quick start

From this repository, with [uv](https://docs.astral.sh/uv/) installed:

```sh
uv sync --extra dev
uv run python examples/demo.py
```

The demo checks 99 outputs, 100 outputs where a duplicate hides a missing ID, and 100 distinct expected IDs: **FAIL → FAIL → PASS**.

To install from a downloaded source checkout without uv:

```sh
python -m pip install .
batch-coverage --help
```

Or build distributable packages with `uv build --no-sources`; the wheel includes the
Skill at `batch_coverage/skill/SKILL.md` inside the installed package.

For a JSONL batch:

```sh
uv run batch-coverage --workspace examples freeze \
  --kind jsonl --input inputs.jsonl --output outputs.jsonl \
  --require-field summary
```

Keep the generated `batch_id` UUID. Use it for later scans in the same workspace:

```sh
uv run batch-coverage --workspace examples audit BATCH_ID
uv run batch-coverage --workspace examples pending BATCH_ID
uv run batch-coverage --workspace examples status BATCH_ID
```

Replace `BATCH_ID` with the returned UUID. `pending` gives IDs to repair; inspect the audit too, because unexpected or malformed rows may need attention outside those IDs. `pending` and `status` scan the current files rather than returning a cached verdict.

## Input and output contracts

| Mode | Input | Expected output |
| --- | --- | --- |
| `jsonl` | JSON objects, one per line, each with a unique nonempty string ID | Exactly one JSON object per input ID, with required fields present |
| `files` | Files selected from an input directory | One JSON object file per selected input, at its mapped output path |

JSONL uses `id` by default. Set `--id-field record_id` to use another key. Repeat `--require-field` to require multiple top-level fields. These are presence checks: a present value may be `null`. They do not assess the meaning or quality of the values.

JSON must use UTF-8-compatible strings, unique property names, and finite numbers.
Unknown MCP arguments are rejected rather than silently ignored.

For files, the defaults are `--input-glob '**/*'` and `--output-template '{relative}.json'`. An input `report.pdf` therefore expects `report.pdf.json`; `nested/report.pdf` expects `nested/report.pdf.json`.

```sh
uv run batch-coverage --workspace /absolute/path/to/data freeze \
  --kind files --input documents --output results \
  --input-glob '**/*.pdf' --output-template '{relative}.json' \
  --require-field summary
```

Use a dedicated output directory: unexpected output files fail the audit. Paths are resolved within `--workspace`, which defaults to the current directory.

Each freeze stores an immutable input snapshot and SHA-256 fingerprints in local SQLite metadata. The commands read inputs and outputs; they do not generate, rewrite, or remove those files. If input content or the selected input set changes, the original batch reports `INPUT_CHANGED`. Freeze a new batch only when that change belongs to the user's intended scope.

## Audit outcomes

| Outcome | Meaning | `audit` exit code |
| --- | --- | --- |
| `PASS` | Frozen inputs are unchanged and every expected output passes the declared checks | `0` |
| `FAIL` | Coverage or output checks failed | `1` |
| `INPUT_CHANGED` | The current inputs differ from the frozen snapshot | `1` |
| Usage error | Invalid command or arguments | `2` |

`PASS` is a coverage result. It does not prove factual accuracy, semantic correctness, or that an existing output was produced by this run. Keep the user's content acceptance criteria and review them separately.

Wait for all writers to finish before auditing. A filesystem scan is not an atomic snapshot of concurrently changing outputs.

## Use as an MCP server

Start the server with an absolute data workspace:

```sh
uv run batch-coverage --workspace /absolute/path/to/data serve
```

Add an entry to your MCP host's configuration, replacing both paths:

```json
{
  "mcpServers": {
    "batch-coverage": {
      "command": "uv",
      "args": [
        "--directory", "/absolute/path/to/batch-coverage",
        "run", "batch-coverage",
        "--workspace", "/absolute/path/to/data",
        "serve"
      ]
    }
  }
}
```

| Tool | Purpose |
| --- | --- |
| `freeze_inputs` | Capture the actual input set and output requirements; return a batch ID |
| `audit` | Check the frozen batch against current inputs and outputs |
| `pending` | Return item IDs that still need repair, after a fresh scan |
| `status` | Return a fresh batch status |

The MCP server checks calls made through these tools. It cannot force an agent to call them or intercept another tool's work. CLI and MCP checks run locally without model inference or API credentials; dependency installation needs network access unless already cached.

## Bundled Skill

[skills/batch-coverage/SKILL.md](skills/batch-coverage/SKILL.md) tells an agent to freeze before processing, resume pending IDs, and inspect the final audit without weakening the original scope or content requirements. It is bundled with this repository; nothing is installed into global skill or MCP configuration automatically. Load or install it using your host's supported mechanism.

## Related work

- [Pavo](https://github.com/eidos-agi/pavo) checks recording-batch artifact coverage and verifies recorded manifests. It also includes audio processing and human review workflows.
- [spec-kit-verify-tasks](https://github.com/datastone-inc/spec-kit-verify-tasks) checks whether tasks marked complete have implementation evidence.

`batch-coverage` focuses on frozen file/JSONL input sets and their mapped outputs. Coverage checking is an established idea; this project provides a small local CLI/MCP implementation for that scope.

## Development

```sh
uv sync --extra dev
uv run --locked ruff check .
uv run --locked pytest
uv build --no-sources
```

The GitHub workflow runs the checks and demo on Python 3.11 and 3.14, then builds
and checks the installed wheel entry point. See [CHANGELOG.md](CHANGELOG.md) for
release changes.

Licensed under [MIT](LICENSE).