batch-coverage

中文说明

Check that an agent processed every item in a batch. Freeze the real inputs before work, audit the outputs, and resume from the IDs that still need repair.

An agent can report “all 100 items are done” after producing 97 outputs. batch-coverage checks the frozen input set against the current output set and reports missing, duplicate, unexpected, or malformed results. Changed inputs invalidate the batch instead of silently changing what “complete” means.

Python 3.11+ · CLI and stdio MCP · SQLite · CPU only · no API key

Quick start

From this repository, with uv installed:

uv sync --extra dev
uv run python examples/demo.py

The demo checks 99 outputs, 100 outputs where a duplicate hides a missing ID, and 100 distinct expected IDs: FAIL → FAIL → PASS.

To install from a downloaded source checkout without uv:

python -m pip install .
batch-coverage --help

Or build distributable packages with uv build --no-sources; the wheel includes the Skill at batch_coverage/skill/SKILL.md inside the installed package.

For a JSONL batch:

uv run batch-coverage --workspace examples freeze \
  --kind jsonl --input inputs.jsonl --output outputs.jsonl \
  --require-field summary

Keep the generated batch_id UUID. Use it for later scans in the same workspace:

uv run batch-coverage --workspace examples audit BATCH_ID
uv run batch-coverage --workspace examples pending BATCH_ID
uv run batch-coverage --workspace examples status BATCH_ID

Replace BATCH_ID with the returned UUID. pending gives IDs to repair; inspect the audit too, because unexpected or malformed rows may need attention outside those IDs. pending and status scan the current files rather than returning a cached verdict.

Related MCP server: trusted-transcription

Input and output contracts

JSONL uses id by default. Set --id-field record_id to use another key. Repeat --require-field to require multiple top-level fields. These are presence checks: a present value may be null. They do not assess the meaning or quality of the values.

JSON must use UTF-8-compatible strings, unique property names, and finite numbers. Unknown MCP arguments are rejected rather than silently ignored.

For files, the defaults are --input-glob '**/*' and --output-template '{relative}.json'. An input report.pdf therefore expects report.pdf.json; nested/report.pdf expects nested/report.pdf.json.

uv run batch-coverage --workspace /absolute/path/to/data freeze \
  --kind files --input documents --output results \
  --input-glob '**/*.pdf' --output-template '{relative}.json' \
  --require-field summary

Use a dedicated output directory: unexpected output files fail the audit. Paths are resolved within --workspace, which defaults to the current directory.

Each freeze stores an immutable input snapshot and SHA-256 fingerprints in local SQLite metadata. The commands read inputs and outputs; they do not generate, rewrite, or remove those files. If input content or the selected input set changes, the original batch reports INPUT_CHANGED. Freeze a new batch only when that change belongs to the user's intended scope.

Audit outcomes

PASS is a coverage result. It does not prove factual accuracy, semantic correctness, or that an existing output was produced by this run. Keep the user's content acceptance criteria and review them separately.

Wait for all writers to finish before auditing. A filesystem scan is not an atomic snapshot of concurrently changing outputs.

Use as an MCP server

Start the server with an absolute data workspace:

uv run batch-coverage --workspace /absolute/path/to/data serve

Add an entry to your MCP host's configuration, replacing both paths:

{
  "mcpServers": {
    "batch-coverage": {
      "command": "uv",
      "args": [
        "--directory", "/absolute/path/to/batch-coverage",
        "run", "batch-coverage",
        "--workspace", "/absolute/path/to/data",
        "serve"
      ]
    }
  }
}

The MCP server checks calls made through these tools. It cannot force an agent to call them or intercept another tool's work. CLI and MCP checks run locally without model inference or API credentials; dependency installation needs network access unless already cached.

Bundled Skill

skills/batch-coverage/SKILL.md tells an agent to freeze before processing, resume pending IDs, and inspect the final audit without weakening the original scope or content requirements. It is bundled with this repository; nothing is installed into global skill or MCP configuration automatically. Load or install it using your host's supported mechanism.

Related work

• Pavo checks recording-batch artifact coverage and verifies recorded manifests. It also includes audio processing and human review workflows.

• spec-kit-verify-tasks checks whether tasks marked complete have implementation evidence.

batch-coverage focuses on frozen file/JSONL input sets and their mapped outputs. Coverage checking is an established idea; this project provides a small local CLI/MCP implementation for that scope.

Development

uv sync --extra dev
uv run --locked ruff check .
uv run --locked pytest
uv build --no-sources

The GitHub workflow runs the checks and demo on Python 3.11 and 3.14, then builds and checks the installed wheel entry point. See CHANGELOG.md for release changes.

Licensed under MIT.