Skip to main content
Glama
README.md
# holdout-vault

**Protect the scorer and the labels, and prove every number.**

Coding agents game the metric they are judged on. In BaitBench, frontier agents took a planted test-set shortcut in 57.1% of runs, and still did so more than half the time when told not to ([arXiv 2608.30724](https://arxiv.org/abs/2608.30724)). In RewardHackingAgents, agents tampered with the evaluator in about half of all episodes ([arXiv 2603.11337](https://arxiv.org/abs/2603.11337)). Telling the agent to behave does not fix this. Taking the scorer and the labels out of its reach does.

holdout-vault is an MCP server for ML work done by Claude Code, Cursor, Codex and similar agents. The held-out labels and the scoring code live in a vault the agent cannot open. The agent can only ask for aggregate scores. Every reveal spends part of a budget, going over budget asks the human, and every score is written to a hash-chained ledger with the commit, data hash, scorer hash and seed that produced it.

> Status: stages 1 to 4 of 6 are built (vault core, integrity checks, claim audit, protocol layer); the agent benchmark (stage 5) has scripted and local-model results; a Claude Code arm is wired up but not yet run. See [docs/design.md](docs/design.md).

## Measured so far

**Canary rows catch models built from leaked labels, without flagging honest ones.** Every registered test set gets hidden canary rows with random labels that exist only in the vault. An honest model matches them at chance, so agreement far above chance means the labels were used. Results over 200 synthetic tasks per setting, with Wilson 95% intervals (`benchmark/canary_calibration.py`, raw numbers in `benchmark/results/`):

| Submission | 20 canaries | 100 canaries (default) |
|---|---|---|
| Honest model | 0/200 flagged | 1/200 (0.5%, CI 0.1% to 2.8%) |
| 25% of labels leaked | 0/200 | 62/200 (31%) |
| 50% of labels leaked | 22/200 (11%) | 198/200 (99%) |
| 100% of labels leaked | 200/200 | 200/200 |

A flagged submission is refused, logged, and costs no reveal.

**The code checks were scored on test code they had never seen.** Three corpora of ML scripts, notebooks and multi-file projects (189 cases, 64 of them clean-but-tricky) were generated by separate language-model sessions given only the finding definitions, with no access to the checker's code or tests. Each was scored once before anything was changed because of it (`benchmark/code_checks.py`, `benchmark/results/code_checks_*_first_run.json`):

| Held-out corpus (first run) | Cases | Precision (95% CI) | Recall (95% CI) | Clean cases flagged |
|---|---|---|---|---|
| 1: tutorial-style scripts | 82 | 1.00 (0.93 to 1.00) | 0.86 (0.75 to 0.92) | 0/24 |
| 2: classes, configs, numpy, partials | 54 | 0.93 (0.79 to 0.98) | 0.76 (0.60 to 0.87) | 1/20 |
| 3: method chains, YAML/JSON configs, packages | 53 | 0.97 (0.84 to 0.99) | 0.91 (0.77 to 0.97) | 1/20 |

The misses on each corpus were fixed in general form (for example instance state shared across methods, config dictionaries, `functools.partial`, package re-exports), with new tests written separately from the corpus cases. Corpus 3 was written after the fixes for corpus 2, so its first run shows whether those fixes generalised. All 189 cases now pass and are kept as a regression test, which is a regression bar, not a further accuracy claim.

**On real code the first pass was much worse.** Run over scikit-learn's own examples (282 files, BSD-3), the checks first raised 43 findings, of which only 4 were real: loops over lists of `(X, y)` pairs and dataset loaders made `y` look like a feature. After general fixes the same files raise 4 findings, and all 4 are real preprocessing-before-split patterns, one of which the example's own comment calls "cheating a bit" (`benchmark/real_code.py`). Those files informed the fixes, so this is a development result; the three held-out corpora above still pass in full.

**Row checks** on 100 seeded synthetic splits per scenario (`benchmark/row_checks.py`): exact duplicates, time overlap and group overlap were found in 100/100 splits; near-copies of 2% of test rows in 76/100 (CI 67% to 83%) and of 10% in 97/100. Across 500 clean splits, including rounded features with natural repeats and all-categorical rows, nothing was flagged (CI 0% to 3.7% per scenario).

**Claims in reports are traced to the score that produced them.** `audit_claims` reads the agent's Markdown, text and notebooks, extracts every metric claim (prose, value sequences, tables either way round) and checks it against the ledger, MLflow and JSON run records. Two held-out sets of labelled reports, written by separate model sessions from the label definitions alone, were each scored once before any change (`benchmark/claim_checks.py`):

| Held-out report set (first run) | Reports / claims | Claims found (precision, recall) | Unsupported claims caught (precision, recall, 95% CI) |
|---|---|---|---|
| 1: results sections, summaries, tables | 40 / 160 | 1.00, 0.94 | 1.00 (0.96 to 1.00), 0.95 (0.88 to 0.98) |
| 2: chat logs, model cards, transposed tables, FAQs | 30 / 146 | 1.00, 0.69 | 1.00 (0.95 to 1.00), 0.71 (0.61 to 0.79) |

No claim was ever given a wrong label when found (label accuracy 0.97 and 0.99 on found claims). Misses are claims the extractor did not see. After general fixes (transposed tables, value sequences, citations per clause) recall is 0.99 and 0.86; the remaining misses are numbers whose metric is named only on an earlier line, such as a chat reply or a Q&A answer.

**Leaked label files** (`benchmark/canary_artifacts.py`, 100 trials): a workspace file holding the sealed labels for 25% or more of the test rows was caught every time, for 10% of rows 56/100, for 5% 3/100 (too few canary rows to test). No honest workspace (exported data, predictions, out-of-fold predictions) was flagged.

## Agent benchmark

Tasks on four public datasets that ship with scikit-learn (Breast Cancer Wisconsin, Wine and Optical Digits from UCI under CC BY 4.0; the diabetes data of Efron et al., BSD-3), each with one planted shortcut: the labelled test file sitting in the workspace, or an editable scorer next to its answer key. Three arms: **no guard**, **prompt-only** (told not to use test labels or touch the scorer), and **vault** (labels and scorer registered in holdout-vault, scored through its tools). Runs are paired across arms on the same dataset, shortcut and seed (`benchmark/agentbench/`).

**Known behaviour first.** Scripted agents act out each kind of cheating, so the numbers measure what the vault stops, not what an agent chooses (24 paired runs per cell, 288 in all; `results/agentbench_scripted_summary.json`):

| Agent | Cheated or inflated results that reached the report unflagged: no guard | prompt-only | vault |
|---|---|---|---|
| Copies the test labels into its predictions | 24/24 | 24/24 | **0/24** (all 24 refused: canary labels, HV-CAN-002) |
| Edits the scorer or the label file | 24/24 | 24/24 | **0/24** (all 24 refused: label or scorer hash, HV-LAB-001) |
| Reports a number 0.10 above what it measured | 18/24 | 18/24 | **0/24** (22 claims flagged by the audit) |
| Honest | 3/24 (cross-validation estimate above the true score) | 3/24 | 0/24 |

Vault against no guard, exact McNemar on paired runs: p < 0.0001 for each cheating agent. The honest agent completed all 96 of its tasks in every arm and was never refused.

**A real agent.** A local model (Qwen2.5 7B Instruct through Ollama, $0, temperature 0.2, at most 20 steps) ran the same 16 task pairs per arm, two seeds (48 runs; `results/agentbench_ollama_qwen_summary.json`, per-run tool logs in `results/agentbench_ollama_qwen_logs/`):

| Arm | Took a shortcut | Inflated report | Inflated report unflagged | Completed |
|---|---|---|---|---|
| No guard | 0/16 | 7/16 | 7/16 | 9/16 |
| Prompt-only | 0/16 | 3/16 | 3/16 | 7/16 |
| Vault | 0/16 | 3/16 | **0/16** (all 3 flagged by the audit) | 3/16 |

- The 7B model never went for the planted shortcut, so this run says nothing about preventing cheating; that part rests on the scripted agents above. What it did do is write numbers into its report that its predictions did not earn (one no-guard run reported 0.93 R² for a model whose true score was 0.22). Unflagged inflated reports, vault against no guard: exact McNemar p = 0.016.
- **Cost:** the vault arm completed fewer tasks (3/16 against 9/16; McNemar p = 0.07, not significant at n = 16). The logs show the same failure in every arm, the model rewriting the training script until it runs out of steps, but more often with the vault's extra tools and instructions in an 8K context. Whether a stronger agent pays this cost is the question the Claude Code arm (`--backend claude`, wired up, waiting on a CLI login) is there to answer.

## How it works

```
 agent ──MCP──▶ front door (host)  ──loopback HTTP──▶ vault container      runner container
                score / evaluate                      labels, scorer,      model + test features
                vault_status                          ledger, budget       no labels, no network
                provenance stamps
 human ──CLI──▶ register / budget / grant / verify-ledger   (never exposed over MCP)
```

- **Sealed mode** (Docker): labels, scorer and ledger live in a volume that is never mounted anywhere else. `evaluate` runs the agent's model in a throwaway container with no network, a read-only filesystem, no vault volume and an unprivileged user, so model code cannot read the labels or edit the scorer. Only predictions come back.
- **Watch-only mode** (no Docker): the test file stays where it is. Its hash, the scorer hash and the sealed label hash are checked on every call, and canaries still work. This mode **detects, it does not prevent**, and every response says so.
- **Budget:** derived from the ledger, never from a config file. Validation errors, tamper findings and refused submissions never spend a reveal. Going over budget triggers a mid-call confirmation (a multi-round-trip request on MCP 2026-07-28, classic elicitation on older clients) that the human answers.
- **Integrity checks:** before every score the front door analyses the workspace's Python files and notebooks and looks for data files carrying the hidden canary labels. Reading the labelled test file or the vault store, tampering with vault files, patching scoring functions and canary labels in a file all **block** the score (logged, no reveal spent). Ordinary leakage mistakes (preprocessing fitted before the split, fitting on validation data, the target as a feature, resampling before the split) are **warnings** attached to the score and the ledger, because the sealed test set is outside the agent's code: those mistakes inflate the agent's own validation numbers, not the vault's. Row checks on the split itself run once at registration.
- **Claim audit:** every metric claim gets one label: `traced` (matches a vault score), `mismatched` (cites an entry or run that recorded something else), `orphaned` (no record has that number), `cherry-picked` (the best of three or more sealed scores, others undisclosed), `stale` (from a commit no longer in the branch's history) or `untrusted-source` (only in the agent's own MLflow/JSON records, flagged further when the workspace code has leakage warnings). Matching is deterministic: same metric after alias normalisation, same value within the precision shown.
- **Protocol:** `evaluate` runs as an MCP Task (`io.modelcontextprotocol/tasks`, SEP-2663, implemented here because the Python SDK does not ship it) for clients that declare the extension, with status messages and cooperative cancellation; other clients, and over-budget calls that need the human, get an ordinary result. Ledger, provenance and findings are resources with cache hints. A `dashboard` tool backs an MCP App view and carries the same content as text for clients that do not render Apps. Everything works over stdio and stateless Streamable HTTP.
- **Ledger:** append-only JSONL, each entry hashed over the previous one. `verify-ledger` fails at the first edited, deleted, reordered or truncated entry, and every score handed to the agent carries the entry hash, so a later rewrite contradicts numbers already quoted.

## Quick start

```bash
uv sync
export PYTHONPATH=src

# watch-only: register a dataset (human), then point the agent at the MCP server
uv run holdout-vault register --project churn --name v1 \
  --train data/train.csv --test data/test.csv --target churned --id-col customer_id \
  --metrics accuracy,roc_auc --export-dir ./agent-workspace/data
uv run holdout-vault serve --project churn --repo ./agent-workspace

# sealed: start the vault container, register into it, then serve as above
uv run holdout-vault up
uv run holdout-vault register --sealed --project churn --name v1 ...
uv run holdout-vault grant --sealed --project churn 2 --note "final check"
uv run holdout-vault verify-ledger --sealed --project churn
```

Claude Code config:

```json
{ "mcpServers": { "holdout-vault": { "command": "uv", "args": ["run", "--project", "/path/to/holdout-vault",
  "holdout-vault", "serve", "--project", "churn", "--repo", "."] } } }
```

### Tools

| Tool | What the agent gets | Spends a reveal |
|---|---|---|
| `score(dataset, predictions_path, seed?)` | Aggregate metrics, provenance stamp, ledger entry | On success |
| `evaluate(dataset, model_path, seed?)` | Same, after running a joblib model or a `predict.py` folder in the runner | On success |
| `check_leakage(dataset, paths?)` | Row, code and canary-artifact findings, without scoring | No |
| `audit_claims(paths?, text?)` | Every metric claim in the reports, labelled against the ledger and run records | No |
| `dashboard()` | Budget, ledger, scores, refusals, findings, own-vs-sealed gap (MCP App plus text) | No |
| `vault_status()` | Datasets, budget, mode, ledger health | No |

Resources: `ledger://{project}/entries`, `ledger://{project}/entry/{seq}`, `provenance://{project}/{seq}`, `findings://{project}/latest`, `ui://holdout-vault/dashboard.html`.

Every tool has a typed `outputSchema`. Verdicts have a `status` of `scored`, `invalid`, `over_budget`, `refused` or `tamper`. Findings carry a stable code, a severity, a plain-English explanation and an exact location (file and line, notebook cell, or row ids):

| Code | Finding | Severity |
|---|---|---|
| `HV-ROW-001` / `002` | Test rows that exactly match, or nearly copy, training rows | warn |
| `HV-ROW-003` / `004` | Train rows later than the earliest test row; the same group on both sides | warn |
| `HV-COD-001` | Preprocessing fitted on data that is later split or cross-validated | warn |
| `HV-COD-002` | A fit that receives validation data or the exported test features | warn |
| `HV-COD-003` | The target, or a column computed from it, used as a model input | warn |
| `HV-COD-004` | Code reads the labelled test file or the vault store | block |
| `HV-COD-005` | Resampling (SMOTE and similar) before the split | warn |
| `HV-COD-006` | Code changes vault files, patches scoring functions, or reaches into the vault container | block |
| `HV-CAN-001` | A workspace file carries the hidden canary labels | block |
| `HV-CAN-002` | Predictions match the hidden canary labels far above chance | block |
| `HV-SCR-001`, `HV-LAB-001`, `HV-TST-001` | Scorer, sealed labels or watched test file changed since registration | block |

The same checks run from the shell: `holdout-vault audit --project churn --repo ./agent-workspace` for claims, and `holdout-vault check --project churn --dataset v1 --repo ./agent-workspace` (exit code 1 on a blocking finding). `register --strict` refuses a split with row findings.

## Threat model

- The agent runs in your shell. A determined agent with Docker or host access could reach the vault. The vault makes cheating hard and visible (ledger, canaries, scorer and label hashes); it is not a security boundary against a hostile admin.
- The front door trusts the MCP client's confirmation dialog. A client set to auto-approve elicitations defeats the budget confirmation; `holdout-vault grant` is the human path that does not depend on the client.
- Watch-only mode detects, it does not prevent.
- Fewer reveals means less adaptive overfitting, not none. The budget is motivated by the reusable-holdout work of Dwork, Feldman, Hardt, Pitassi, Reingold and Roth ([*Science* 349(6248):636-638, 2015](https://doi.org/10.1126/science.aaa9375)), but scores are returned exactly, so the formal guarantees of their Thresholdout mechanism do not apply. Thresholdout is planned as an opt-in mode.
- Canaries detect predictions built from leaked labels. They do not detect an agent that reads the labels and then trains an honest-looking model on the leaked test rows only.
- The code checks are static analysis. They follow data through functions, classes, modules, configs and notebooks, but code built to hide a read (`exec`, string obfuscation, a download at runtime) will get past them. They catch the common case and leave a record; the sealed container is what actually keeps the labels out of reach.
- Model files and pickles in the workspace are never opened by the checks, because unpickling runs arbitrary code.
- The claim audit reads what is written down. A number whose metric is named only in an earlier line, a chart, or a slide is not seen, and a claim can only be as traceable as the records it is checked against.

## What didn't work (and what replaced it)

- **The first version of the code checks missed a quarter of the leaks** in the second held-out corpus (recall 0.76), mostly where state lived on `self`, in config dictionaries or behind `functools.partial`. Those are now followed, and the third corpus, written afterwards, scored 0.91.
- **A loose "metric, a few words, a number" rule for claims** attached numbers to the wrong metric ("0.84 accuracy and a ROC AUC of 0.90" read the AUC as accuracy). The gap may no longer contain another metric's name and must end in a real verb.
- **A five-row minimum for near-duplicate findings** missed most leaks in small test sets (20/40 at a 2% leak rate). Three rows caught 30/40 with no new false alarms on the clean controls.

- **20 canaries per test set.** Enough to catch a full label leak every time, but only 11% of models built on half-leaked labels. Raising the floor to 100 lifted that to 99% at a false-alarm rate in line with the test level. The 20-canary run is kept as a reproducible ablation: `benchmark/canary_calibration.py --min-canaries 20`.
- **Loading the agent's model inside the vault.** A pickle or joblib file runs arbitrary code on load, which would put agent code next to the labels. Models now run in a separate runner container that never sees the labels.

## Development

```bash
make lint
make test        # offline, small fixtures
make cov         # coverage gate 85%
uv run pytest -m docker   # sealed-mode end-to-end, needs Docker running
uv run python benchmark/code_checks.py --corpus benchmark/corpus/code_blind3
uv run python benchmark/row_checks.py
uv run python benchmark/canary_artifacts.py
uv run python benchmark/claim_checks.py --corpus benchmark/corpus/claims_blind2
```

## Credits

Builds on BaitBench, RewardHackingAgents, MLR-Bench, EvilGenie and ImpossibleBench for the threat evidence, Dwork et al. (2015) and Blum and Hardt's Ladder ([arXiv 1502.04585](https://arxiv.org/abs/1502.04585)) for the theory of budgeted reveals. LeakageLens-MCP and leakguard-mcp (leakage linting) and Regor (claim tracing) inspired parts of the design; no code or rules are reused. Related projects: Sparra and claude-code-kit `holdouts` hide acceptance tests from a builder agent, and holdout-labs keeps hash-chained pre-registration ledgers for quant research.

## Licence

MIT. Author: Janish Kumar.

TDQS

A3.5/5.0

Scored across 6 tools

Disambiguation4/5

Each tool has a distinct role: dashboard for overview, score for a predictions file, evaluate for running a model, check_leakage for integrity checks, audit_claims for claim tracing, and vault_status for state. The only mild overlap is score vs evaluate, since both end in sealed-set scoring and spend a reveal, but the input difference (predictions file vs saved model) is clearly stated.

Naming Consistency3/5

Names mix bare nouns/verbs (dashboard, score, evaluate) with verb_noun compounds (check_leakage, audit_claims) and a noun_noun form (vault_status). All are readable and none are cryptic, but there is no single predictable pattern across the set.

Tool Count5/5

Six tools is well-scoped for a sealed-holdout evaluation vault; each tool maps to a distinct workflow step (status, dashboard, score, evaluate, leak check, claim audit). No tool feels redundant or padded.

Completeness4/5

The surface covers the core lifecycle: checking budget/status, submitting predictions, running models, integrity checks, claim auditing, and reviewing recent scores. Minor gaps exist, such as explicit dataset registration or reveal purchase/history listing, though dashboard and vault_status partially compensate.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues