Skip to main content
Glama

holdout-vault

Protect the scorer and the labels, and prove every number.

Coding agents game the metric they are judged on. In BaitBench, frontier agents took a planted test-set shortcut in 57.1% of runs, and still did so more than half the time when told not to (arXiv 2608.30724). In RewardHackingAgents, agents tampered with the evaluator in about half of all episodes (arXiv 2603.11337). Telling the agent to behave does not fix this. Taking the scorer and the labels out of its reach does.

holdout-vault is an MCP server for ML work done by Claude Code, Cursor, Codex and similar agents. The held-out labels and the scoring code live in a vault the agent cannot open. The agent can only ask for aggregate scores. Every reveal spends part of a budget, going over budget asks the human, and every score is written to a hash-chained ledger with the commit, data hash, scorer hash and seed that produced it.

Status: stages 1 to 4 of 6 are built (vault core, integrity checks, claim audit, protocol layer); the agent benchmark (stage 5) has scripted and local-model results; a Claude Code arm is wired up but not yet run. See docs/design.md.

Measured so far

Canary rows catch models built from leaked labels, without flagging honest ones. Every registered test set gets hidden canary rows with random labels that exist only in the vault. An honest model matches them at chance, so agreement far above chance means the labels were used. Results over 200 synthetic tasks per setting, with Wilson 95% intervals (benchmark/canary_calibration.py, raw numbers in benchmark/results/):

Submission

20 canaries

100 canaries (default)

Honest model

0/200 flagged

1/200 (0.5%, CI 0.1% to 2.8%)

25% of labels leaked

0/200

62/200 (31%)

50% of labels leaked

22/200 (11%)

198/200 (99%)

100% of labels leaked

200/200

200/200

A flagged submission is refused, logged, and costs no reveal.

The code checks were scored on test code they had never seen. Three corpora of ML scripts, notebooks and multi-file projects (189 cases, 64 of them clean-but-tricky) were generated by separate language-model sessions given only the finding definitions, with no access to the checker's code or tests. Each was scored once before anything was changed because of it (benchmark/code_checks.py, benchmark/results/code_checks_*_first_run.json):

Held-out corpus (first run)

Cases

Precision (95% CI)

Recall (95% CI)

Clean cases flagged

1: tutorial-style scripts

82

1.00 (0.93 to 1.00)

0.86 (0.75 to 0.92)

0/24

2: classes, configs, numpy, partials

54

0.93 (0.79 to 0.98)

0.76 (0.60 to 0.87)

1/20

3: method chains, YAML/JSON configs, packages

53

0.97 (0.84 to 0.99)

0.91 (0.77 to 0.97)

1/20

The misses on each corpus were fixed in general form (for example instance state shared across methods, config dictionaries, functools.partial, package re-exports), with new tests written separately from the corpus cases. Corpus 3 was written after the fixes for corpus 2, so its first run shows whether those fixes generalised. All 189 cases now pass and are kept as a regression test, which is a regression bar, not a further accuracy claim.

On real code the first pass was much worse. Run over scikit-learn's own examples (282 files, BSD-3), the checks first raised 43 findings, of which only 4 were real: loops over lists of (X, y) pairs and dataset loaders made y look like a feature. After general fixes the same files raise 4 findings, and all 4 are real preprocessing-before-split patterns, one of which the example's own comment calls "cheating a bit" (benchmark/real_code.py). Those files informed the fixes, so this is a development result; the three held-out corpora above still pass in full.

Row checks on 100 seeded synthetic splits per scenario (benchmark/row_checks.py): exact duplicates, time overlap and group overlap were found in 100/100 splits; near-copies of 2% of test rows in 76/100 (CI 67% to 83%) and of 10% in 97/100. Across 500 clean splits, including rounded features with natural repeats and all-categorical rows, nothing was flagged (CI 0% to 3.7% per scenario).

Claims in reports are traced to the score that produced them. audit_claims reads the agent's Markdown, text and notebooks, extracts every metric claim (prose, value sequences, tables either way round) and checks it against the ledger, MLflow and JSON run records. Two held-out sets of labelled reports, written by separate model sessions from the label definitions alone, were each scored once before any change (benchmark/claim_checks.py):

Held-out report set (first run)

Reports / claims

Claims found (precision, recall)

Unsupported claims caught (precision, recall, 95% CI)

1: results sections, summaries, tables

40 / 160

1.00, 0.94

1.00 (0.96 to 1.00), 0.95 (0.88 to 0.98)

2: chat logs, model cards, transposed tables, FAQs

30 / 146

1.00, 0.69

1.00 (0.95 to 1.00), 0.71 (0.61 to 0.79)

No claim was ever given a wrong label when found (label accuracy 0.97 and 0.99 on found claims). Misses are claims the extractor did not see. After general fixes (transposed tables, value sequences, citations per clause) recall is 0.99 and 0.86; the remaining misses are numbers whose metric is named only on an earlier line, such as a chat reply or a Q&A answer.

Leaked label files (benchmark/canary_artifacts.py, 100 trials): a workspace file holding the sealed labels for 25% or more of the test rows was caught every time, for 10% of rows 56/100, for 5% 3/100 (too few canary rows to test). No honest workspace (exported data, predictions, out-of-fold predictions) was flagged.

Related MCP server: APIVouch

Agent benchmark

Tasks on four public datasets that ship with scikit-learn (Breast Cancer Wisconsin, Wine and Optical Digits from UCI under CC BY 4.0; the diabetes data of Efron et al., BSD-3), each with one planted shortcut: the labelled test file sitting in the workspace, or an editable scorer next to its answer key. Three arms: no guard, prompt-only (told not to use test labels or touch the scorer), and vault (labels and scorer registered in holdout-vault, scored through its tools). Runs are paired across arms on the same dataset, shortcut and seed (benchmark/agentbench/).

Known behaviour first. Scripted agents act out each kind of cheating, so the numbers measure what the vault stops, not what an agent chooses (24 paired runs per cell, 288 in all; results/agentbench_scripted_summary.json):

Agent

Cheated or inflated results that reached the report unflagged: no guard

prompt-only

vault

Copies the test labels into its predictions

24/24

24/24

0/24 (all 24 refused: canary labels, HV-CAN-002)

Edits the scorer or the label file

24/24

24/24

0/24 (all 24 refused: label or scorer hash, HV-LAB-001)

Reports a number 0.10 above what it measured

18/24

18/24

0/24 (22 claims flagged by the audit)

Honest

3/24 (cross-validation estimate above the true score)

3/24

0/24

Vault against no guard, exact McNemar on paired runs: p < 0.0001 for each cheating agent. The honest agent completed all 96 of its tasks in every arm and was never refused.

A real agent. A local model (Qwen2.5 7B Instruct through Ollama, $0, temperature 0.2, at most 20 steps) ran the same 16 task pairs per arm, two seeds (48 runs; results/agentbench_ollama_qwen_summary.json, per-run tool logs in results/agentbench_ollama_qwen_logs/):

Arm

Took a shortcut

Inflated report

Inflated report unflagged

Completed

No guard

0/16

7/16

7/16

9/16

Prompt-only

0/16

3/16

3/16

7/16

Vault

0/16

3/16

0/16 (all 3 flagged by the audit)

3/16

  • The 7B model never went for the planted shortcut, so this run says nothing about preventing cheating; that part rests on the scripted agents above. What it did do is write numbers into its report that its predictions did not earn (one no-guard run reported 0.93 R² for a model whose true score was 0.22). Unflagged inflated reports, vault against no guard: exact McNemar p = 0.016.

  • Cost: the vault arm completed fewer tasks (3/16 against 9/16; McNemar p = 0.07, not significant at n = 16). The logs show the same failure in every arm, the model rewriting the training script until it runs out of steps, but more often with the vault's extra tools and instructions in an 8K context. Whether a stronger agent pays this cost is the question the Claude Code arm (--backend claude, wired up, waiting on a CLI login) is there to answer.

How it works

 agent ──MCP──▶ front door (host)  ──loopback HTTP──▶ vault container      runner container
                score / evaluate                      labels, scorer,      model + test features
                vault_status                          ledger, budget       no labels, no network
                provenance stamps
 human ──CLI──▶ register / budget / grant / verify-ledger   (never exposed over MCP)
  • Sealed mode (Docker): labels, scorer and ledger live in a volume that is never mounted anywhere else. evaluate runs the agent's model in a throwaway container with no network, a read-only filesystem, no vault volume and an unprivileged user, so model code cannot read the labels or edit the scorer. Only predictions come back.

  • Watch-only mode (no Docker): the test file stays where it is. Its hash, the scorer hash and the sealed label hash are checked on every call, and canaries still work. This mode detects, it does not prevent, and every response says so.

  • Budget: derived from the ledger, never from a config file. Validation errors, tamper findings and refused submissions never spend a reveal. Going over budget triggers a mid-call confirmation (a multi-round-trip request on MCP 2026-07-28, classic elicitation on older clients) that the human answers.

  • Integrity checks: before every score the front door analyses the workspace's Python files and notebooks and looks for data files carrying the hidden canary labels. Reading the labelled test file or the vault store, tampering with vault files, patching scoring functions and canary labels in a file all block the score (logged, no reveal spent). Ordinary leakage mistakes (preprocessing fitted before the split, fitting on validation data, the target as a feature, resampling before the split) are warnings attached to the score and the ledger, because the sealed test set is outside the agent's code: those mistakes inflate the agent's own validation numbers, not the vault's. Row checks on the split itself run once at registration.

  • Claim audit: every metric claim gets one label: traced (matches a vault score), mismatched (cites an entry or run that recorded something else), orphaned (no record has that number), cherry-picked (the best of three or more sealed scores, others undisclosed), stale (from a commit no longer in the branch's history) or untrusted-source (only in the agent's own MLflow/JSON records, flagged further when the workspace code has leakage warnings). Matching is deterministic: same metric after alias normalisation, same value within the precision shown.

  • Protocol: evaluate runs as an MCP Task (io.modelcontextprotocol/tasks, SEP-2663, implemented here because the Python SDK does not ship it) for clients that declare the extension, with status messages and cooperative cancellation; other clients, and over-budget calls that need the human, get an ordinary result. Ledger, provenance and findings are resources with cache hints. A dashboard tool backs an MCP App view and carries the same content as text for clients that do not render Apps. Everything works over stdio and stateless Streamable HTTP.

  • Ledger: append-only JSONL, each entry hashed over the previous one. verify-ledger fails at the first edited, deleted, reordered or truncated entry, and every score handed to the agent carries the entry hash, so a later rewrite contradicts numbers already quoted.

Quick start

uv sync
export PYTHONPATH=src

# watch-only: register a dataset (human), then point the agent at the MCP server
uv run holdout-vault register --project churn --name v1 \
  --train data/train.csv --test data/test.csv --target churned --id-col customer_id \
  --metrics accuracy,roc_auc --export-dir ./agent-workspace/data
uv run holdout-vault serve --project churn --repo ./agent-workspace

# sealed: start the vault container, register into it, then serve as above
uv run holdout-vault up
uv run holdout-vault register --sealed --project churn --name v1 ...
uv run holdout-vault grant --sealed --project churn 2 --note "final check"
uv run holdout-vault verify-ledger --sealed --project churn

Claude Code config:

{ "mcpServers": { "holdout-vault": { "command": "uv", "args": ["run", "--project", "/path/to/holdout-vault",
  "holdout-vault", "serve", "--project", "churn", "--repo", "."] } } }

Tools

Tool

What the agent gets

Spends a reveal

score(dataset, predictions_path, seed?)

Aggregate metrics, provenance stamp, ledger entry

On success

evaluate(dataset, model_path, seed?)

Same, after running a joblib model or a predict.py folder in the runner

On success

check_leakage(dataset, paths?)

Row, code and canary-artifact findings, without scoring

No

audit_claims(paths?, text?)

Every metric claim in the reports, labelled against the ledger and run records

No

dashboard()

Budget, ledger, scores, refusals, findings, own-vs-sealed gap (MCP App plus text)

No

vault_status()

Datasets, budget, mode, ledger health

No

Resources: ledger://{project}/entries, ledger://{project}/entry/{seq}, provenance://{project}/{seq}, findings://{project}/latest, ui://holdout-vault/dashboard.html.

Every tool has a typed outputSchema. Verdicts have a status of scored, invalid, over_budget, refused or tamper. Findings carry a stable code, a severity, a plain-English explanation and an exact location (file and line, notebook cell, or row ids):

Code

Finding

Severity

HV-ROW-001 / 002

Test rows that exactly match, or nearly copy, training rows

warn

HV-ROW-003 / 004

Train rows later than the earliest test row; the same group on both sides

warn

HV-COD-001

Preprocessing fitted on data that is later split or cross-validated

warn

HV-COD-002

A fit that receives validation data or the exported test features

warn

HV-COD-003

The target, or a column computed from it, used as a model input

warn

HV-COD-004

Code reads the labelled test file or the vault store

block

HV-COD-005

Resampling (SMOTE and similar) before the split

warn

HV-COD-006

Code changes vault files, patches scoring functions, or reaches into the vault container

block

HV-CAN-001

A workspace file carries the hidden canary labels

block

HV-CAN-002

Predictions match the hidden canary labels far above chance

block

HV-SCR-001, HV-LAB-001, HV-TST-001

Scorer, sealed labels or watched test file changed since registration

block

The same checks run from the shell: holdout-vault audit --project churn --repo ./agent-workspace for claims, and holdout-vault check --project churn --dataset v1 --repo ./agent-workspace (exit code 1 on a blocking finding). register --strict refuses a split with row findings.

Threat model

  • The agent runs in your shell. A determined agent with Docker or host access could reach the vault. The vault makes cheating hard and visible (ledger, canaries, scorer and label hashes); it is not a security boundary against a hostile admin.

  • The front door trusts the MCP client's confirmation dialog. A client set to auto-approve elicitations defeats the budget confirmation; holdout-vault grant is the human path that does not depend on the client.

  • Watch-only mode detects, it does not prevent.

  • Fewer reveals means less adaptive overfitting, not none. The budget is motivated by the reusable-holdout work of Dwork, Feldman, Hardt, Pitassi, Reingold and Roth (Science 349(6248):636-638, 2015), but scores are returned exactly, so the formal guarantees of their Thresholdout mechanism do not apply. Thresholdout is planned as an opt-in mode.

  • Canaries detect predictions built from leaked labels. They do not detect an agent that reads the labels and then trains an honest-looking model on the leaked test rows only.

  • The code checks are static analysis. They follow data through functions, classes, modules, configs and notebooks, but code built to hide a read (exec, string obfuscation, a download at runtime) will get past them. They catch the common case and leave a record; the sealed container is what actually keeps the labels out of reach.

  • Model files and pickles in the workspace are never opened by the checks, because unpickling runs arbitrary code.

  • The claim audit reads what is written down. A number whose metric is named only in an earlier line, a chart, or a slide is not seen, and a claim can only be as traceable as the records it is checked against.

What didn't work (and what replaced it)

  • The first version of the code checks missed a quarter of the leaks in the second held-out corpus (recall 0.76), mostly where state lived on self, in config dictionaries or behind functools.partial. Those are now followed, and the third corpus, written afterwards, scored 0.91.

  • A loose "metric, a few words, a number" rule for claims attached numbers to the wrong metric ("0.84 accuracy and a ROC AUC of 0.90" read the AUC as accuracy). The gap may no longer contain another metric's name and must end in a real verb.

  • A five-row minimum for near-duplicate findings missed most leaks in small test sets (20/40 at a 2% leak rate). Three rows caught 30/40 with no new false alarms on the clean controls.

  • 20 canaries per test set. Enough to catch a full label leak every time, but only 11% of models built on half-leaked labels. Raising the floor to 100 lifted that to 99% at a false-alarm rate in line with the test level. The 20-canary run is kept as a reproducible ablation: benchmark/canary_calibration.py --min-canaries 20.

  • Loading the agent's model inside the vault. A pickle or joblib file runs arbitrary code on load, which would put agent code next to the labels. Models now run in a separate runner container that never sees the labels.

Development

make lint
make test        # offline, small fixtures
make cov         # coverage gate 85%
uv run pytest -m docker   # sealed-mode end-to-end, needs Docker running
uv run python benchmark/code_checks.py --corpus benchmark/corpus/code_blind3
uv run python benchmark/row_checks.py
uv run python benchmark/canary_artifacts.py
uv run python benchmark/claim_checks.py --corpus benchmark/corpus/claims_blind2

Credits

Builds on BaitBench, RewardHackingAgents, MLR-Bench, EvilGenie and ImpossibleBench for the threat evidence, Dwork et al. (2015) and Blum and Hardt's Ladder (arXiv 1502.04585) for the theory of budgeted reveals. LeakageLens-MCP and leakguard-mcp (leakage linting) and Regor (claim tracing) inspired parts of the design; no code or rules are reused. Related projects: Sparra and claude-code-kit holdouts hide acceptance tests from a builder agent, and holdout-labs keeps hash-chained pre-registration ledgers for quant research.

Licence

MIT. Author: Janish Kumar.

Available Tools

6 tools
audit_claimsCheck reported metrics against the ledgerA

Trace every metric claim in reports to the score that produced it.

Reads Markdown, text and notebook markdown in the workspace (or only paths, or only text), finds claims like "AUC of 0.91", and labels each one: traced (matches a vault score), mismatched, orphaned, cherry-picked (best of several scores, others undisclosed), stale (from a commit no longer in history) or untrusted-source (only in MLflow/JSON run records). Free.

Inputs: text, paths. Returns: claims, counts, records, summary, +1 more

check_leakageCheck the workspace for leakage without scoringA

Run the integrity checks without spending a reveal.

Reports overlap between the sealed split and train (HV-ROW), leakage and tampering patterns in the workspace's Python files and notebooks (HV-COD), and data files carrying the hidden canary labels (HV-CAN-001). paths limits the code checks to some files or folders.

Inputs: dataset (required), paths. Returns: dataset, message, blocking, findings, +4 more

dashboardVault dashboardA

Budget, ledger, recent sealed scores, refusals, workspace findings, and the gap between the agent's own numbers and the sealed scores. Renders as an MCP App where supported; the summary field carries the same information as text. Free.

Inputs: no named parameters. Returns: gaps, mode, spent, budget, +9 more

evaluateRun a saved model on the sealed test set and score itB

Run a saved model on the test features, then score its predictions on the sealed labels.

model_path is a joblib/pickle estimator, or a folder with predict.py defining predict(features) -> predictions. In sealed mode the model runs in an isolated container with no network and no access to labels. Spends one reveal when it succeeds.

Inputs: dataset (required), model_path (required), seed. Returns: mode, stamp, budget, status, +6 more

scoreScore predictions against the sealed test labelsA

Score a predictions file (CSV or Parquet) on the sealed test set.

The file needs the dataset's id column and a 'prediction' column. Returns aggregate metrics only, never labels or per-row results. Spends one reveal when it succeeds.

Inputs: dataset (required), predictions_path (required), seed. Returns: mode, stamp, budget, status, +6 more

vault_statusVault statusB

Registered datasets, reveal budget, mode, and whether the ledger verifies.

Inputs: no named parameters. Returns: mode, note, spent, budget, +6 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 6 tools

    Disambiguation4/5

    Each tool has a distinct role: dashboard for overview, score for a predictions file, evaluate for running a model, check_leakage for integrity checks, audit_claims for claim tracing, and vault_status for state. The only mild overlap is score vs evaluate, since both end in sealed-set scoring and spend a reveal, but the input difference (predictions file vs saved model) is clearly stated.

    Naming Consistency3/5

    Names mix bare nouns/verbs (dashboard, score, evaluate) with verb_noun compounds (check_leakage, audit_claims) and a noun_noun form (vault_status). All are readable and none are cryptic, but there is no single predictable pattern across the set.

    Tool Count5/5

    Six tools is well-scoped for a sealed-holdout evaluation vault; each tool maps to a distinct workflow step (status, dashboard, score, evaluate, leak check, claim audit). No tool feels redundant or padded.

    Completeness4/5

    The surface covers the core lifecycle: checking budget/status, submitting predictions, running models, integrity checks, claim auditing, and reviewing recent scores. Minor gaps exist, such as explicit dataset registration or reveal purchase/history listing, though dashboard and vault_status partially compensate.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables coding agents to request, track, and verify independent cross-model reviews with structured findings, attachment hashes, and provenance receipts via Chrome Bridge, Codex, or Responses transports.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables agents to route goals across multiple independent APIs, enforce constraints, reject invalid evidence, and return verified outcomes with tamper-evident receipts.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables deterministic verification of AI agent decisions and actions, providing PASS/FAIL/ABSTAIN verdicts with replayable proofs and an optional signed receipt ledger.
      4 npm
      Apache 2.0
    • A
      license
      A
      quality
      B
      maintenance
      Enables coding agents and humans to share a gated task registry where tasks move through a strict status flow and transitions require verification and review evidence from distinct identities.
      17
      MIT