holdout-vault
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
holdout-vault holds the sealed test labels and the scorer for this project.
Train on train.csv. Then either write a predictions CSV (the id column, 'prediction', plus 'proba' or
'proba_' for probability metrics) and call score, or save the model (joblib file, or a folder
with predict.py defining predict(features)) and call evaluate.
Each successful score spends one reveal from a limited budget and is recorded in a tamper-evident
ledger. Report only numbers returned by this server, and quote the ledger entry they came from.
Before each score the server checks the workspace code and data files for leakage; check_leakage
runs the same checks for free. Before reporting results, call audit_claims on the report to confirm
every number traces to a ledger entry.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| dashboardA | Budget, ledger, recent sealed scores, refusals, workspace findings, and the gap between the
agent's own numbers and the sealed scores. Renders as an MCP App where supported; the |
| scoreA | Score a predictions file (CSV or Parquet) on the sealed test set. The file needs the dataset's id column and a 'prediction' column. Returns aggregate metrics only, never labels or per-row results. Spends one reveal when it succeeds. |
| evaluateB | Run a saved model on the test features, then score its predictions on the sealed labels. model_path is a joblib/pickle estimator, or a folder with predict.py defining predict(features) -> predictions. In sealed mode the model runs in an isolated container with no network and no access to labels. Spends one reveal when it succeeds. |
| check_leakageA | Run the integrity checks without spending a reveal. Reports overlap between the sealed split and train (HV-ROW), leakage and tampering patterns in
the workspace's Python files and notebooks (HV-COD), and data files carrying the hidden canary
labels (HV-CAN-001). |
| audit_claimsA | Trace every metric claim in reports to the score that produced it. Reads Markdown, text and notebook markdown in the workspace (or only |
| vault_statusB | Registered datasets, reveal budget, mode, and whether the ledger verifies. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
| holdout-vault dashboard |
TDQS
Scored across 6 tools
Each tool has a distinct role: dashboard for overview, score for a predictions file, evaluate for running a model, check_leakage for integrity checks, audit_claims for claim tracing, and vault_status for state. The only mild overlap is score vs evaluate, since both end in sealed-set scoring and spend a reveal, but the input difference (predictions file vs saved model) is clearly stated.
Names mix bare nouns/verbs (dashboard, score, evaluate) with verb_noun compounds (check_leakage, audit_claims) and a noun_noun form (vault_status). All are readable and none are cryptic, but there is no single predictable pattern across the set.
Six tools is well-scoped for a sealed-holdout evaluation vault; each tool maps to a distinct workflow step (status, dashboard, score, evaluate, leak check, claim audit). No tool feels redundant or padded.
The surface covers the core lifecycle: checking budget/status, submitting predictions, running models, integrity checks, claim auditing, and reviewing recent scores. Minor gaps exist, such as explicit dataset registration or reveal purchase/history listing, though dashboard and vault_status partially compensate.