Skip to main content
Glama

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault

No arguments

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

holdout-vault holds the sealed test labels and the scorer for this project. Train on train.csv. Then either write a predictions CSV (the id column, 'prediction', plus 'proba' or 'proba_' for probability metrics) and call score, or save the model (joblib file, or a folder with predict.py defining predict(features)) and call evaluate. Each successful score spends one reveal from a limited budget and is recorded in a tamper-evident ledger. Report only numbers returned by this server, and quote the ledger entry they came from. Before each score the server checks the workspace code and data files for leakage; check_leakage runs the same checks for free. Before reporting results, call audit_claims on the report to confirm every number traces to a ledger entry.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": false
}
prompts
{
  "listChanged": false
}
resources
{
  "subscribe": false,
  "listChanged": false
}

Tools

Functions exposed to the LLM to take actions

NameDescription
dashboardA

Budget, ledger, recent sealed scores, refusals, workspace findings, and the gap between the agent's own numbers and the sealed scores. Renders as an MCP App where supported; the summary field carries the same information as text. Free.

scoreA

Score a predictions file (CSV or Parquet) on the sealed test set.

The file needs the dataset's id column and a 'prediction' column. Returns aggregate metrics only, never labels or per-row results. Spends one reveal when it succeeds.

evaluateB

Run a saved model on the test features, then score its predictions on the sealed labels.

model_path is a joblib/pickle estimator, or a folder with predict.py defining predict(features) -> predictions. In sealed mode the model runs in an isolated container with no network and no access to labels. Spends one reveal when it succeeds.

check_leakageA

Run the integrity checks without spending a reveal.

Reports overlap between the sealed split and train (HV-ROW), leakage and tampering patterns in the workspace's Python files and notebooks (HV-COD), and data files carrying the hidden canary labels (HV-CAN-001). paths limits the code checks to some files or folders.

audit_claimsA

Trace every metric claim in reports to the score that produced it.

Reads Markdown, text and notebook markdown in the workspace (or only paths, or only text), finds claims like "AUC of 0.91", and labels each one: traced (matches a vault score), mismatched, orphaned, cherry-picked (best of several scores, others undisclosed), stale (from a commit no longer in history) or untrusted-source (only in MLflow/JSON run records). Free.

vault_statusB

Registered datasets, reveal budget, mode, and whether the ledger verifies.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription
holdout-vault dashboard

TDQS

A3.5/5.0

Scored across 6 tools

Disambiguation4/5

Each tool has a distinct role: dashboard for overview, score for a predictions file, evaluate for running a model, check_leakage for integrity checks, audit_claims for claim tracing, and vault_status for state. The only mild overlap is score vs evaluate, since both end in sealed-set scoring and spend a reveal, but the input difference (predictions file vs saved model) is clearly stated.

Naming Consistency3/5

Names mix bare nouns/verbs (dashboard, score, evaluate) with verb_noun compounds (check_leakage, audit_claims) and a noun_noun form (vault_status). All are readable and none are cryptic, but there is no single predictable pattern across the set.

Tool Count5/5

Six tools is well-scoped for a sealed-holdout evaluation vault; each tool maps to a distinct workflow step (status, dashboard, score, evaluate, leak check, claim audit). No tool feels redundant or padded.

Completeness4/5

The surface covers the core lifecycle: checking budget/status, submitting predictions, running models, integrity checks, claim auditing, and reviewing recent scores. Minor gaps exist, such as explicit dataset registration or reveal purchase/history listing, though dashboard and vault_status partially compensate.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues