holdout-governance
OfficialThis server exposes a holdout-governance API for validating, assessing, scaffolding, and attaching evidence to research artifacts.
gov_check: Run the holdout gate chain on a manifest and get a release / review_needed / block decision.
gov_report: Generate a read-only governance assessment of a research artifact.
gov_init: Scaffold a holdout policy + artifact project in a specified directory.
gov_attach: Attach gate evidence, attachments, declarations, or review approvals to a manifest (resetting its decision to pending).
Holdout Governance
Fail-closed evidence manifests for financial AI research and AI-generated outputs.
holdout-governance records one research run or AI-generated output as a small JSON manifest:
what evidence artifacts were used;
the latest allowed timestamp for the decision;
which checks passed;
whether AI was used, and which prompt version;
whether a person approved the run; and
that the result stays research-only.
It is a local validation tool. It does not fetch market data, call a model, place orders, or give investment advice.
Philosophy
Part of the Holdout toolchain — the governance layer. The name comes from the holdout set and the holdout juror: the thing you do not touch early, and the person who does not go along until the evidence is in.
The product philosophy is:
evidence before assertion;
default deny when evidence is missing;
policy as data, not hard-coded judgment;
AI may propose, but humans still approve release.
That means one manifest, one verdict, before anything ships.
Related MCP server: sigmodx-mcp
Product flow
This package is the wrapper above the other Holdout tools. The main flow is:
data -> adjust -> timing -> backtest -> falsify -> review -> publish
holdout-governance sits across that flow as the final release gate. It
records what was checked, what was attached, what remains missing, and whether
the result is approved for release.
Quick start
python -m pip install -e . pytest
python examples/demo.pyValidate a manifest:
gov validate --manifest examples/ai-research-manifest.json
gov report --manifest examples/ai-research-manifest.jsonCheck ledger health (fail-closed: bad json / duplicate ids / hash-chain breaks / out-of-order timestamps):
gov health --ledger examples/gov-demo/ledger/ledger.jsonlReal end-to-end demo (verified 2026-09-02)
examples/gov-demo/ is a complete, runnable chain: it executes the real
imm / padj / lf / fl binaries through gov check, then shows the
hash-chain catching a tampered ledger and blocking the release:
cd examples/gov-demo
./run-demo.sh # Linux / macOS
.\run-demo.ps1 # Windows PowerShellWhat you should see:
1. GREEN PATH gov check -> decision: release (exit 0) # 4 gates, real tools
2. TAMPER append a fake event to ledger.jsonl
3. RED PATH gov check -> decision: block (exit 2) # hash chain detected it
4. RESTORE gov check -> decision: release (exit 0) # restoredThe demo directory contains the fixture data (A-share bars, adjustment
actions, a factor pipeline, a pre-registered claim) plus the generated
policy.yml and artifact.json, so the flow is fully reproducible with
pip install of the six Holdout tools and gov.
GitHub Action (fail-closed gate in CI)
holdout-labs/holdout-governance is a reusable action that runs gov check
on your manifest. When the gate chain blocks (missing or stale evidence), the
action step fails — which is the point: nothing ships without evidence.
steps:
- uses: holdout-labs/holdout-governance@v0.4.2
with:
manifest: research/artifact.json # path to your artifact.json
# policy: research/policy.yml # optional; defaults to beside the manifest
# ref: v0.4.2 # install ref (default: main)Notes:
Install is
pip install git+https://github.com/holdout-labs/holdout-governance.git@<ref>; pinrefto a release tag for reproducibility (default:main).The gate needs its evidence tools on
PATH(imm/padj/lf/fl) to ever pass; without them it fails closed by design. The tools live in the sibling repos:falsification-ledger,pit-adjuster,factor-qc,lookahead-free,ashare-data-immunity.Exit codes:
0= release,1= review needed,2= block.
Contract (v0.2, frozen)
The governance contract lives in schema/ and is locked by tests:
schema/artifact.schema.json—holdout.artifact.v0.2schema/policy.schema.json+schema/policy.example.yml—holdout.policy.v0.1examples/artifact.example.json— a conformingresearch_conclusionartifactdocs/migration-v1-to-v0.2.md— upgrade path from the v1 manifest
gov check — scenario 1 (done, M1)
AI-generated research conclusions must pass data + timing + evidence gates before they can ship:
# scaffold a research-conclusion project
gov init --dir research/ --name momentum-oos-review
# point gate-inputs.json at your data (imm / padj / lf / fl commands),
# then run the gate chain and decide:
gov check --manifest research/artifact.json
# exit 0 = release, 1 = review_needed, 2 = block
# artifact.json is written back with decision, missing and gate evidence;
# raw tool outputs are persisted under research/reports/ (sha256-referenced)
gov report --manifest research/artifact.json # human-readablegov attach (done)
Attach evidence to an artifact before checking — the agent workflow:
gov attach --manifest research/artifact.json \
--gate data_integrity --status pass --tool imm --report-ref sha256:...
gov attach --manifest research/artifact.json --attachment sources=docs/sources.md
gov attach --manifest research/artifact.json --declaration contains_returns=true
gov attach --manifest research/artifact.json --review approved --reviewer research-ownerAttaching evidence resets decision to pending — a decision is only as
good as the evidence it was computed from, so any evidence change invalidates
it until the next gov check. The same operation is exposed to agents as the
gov_attach MCP tool.
Stable evidence fingerprints (done)
report_ref is a sha256: of the gate's tool output. Some tools stamp their
output with run-time fields (imm audit emits checked_at / audit_date),
so re-running the same check on the same data would change the fingerprint
and dirty every artifact.json diff. Gate specs can declare those fields:
{
"data_integrity": {
"cmd": ["imm", "audit", "--watchlist", "watchlist.json", "--history-root", "history", "--audit-root", "audit"],
"volatile_keys": ["audit_date", "checked_at"]
}
}volatile_keys are stripped (deep) from the JSON before hashing, with keys
sorted for a canonical form. The raw tool output is still persisted as
the gate report and the real run time stays in the gate entry's run_at —
nothing is lost, only the noise stops changing the fingerprint. Non-JSON
output keeps its byte-exact hash.
The acceptance suite (tests/test_m1_scenario1.py) runs 10 seeded-defect
samples (survivorship ×3, look-ahead ×3, adjustment drift ×2, missing
evidence ×2) against the real imm / lf / padj binaries — all 10 are
blocked, zero false passes — plus a clean control that must release.
Scenarios 2 & 3 (done, M2)
strategy_advice — must carry backtest evidence:
backtest_reportandrobustness_reportattachments, plus thestatistical_qualitygate (realqcrun). A backtest whosen_trialsis not declared is a refusal, not a failure:qcrefuses to judge →review_needed. A real overfitting blocker (DSR/PBO/haircut/MinTRL) →block. Acceptance suite:tests/test_m2_scenario23.py.public_copy — must carry
sources; when the copy declares return figures (declarations.contains_returns),limitationsbecomes required (conditional attachment, expressed inpolicy.yml, not code). A passinggov reportprints the attachments and serves as the publication note.
Contract extension (frozen): artifact.declarations (boolean flags) and
policy conditional_attachments (when/require).
Release integration (done, M3)
CI —
.github/workflows/ci.yml: pytest matrix (3.11/3.12) + a fail-closed smoke (scaffold →gov checkmust exit 2 without evidence).Reusable action —
.github/actions/gov-check: composite action that runsgov checkon any artifact in your workflows.pre-commit hook —
.pre-commit-hooks.yaml:gov check --manifeston everyartifact.json; a block refuses the commit. Wire it with:# .pre-commit-config.yaml repos: - repo: https://github.com/holdout-labs/holdout-governance rev: v0.4.0 hooks: - id: gov-checkAgent interface — two ways to call gov from code/agents:
gov api --port 8000— stdlib HTTP JSON API:GET /health,POST /check//report//init(no extra dependencies).gov mcp— MCP stdio server (pip install 'holdout-governance[mcp]') exposinggov_check,gov_report,gov_init,gov_attachtools for Claude / Cursor / any MCP client.
Fit
Use this package as the wrapper above the existing Holdout tools:
Need | Existing tool |
Data quality and snapshots |
|
Point-in-time price meaning |
|
Timing and future-data leakage |
|
Backtest quality |
|
Claims and evidence trail |
|
Past mistakes and reminders |
|
This package records that the checks passed. It does not authorize execution.
Development
python -m pip install -e .[test] pytest
python -m pytestAvailable Tools
4 toolsgov_attachA
Attach gate evidence, attachments, declarations or a human review to an artifact (decision resets to pending — run gov_check afterwards). Record gate + status + tool + report_ref after a gate tool produced evidence, or review + reviewer after a human reviewed. Attachments/declarations are name=value strings; repeated calls merge, never drop existing entries. Do NOT claim a gate passed when no tool ran — gov_check trusts the journaled status. Returns the updated gates, attachments, declarations and review.
| Name | Required | Description | Default |
|---|---|---|---|
| gate | No | gate_id to record together with status, e.g. data_integrity or temporal_integrity. | |
| tool | No | Name of the tool that produced the evidence, e.g. imm, padj, lf, fl, qc. | |
| review | No | Human review outcome: approved or not_recorded. | |
| status | No | Gate outcome: pass, fail, warn or not_run. | |
| manifest | Yes | Path to artifact.json (v0.2) that will receive the evidence. | |
| reviewer | No | Name of the human reviewer (auditable declaration, not authentication). | |
| attachment | No | Attachment as name=value, e.g. sources=docs/sources.md. Repeat by calling again. | |
| report_ref | No | Evidence reference for the report file, e.g. sha256:<hex> or a relative path. | |
| declaration | No | Declaration as name=true|false, e.g. contains_returns=true. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral burden and delivers: it discloses the side effect ('decision resets to pending'), the required follow-up ('run gov_check afterwards'), merge semantics ('repeated calls merge, never drop existing entries'), the trust model ('gov_check trusts the journaled status'), and the return payload. This exceeds what any annotation set typically covers.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Five sentences, zero filler, and the core purpose is front-loaded. Every sentence carries distinct information: what it does, the two usage modes, merge behavior, the trust warning, and the return value. This is dense but optimally organized for an agent scanning top-down.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 9-parameter tool with two invocation modes and an output schema, the description is complete: it covers the correct parameter combinations, the side effect, the follow-up step, and the key failure mode to avoid. The output schema handles return-value detail, so the description's brief 'Returns the updated gates...' statement is sufficient. No critical information for correct invocation is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds real value beyond the schema: it explains that attachments/declarations are name=value strings, and — critically — defines the two valid parameter groupings (gate+status+tool+report_ref vs review+reviewer) that the flattened schema cannot express. The per-parameter definitions remain in the schema, which is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'Attach gate evidence, attachments, declarations or a human review to an artifact.' This is a concrete mutation action with clear objects, and it distinguishes the tool from siblings gov_init (creation), gov_check (verification), and gov_report (reporting). An agent can identify what gov_attach does without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use conditions: 'Record gate + status + tool + report_ref after a gate tool produced evidence, or review + reviewer after a human reviewed.' It also provides a when-not warning ('Do NOT claim a gate passed when no tool ran') and names gov_check as the required follow-up. However, it never contrasts with gov_init or gov_report, so routing among those siblings is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
gov_checkA
Run the full holdout gate chain on a research artifact and return the verdict (release / review_needed / block) as JSON. Executes every required gate of the artifact kind with the real tools (fail-closed: missing tool, crash or timeout records not_run and blocks) and writes the result back. Use as the final step before publishing any AI-assisted research claim — after gov_init created the project and gov_attach recorded the evidence. Do NOT call it before evidence is attached: missing required gates produce review_needed or block by design. Returns decision, exit_code, missing, gates and policy_ref_ok; treat exit_code 2 (block) as do-not-publish.
| Name | Required | Description | Default |
|---|---|---|---|
| kind | No | Optional kind override, e.g. research_conclusion. Only use when the manifest's declared kind is wrong; otherwise leave unset. | |
| policy | No | Optional path to policy.yml. Defaults to policy.yml next to the manifest. | |
| manifest | Yes | Path to artifact.json (holdout v0.2 manifest) that records the research run. | |
| gate_inputs | No | Optional path to gate-inputs.json. Defaults to gate-inputs.json next to the manifest. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral burden and meets it well. It discloses fail-closed behavior ('missing tool, crash or timeout records not_run and blocks'), the side effect of writing the result back, and the operational meaning of exit_code 2 as do-not-publish. This gives the agent a clear picture of safety and side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is four dense sentences with no filler: purpose and verdicts come first, then usage constraints, then return payload and error handling. Every sentence adds operational value, and critical information is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a gate-checking tool with an output schema, the description covers return fields (decision, exit_code, missing, gates, policy_ref_ok), failure semantics, prerequisites, and the do-not-publish signal. Combined with the fully described schema and output schema, an agent has everything needed to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the input schema already documents all four parameters including defaults and when kind override should be used. The description reinforces the manifest and evidence-sequencing context but does not add parameter-level detail beyond the schema, so the baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Run the full holdout gate chain on a research artifact' and lists the exact verdict values (release / review_needed / block). Its lifecycle framing ('final step before publishing') distinguishes it from the related gov_init and gov_attach tools, so an agent can identify what it does without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit usage timing ('Use as the final step before publishing'), prerequisite ordering ('after gov_init created the project and gov_attach recorded the evidence'), and a clear don't-call condition ('Do NOT call it before evidence is attached'). It does not explicitly name gov_report as the alternative for reporting, so it stops just short of fully contrasting all siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
gov_initA
Scaffold a holdout governance project in a directory (created if missing): writes policy.yml and gate-inputs.json only if absent, and creates a new artifact.json with decision=pending. Never overwrites an existing artifact.json — if present it returns an error and writes nothing. Use once at the start of a research run, then attach evidence with gov_attach and decide with gov_check. Returns the created files and the policy_ref SHA.
| Name | Required | Description | Default |
|---|---|---|---|
| kind | No | Artifact kind, which selects the required gates: research_conclusion, strategy_advice, public_copy or code. | research_conclusion |
| name | No | Optional human-readable artifact id. Defaults to a timestamped id. | |
| directory | Yes | Directory to scaffold into. Created if missing (like mkdir -p). |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden and does so well: it discloses that files are created, conditionally written only if absent, artifact.json is never overwritten, and an error is returned with no writes on conflict. It also states the return value (created files and policy_ref SHA).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence earns its place: what is written, the no-overwrite guarantee, the usage workflow, and the return value. It is front-loaded with the primary action and files.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity, an output schema, and full schema param coverage, the description provides enough context for correct invocation: setup workflow, safety behavior, failure condition, and return payload. Nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents kind, name, and directory. The description adds little beyond the schema's existing notes, though it does reinforce that directory is created if missing. This matches the baseline for high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('scaffold'), resource ('holdout governance project'), and exactly what is written (policy.yml, gate-inputs.json, artifact.json). It also distinguishes itself from siblings by referencing gov_attach and gov_check in the stated workflow.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says to use it once at the start of a research run and then use gov_attach and gov_check, which gives an agent clear sequencing guidance. It also states the error condition where the tool should not be used (if artifact.json already exists).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
gov_reportA
Read-only governance assessment of a research artifact, returned as JSON: current decision state, missing required gates, pass/fail per recorded gate, and policy reference check — WITHOUT running any gate or changing any file. Use to inspect an artifact before attaching evidence or to understand why a previous check did not release. Prefer gov_check for fresh gate execution; gov_report never mutates and never spawns tools.
| Name | Required | Description | Default |
|---|---|---|---|
| policy | No | Optional path to policy.yml. Defaults to policy.yml next to the manifest. | |
| manifest | Yes | Path to artifact.json (v0.2) or a v1 manifest to assess. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully carries the behavioral disclosure burden. It clearly states the tool is read-only, runs no gates, changes no files, and never spawns tools. This gives an agent accurate expectations about side effects and safety.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact yet information-dense. Every sentence earns its place: the first defines behavior and output, the second gives use cases, and the third names the sibling alternative and reinforces non-mutation. Key constraints are front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is complete for a read-only inspection tool. It covers purpose, output shape, usage timing, alternatives, and side-effect guarantees. An output schema exists, so return-value details need not be repeated, and the sibling context makes the tool's role in the workflow clear.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters (manifest path and optional policy path). The description adds no additional parameter-level detail beyond what the schema provides, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: a read-only governance assessment of a research artifact, returned as JSON. It enumerates the exact contents (decision state, missing gates, pass/fail per gate, policy reference check) and clearly distinguishes itself from gov_check by emphasizing it does not run gates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use guidance: inspect an artifact before attaching evidence or understand why a previous check did not release. It also names the alternative (gov_check) and states the preference for fresh gate execution, plus exclusions (never mutates, never spawns tools).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
TDQS
Each tool has a distinct lifecycle role: initialize, attach evidence, execute governance checks, and read-only inspection. There is no overlap or ambiguity between the four tools.
All tool names follow a clear gov_<verb> pattern with consistent snake_case naming. The prefix clearly groups them under the same domain while the verb indicates the action.
Four tools is a tight, well-scoped set that covers the governance workflow without redundancy. Each tool earns its place and the count feels appropriate for the server's purpose.
The tool set covers the full lifecycle: scaffold the project, attach evidence and reviews, view state without side effects, and run the authoritative gate check. No critical operation appears missing for the stated governance workflow.
Maintenance
Related MCP Connectors
Runtime AI governance: decision gates, human approval, hash-chained audit, compliance mapping.
Static MCP manifest and tool-policy security preflight with signed input-redacted receipts
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
AI governance MCP server for EU AI Act compliance and jurisdiction verification
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceTamper-proof audit trail for AI decisions. 6 tools to record, verify, and export cryptographic proof chains via MCP.1Apache 2.0
- AlicenseAqualityDmaintenanceAudit infrastructure for AI agents to log consequential decisions (invoice, GL, anomaly) and verify attestations via MCP tools.6MIT
- AlicenseAqualityDmaintenanceTamper-evident audit logging for AI decisions. Three tools (record_decision, verify_decision, list_decisions) write to a regulator-grade ledger built on AWS S3 Object Lock with 7-year retention. Designed for EU AI Act Article 12 and FCA SS1/23 evidence requirements. Try zero-config: npx audit-ledger-mcp boots in sandbox mode against a public hosted tenant.3661Apache 2.0

@vaibot/mcp-serverofficial
FlicenseAqualityDmaintenanceGovernance circuit-breaker MCP server that enables AI agents to request risk-based decisions, approve or deny actions, and finalize outcomes with full audit receipts.4
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/holdout-labs/holdout-governance'
If you have feedback or need assistance with the MCP directory API, please join our Discord server