Skip to main content
Glama

Holdout Governance

Fail-closed evidence manifests for financial AI research and AI-generated outputs.

PyPI version Python CI License: MIT Glama score

holdout-governance records one research run or AI-generated output as a small JSON manifest:

  • what evidence artifacts were used;

  • the latest allowed timestamp for the decision;

  • which checks passed;

  • whether AI was used, and which prompt version;

  • whether a person approved the run; and

  • that the result stays research-only.

It is a local validation tool. It does not fetch market data, call a model, place orders, or give investment advice.

Philosophy

Part of the Holdout toolchain — the governance layer. The name comes from the holdout set and the holdout juror: the thing you do not touch early, and the person who does not go along until the evidence is in.

The product philosophy is:

  • evidence before assertion;

  • default deny when evidence is missing;

  • policy as data, not hard-coded judgment;

  • AI may propose, but humans still approve release.

That means one manifest, one verdict, before anything ships.

Related MCP server: sigmodx-mcp

Product flow

This package is the wrapper above the other Holdout tools. The main flow is:

data -> adjust -> timing -> backtest -> falsify -> review -> publish

holdout-governance sits across that flow as the final release gate. It records what was checked, what was attached, what remains missing, and whether the result is approved for release.

Quick start

python -m pip install -e . pytest
python examples/demo.py

Validate a manifest:

gov validate --manifest examples/ai-research-manifest.json
gov report --manifest examples/ai-research-manifest.json

Check ledger health (fail-closed: bad json / duplicate ids / hash-chain breaks / out-of-order timestamps):

gov health --ledger examples/gov-demo/ledger/ledger.jsonl

Real end-to-end demo (verified 2026-09-02)

examples/gov-demo/ is a complete, runnable chain: it executes the real imm / padj / lf / fl binaries through gov check, then shows the hash-chain catching a tampered ledger and blocking the release:

cd examples/gov-demo
./run-demo.sh        # Linux / macOS
.\run-demo.ps1       # Windows PowerShell

What you should see:

1. GREEN PATH   gov check -> decision: release (exit 0)   # 4 gates, real tools
2. TAMPER       append a fake event to ledger.jsonl
3. RED PATH     gov check -> decision: block (exit 2)     # hash chain detected it
4. RESTORE      gov check -> decision: release (exit 0)   # restored

The demo directory contains the fixture data (A-share bars, adjustment actions, a factor pipeline, a pre-registered claim) plus the generated policy.yml and artifact.json, so the flow is fully reproducible with pip install of the six Holdout tools and gov.

GitHub Action (fail-closed gate in CI)

holdout-labs/holdout-governance is a reusable action that runs gov check on your manifest. When the gate chain blocks (missing or stale evidence), the action step fails — which is the point: nothing ships without evidence.

steps:
  - uses: holdout-labs/holdout-governance@v0.4.2
    with:
      manifest: research/artifact.json     # path to your artifact.json
      # policy: research/policy.yml        # optional; defaults to beside the manifest
      # ref: v0.4.2                        # install ref (default: main)

Notes:

  • Install is pip install git+https://github.com/holdout-labs/holdout-governance.git@<ref>; pin ref to a release tag for reproducibility (default: main).

  • The gate needs its evidence tools on PATH (imm / padj / lf / fl) to ever pass; without them it fails closed by design. The tools live in the sibling repos: falsification-ledger, pit-adjuster, factor-qc, lookahead-free, ashare-data-immunity.

  • Exit codes: 0 = release, 1 = review needed, 2 = block.

Contract (v0.2, frozen)

The governance contract lives in schema/ and is locked by tests:

gov check — scenario 1 (done, M1)

AI-generated research conclusions must pass data + timing + evidence gates before they can ship:

# scaffold a research-conclusion project
gov init --dir research/ --name momentum-oos-review
# point gate-inputs.json at your data (imm / padj / lf / fl commands),
# then run the gate chain and decide:
gov check --manifest research/artifact.json
#   exit 0 = release, 1 = review_needed, 2 = block
# artifact.json is written back with decision, missing and gate evidence;
# raw tool outputs are persisted under research/reports/ (sha256-referenced)

gov report --manifest research/artifact.json   # human-readable

gov attach (done)

Attach evidence to an artifact before checking — the agent workflow:

gov attach --manifest research/artifact.json \
  --gate data_integrity --status pass --tool imm --report-ref sha256:...
gov attach --manifest research/artifact.json --attachment sources=docs/sources.md
gov attach --manifest research/artifact.json --declaration contains_returns=true
gov attach --manifest research/artifact.json --review approved --reviewer research-owner

Attaching evidence resets decision to pending — a decision is only as good as the evidence it was computed from, so any evidence change invalidates it until the next gov check. The same operation is exposed to agents as the gov_attach MCP tool.

Stable evidence fingerprints (done)

report_ref is a sha256: of the gate's tool output. Some tools stamp their output with run-time fields (imm audit emits checked_at / audit_date), so re-running the same check on the same data would change the fingerprint and dirty every artifact.json diff. Gate specs can declare those fields:

{
  "data_integrity": {
    "cmd": ["imm", "audit", "--watchlist", "watchlist.json", "--history-root", "history", "--audit-root", "audit"],
    "volatile_keys": ["audit_date", "checked_at"]
  }
}

volatile_keys are stripped (deep) from the JSON before hashing, with keys sorted for a canonical form. The raw tool output is still persisted as the gate report and the real run time stays in the gate entry's run_at — nothing is lost, only the noise stops changing the fingerprint. Non-JSON output keeps its byte-exact hash.

The acceptance suite (tests/test_m1_scenario1.py) runs 10 seeded-defect samples (survivorship ×3, look-ahead ×3, adjustment drift ×2, missing evidence ×2) against the real imm / lf / padj binaries — all 10 are blocked, zero false passes — plus a clean control that must release.

Scenarios 2 & 3 (done, M2)

  • strategy_advice — must carry backtest evidence: backtest_report and robustness_report attachments, plus the statistical_quality gate (real qc run). A backtest whose n_trials is not declared is a refusal, not a failure: qc refuses to judge → review_needed. A real overfitting blocker (DSR/PBO/haircut/MinTRL) → block. Acceptance suite: tests/test_m2_scenario23.py.

  • public_copy — must carry sources; when the copy declares return figures (declarations.contains_returns), limitations becomes required (conditional attachment, expressed in policy.yml, not code). A passing gov report prints the attachments and serves as the publication note.

Contract extension (frozen): artifact.declarations (boolean flags) and policy conditional_attachments (when/require).

Release integration (done, M3)

  • CI.github/workflows/ci.yml: pytest matrix (3.11/3.12) + a fail-closed smoke (scaffold → gov check must exit 2 without evidence).

  • Reusable action.github/actions/gov-check: composite action that runs gov check on any artifact in your workflows.

  • pre-commit hook.pre-commit-hooks.yaml: gov check --manifest on every artifact.json; a block refuses the commit. Wire it with:

    # .pre-commit-config.yaml
    repos:
      - repo: https://github.com/holdout-labs/holdout-governance
        rev: v0.4.0
        hooks:
          - id: gov-check
  • Agent interface — two ways to call gov from code/agents:

    • gov api --port 8000 — stdlib HTTP JSON API: GET /health, POST /check / /report / /init (no extra dependencies).

    • gov mcp — MCP stdio server (pip install 'holdout-governance[mcp]') exposing gov_check, gov_report, gov_init, gov_attach tools for Claude / Cursor / any MCP client.

Fit

Use this package as the wrapper above the existing Holdout tools:

Need

Existing tool

Data quality and snapshots

ashare-data-immunity

Point-in-time price meaning

pit-adjuster

Timing and future-data leakage

lookahead-free

Backtest quality

factor-qc

Claims and evidence trail

falsification-ledger

Past mistakes and reminders

lesson-book

This package records that the checks passed. It does not authorize execution.

Development

python -m pip install -e .[test] pytest
python -m pytest

Available Tools

4 tools
gov_attachA

Attach gate evidence, attachments, declarations or a human review to an artifact (decision resets to pending — run gov_check afterwards). Record gate + status + tool + report_ref after a gate tool produced evidence, or review + reviewer after a human reviewed. Attachments/declarations are name=value strings; repeated calls merge, never drop existing entries. Do NOT claim a gate passed when no tool ran — gov_check trusts the journaled status. Returns the updated gates, attachments, declarations and review.

ParametersJSON Schema
NameRequiredDescriptionDefault
gateNogate_id to record together with status, e.g. data_integrity or temporal_integrity.
toolNoName of the tool that produced the evidence, e.g. imm, padj, lf, fl, qc.
reviewNoHuman review outcome: approved or not_recorded.
statusNoGate outcome: pass, fail, warn or not_run.
manifestYesPath to artifact.json (v0.2) that will receive the evidence.
reviewerNoName of the human reviewer (auditable declaration, not authentication).
attachmentNoAttachment as name=value, e.g. sources=docs/sources.md. Repeat by calling again.
report_refNoEvidence reference for the report file, e.g. sha256:<hex> or a relative path.
declarationNoDeclaration as name=true|false, e.g. contains_returns=true.

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden and delivers: it discloses the side effect ('decision resets to pending'), the required follow-up ('run gov_check afterwards'), merge semantics ('repeated calls merge, never drop existing entries'), the trust model ('gov_check trusts the journaled status'), and the return payload. This exceeds what any annotation set typically covers.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Five sentences, zero filler, and the core purpose is front-loaded. Every sentence carries distinct information: what it does, the two usage modes, merge behavior, the trust warning, and the return value. This is dense but optimally organized for an agent scanning top-down.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 9-parameter tool with two invocation modes and an output schema, the description is complete: it covers the correct parameter combinations, the side effect, the follow-up step, and the key failure mode to avoid. The output schema handles return-value detail, so the description's brief 'Returns the updated gates...' statement is sufficient. No critical information for correct invocation is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real value beyond the schema: it explains that attachments/declarations are name=value strings, and — critically — defines the two valid parameter groupings (gate+status+tool+report_ref vs review+reviewer) that the flattened schema cannot express. The per-parameter definitions remain in the schema, which is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'Attach gate evidence, attachments, declarations or a human review to an artifact.' This is a concrete mutation action with clear objects, and it distinguishes the tool from siblings gov_init (creation), gov_check (verification), and gov_report (reporting). An agent can identify what gov_attach does without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use conditions: 'Record gate + status + tool + report_ref after a gate tool produced evidence, or review + reviewer after a human reviewed.' It also provides a when-not warning ('Do NOT claim a gate passed when no tool ran') and names gov_check as the required follow-up. However, it never contrasts with gov_init or gov_report, so routing among those siblings is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

gov_checkA

Run the full holdout gate chain on a research artifact and return the verdict (release / review_needed / block) as JSON. Executes every required gate of the artifact kind with the real tools (fail-closed: missing tool, crash or timeout records not_run and blocks) and writes the result back. Use as the final step before publishing any AI-assisted research claim — after gov_init created the project and gov_attach recorded the evidence. Do NOT call it before evidence is attached: missing required gates produce review_needed or block by design. Returns decision, exit_code, missing, gates and policy_ref_ok; treat exit_code 2 (block) as do-not-publish.

ParametersJSON Schema
NameRequiredDescriptionDefault
kindNoOptional kind override, e.g. research_conclusion. Only use when the manifest's declared kind is wrong; otherwise leave unset.
policyNoOptional path to policy.yml. Defaults to policy.yml next to the manifest.
manifestYesPath to artifact.json (holdout v0.2 manifest) that records the research run.
gate_inputsNoOptional path to gate-inputs.json. Defaults to gate-inputs.json next to the manifest.

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden and meets it well. It discloses fail-closed behavior ('missing tool, crash or timeout records not_run and blocks'), the side effect of writing the result back, and the operational meaning of exit_code 2 as do-not-publish. This gives the agent a clear picture of safety and side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is four dense sentences with no filler: purpose and verdicts come first, then usage constraints, then return payload and error handling. Every sentence adds operational value, and critical information is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a gate-checking tool with an output schema, the description covers return fields (decision, exit_code, missing, gates, policy_ref_ok), failure semantics, prerequisites, and the do-not-publish signal. Combined with the fully described schema and output schema, an agent has everything needed to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the input schema already documents all four parameters including defaults and when kind override should be used. The description reinforces the manifest and evidence-sequencing context but does not add parameter-level detail beyond the schema, so the baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: 'Run the full holdout gate chain on a research artifact' and lists the exact verdict values (release / review_needed / block). Its lifecycle framing ('final step before publishing') distinguishes it from the related gov_init and gov_attach tools, so an agent can identify what it does without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit usage timing ('Use as the final step before publishing'), prerequisite ordering ('after gov_init created the project and gov_attach recorded the evidence'), and a clear don't-call condition ('Do NOT call it before evidence is attached'). It does not explicitly name gov_report as the alternative for reporting, so it stops just short of fully contrasting all siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

gov_initA

Scaffold a holdout governance project in a directory (created if missing): writes policy.yml and gate-inputs.json only if absent, and creates a new artifact.json with decision=pending. Never overwrites an existing artifact.json — if present it returns an error and writes nothing. Use once at the start of a research run, then attach evidence with gov_attach and decide with gov_check. Returns the created files and the policy_ref SHA.

ParametersJSON Schema
NameRequiredDescriptionDefault
kindNoArtifact kind, which selects the required gates: research_conclusion, strategy_advice, public_copy or code.research_conclusion
nameNoOptional human-readable artifact id. Defaults to a timestamped id.
directoryYesDirectory to scaffold into. Created if missing (like mkdir -p).

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and does so well: it discloses that files are created, conditionally written only if absent, artifact.json is never overwritten, and an error is returned with no writes on conflict. It also states the return value (created files and policy_ref SHA).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but every sentence earns its place: what is written, the no-overwrite guarantee, the usage workflow, and the return value. It is front-loaded with the primary action and files.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity, an output schema, and full schema param coverage, the description provides enough context for correct invocation: setup workflow, safety behavior, failure condition, and return payload. Nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents kind, name, and directory. The description adds little beyond the schema's existing notes, though it does reinforce that directory is created if missing. This matches the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('scaffold'), resource ('holdout governance project'), and exactly what is written (policy.yml, gate-inputs.json, artifact.json). It also distinguishes itself from siblings by referencing gov_attach and gov_check in the stated workflow.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says to use it once at the start of a research run and then use gov_attach and gov_check, which gives an agent clear sequencing guidance. It also states the error condition where the tool should not be used (if artifact.json already exists).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

gov_reportA

Read-only governance assessment of a research artifact, returned as JSON: current decision state, missing required gates, pass/fail per recorded gate, and policy reference check — WITHOUT running any gate or changing any file. Use to inspect an artifact before attaching evidence or to understand why a previous check did not release. Prefer gov_check for fresh gate execution; gov_report never mutates and never spawns tools.

ParametersJSON Schema
NameRequiredDescriptionDefault
policyNoOptional path to policy.yml. Defaults to policy.yml next to the manifest.
manifestYesPath to artifact.json (v0.2) or a v1 manifest to assess.

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description fully carries the behavioral disclosure burden. It clearly states the tool is read-only, runs no gates, changes no files, and never spawns tools. This gives an agent accurate expectations about side effects and safety.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact yet information-dense. Every sentence earns its place: the first defines behavior and output, the second gives use cases, and the third names the sibling alternative and reinforces non-mutation. Key constraints are front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is complete for a read-only inspection tool. It covers purpose, output shape, usage timing, alternatives, and side-effect guarantees. An output schema exists, so return-value details need not be repeated, and the sibling context makes the tool's role in the workflow clear.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents both parameters (manifest path and optional policy path). The description adds no additional parameter-level detail beyond what the schema provides, so the baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: a read-only governance assessment of a research artifact, returned as JSON. It enumerates the exact contents (decision state, missing gates, pass/fail per gate, policy reference check) and clearly distinguishes itself from gov_check by emphasizing it does not run gates.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use guidance: inspect an artifact before attaching evidence or understand why a previous check did not release. It also names the alternative (gov_check) and states the preference for fresh gate execution, plus exclusions (never mutates, never spawns tools).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

A4.7/5.0
Disambiguation5/5

Each tool has a distinct lifecycle role: initialize, attach evidence, execute governance checks, and read-only inspection. There is no overlap or ambiguity between the four tools.

Naming Consistency5/5

All tool names follow a clear gov_<verb> pattern with consistent snake_case naming. The prefix clearly groups them under the same domain while the verb indicates the action.

Tool Count5/5

Four tools is a tight, well-scoped set that covers the governance workflow without redundancy. Each tool earns its place and the count feels appropriate for the server's purpose.

Completeness5/5

The tool set covers the full lifecycle: scaffold the project, attach evidence and reviews, view state without side effects, and run the authoritative gate check. No critical operation appears missing for the stated governance workflow.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    D
    maintenance
    Audit infrastructure for AI agents to log consequential decisions (invoice, GL, anomaly) and verify attestations via MCP tools.
    6
    MIT
  • A
    license
    A
    quality
    D
    maintenance
    Tamper-evident audit logging for AI decisions. Three tools (record_decision, verify_decision, list_decisions) write to a regulator-grade ledger built on AWS S3 Object Lock with 7-year retention. Designed for EU AI Act Article 12 and FCA SS1/23 evidence requirements. Try zero-config: npx audit-ledger-mcp boots in sandbox mode against a public hosted tenant.
    3
    66
    1
    Apache 2.0
  • F
    license
    A
    quality
    D
    maintenance
    Governance circuit-breaker MCP server that enables AI agents to request risk-based decisions, approve or deny actions, and finalize outcomes with full audit receipts.
    4

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/holdout-labs/holdout-governance'

If you have feedback or need assistance with the MCP directory API, please join our Discord server