Skip to main content
Glama
Sign In
holdout-labs

holdout-governance

Official
by holdout-labs
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

Holdout Governance

Fail-closed evidence manifests for AI-assisted quantitative research.

This is not a factor library, a backtest engine, or a trading system. holdout-governance is infrastructure for the moment when a financial AI agent drafts a research conclusion, calls tools, attaches evidence, and asks whether the result can move forward.

PyPI version Python CI License: MIT Glama score

holdout-governance records one research run or AI-generated output as a small JSON manifest:

  • what evidence artifacts were used;

  • the latest allowed timestamp for the decision;

  • which checks passed;

  • whether AI was used, and which prompt version;

  • whether a person approved the run; and

  • that the result stays research-only.

It is a local validation tool. It does not fetch market data, call a model, place orders, or give investment advice.

Agent boundary

holdout-governance is the boundary around an AI-assisted research workflow. An agent may draft a conclusion, run tools, and attach evidence, but the manifest and policy decide whether the result can move forward. Missing evidence blocks by default; warnings can be downgraded only when the policy says so; final publication still belongs to a person or an external release process.

This is the intended contract: AI can help produce research, but it cannot silently skip evidence, erase accountability, or turn a draft into a released decision on its own.

That makes Holdout different from the usual quant stack. The usual stack helps you generate signals, test portfolios, and run execution. Holdout governs the research artifact around that work: what evidence exists, what checks passed, what policy was applied, and where human approval is required.

Related MCP server: sigmodx-mcp

Philosophy

Part of the Holdout toolchain — the governance layer. The name comes from the holdout set and the holdout juror: the thing you do not touch early, and the person who does not go along until the evidence is in.

The product philosophy is:

  • evidence before assertion;

  • default deny when evidence is missing;

  • policy as data, not hard-coded judgment;

  • AI may propose, but humans still approve release.

That means one manifest, one verdict, before anything ships.

Product flow

This package is the wrapper above the other Holdout tools. The main flow is:

data -> adjust -> timing -> backtest -> falsify -> review -> publish

holdout-governance sits across that flow as the final release gate. It records what was checked, what was attached, what remains missing, and whether the result is approved for release.

Quick start

python -m pip install -c tools/suite-constraints.txt ".[suite,research]"
gov doctor
python examples/demo.py

Validate a manifest:

gov validate --manifest examples/ai-research-manifest.json
gov report --manifest examples/ai-research-manifest.json

Check ledger health (fail-closed: bad json / duplicate ids / hash-chain breaks / out-of-order timestamps):

gov health --ledger examples/gov-demo/ledger/ledger.jsonl

Install and diagnose the toolchain

The suite extra installs the six companion tools at exact versions:

python -m pip install "holdout-governance[suite,research]==0.5.0"
gov doctor

From a source checkout, use python -m pip install -c tools/suite-constraints.txt ".[suite,research]". tools/suite-constraints.txt documents the supported Holdout package combination; it does not lock transitive dependencies. The new extra and commands are introduced in version 0.5.0.

gov doctor prints every package version, expected version, actual command path and CLI version. Missing packages, missing commands, unsupported versions and commands from a conflicting Python environment produce exit 1. gov doctor --json provides the same diagnosis for automation; it does not run research or install anything. Activate the Python environment where you installed the tools first.

The local synthetic research workbench uses the separate research extra: run holdout-research after installing the extras above, or use Start-Factor-Research.cmd from the source checkout. holdout-research --check checks dependencies without creating a study or exposing final-test data. The workbench uses public, reproducible synthetic data, not market-calibrated returns. Final evaluation consumes one opportunity before accessing samples; the shared registry rejects already used or overlapping samples across studies. It is a trusted local workflow, not an OS sandbox. See the workbench guide for limits and saved-run behavior. Version 0.4.3 does not include this workbench.

Real end-to-end demo

examples/gov-demo/run_demo.py runs real imm / padj / lf / fl commands through gov check in a disposable copy of the fixtures. All generated manifests, ledger edits and reports stay in that temporary directory. Failure or interruption leaves the source fixtures unchanged.

python examples/gov-demo/run_demo.py        # Windows / Linux / macOS
bash examples/gov-demo/run-demo.sh          # Linux / macOS
pwsh -File examples/gov-demo/run-demo.ps1    # Windows PowerShell

What you should see:

1. GREEN PATH   real gate chain -> release (exit 0)
2. TAMPER       append an unchained synthetic ledger event
3. BLOCK        exit 2 AND evidence_integrity=fail
4. RESTORE      original ledger bytes -> release (exit 0)
5. STRATEGY     real qc gate on positive synthetic returns -> release (exit 0)
6. STRATEGY BLOCK negative synthetic returns -> block (exit 2), qc gate=fail

Exit 1, missing binaries and configuration errors are rejected as demo failures; they cannot stand in for ledger tamper detection. The strategy attachments and returns are synthetic fixtures, not a verified trading strategy or release approval. CI installs this checkout plus six companion packages from fixed commit IDs into a clean environment on Windows and Linux, runs gov doctor, and then runs this chain. Unit tests with substituted processes do not replace that integration check.

GitHub Action (fail-closed gate in CI)

holdout-labs/holdout-governance is a reusable action that runs gov check on your manifest. When the gate chain blocks (missing or stale evidence), the action step fails — which is the point: nothing ships without evidence.

steps:
  - uses: holdout-labs/holdout-governance@v0.5.0
    with:
      manifest: research/artifact.json     # path to your artifact.json
      # policy: research/policy.yml        # optional; defaults to beside the manifest
      # suite: "true"                     # install the supported companion tools

Notes:

  • The updated action installs from its own downloaded source directory by default: uses: ...@<tag-or-SHA> selects both the action and the installed package. ref is an optional explicit installation override; it no longer defaults to main. Set suite: "true" to install the supported companion tools. Use a commit SHA for an immutable action reference. This behavior is introduced in v0.5.0.

  • The gate needs its evidence tools on PATH (imm / padj / lf / fl) to ever pass; without them it fails closed by design. The tools live in the sibling repos: falsification-ledger, pit-adjuster, factor-qc, lookahead-free, ashare-data-immunity.

  • Exit codes: 0 = release, 1 = review needed, 2 = block.

Contract (v0.2, frozen)

The governance contract lives in schema/ and is locked by tests:

gov check — scenario 1 (done, M1)

AI-generated research conclusions must pass data + timing + evidence gates before they can ship:

# scaffold a research-conclusion project
gov init --dir research/ --name momentum-oos-review
# point gate-inputs.json at your data (imm / padj / lf / fl commands),
# then run the gate chain and decide:
gov check --manifest research/artifact.json
#   exit 0 = release, 1 = review_needed, 2 = block
# artifact.json is written back with decision, missing and gate evidence;
# raw tool outputs are persisted under research/reports/ (sha256-referenced)

gov report --manifest research/artifact.json   # human-readable

gov attach (done)

Attach evidence to an artifact before checking — the agent workflow:

gov attach --manifest research/artifact.json \
  --gate data_integrity --status pass --tool imm --report-ref sha256:...
gov attach --manifest research/artifact.json --attachment sources=docs/sources.md
gov attach --manifest research/artifact.json --declaration contains_returns=true
gov attach --manifest research/artifact.json --review approved --reviewer research-owner

Attaching evidence resets decision to pending — a decision is only as good as the evidence it was computed from, so any evidence change invalidates it until the next gov check. The same operation is exposed to agents as the gov_attach MCP tool.

Stable evidence fingerprints (done)

report_ref is a sha256: of the gate's tool output. Some tools stamp their output with run-time fields (imm audit emits checked_at / audit_date), so re-running the same check on the same data would change the fingerprint and dirty every artifact.json diff. Gate specs can declare those fields:

{
  "data_integrity": {
    "cmd": ["imm", "audit", "--watchlist", "watchlist.json", "--history-root", "history", "--audit-root", "audit"],
    "volatile_keys": ["audit_date", "checked_at"]
  }
}

volatile_keys are stripped (deep) from the JSON before hashing, with keys sorted for a canonical form. The raw tool output is still persisted as the gate report and the real run time stays in the gate entry's run_at — nothing is lost, only the noise stops changing the fingerprint. Non-JSON output keeps its byte-exact hash.

The acceptance suite (tests/test_m1_scenario1.py) runs 10 seeded-defect samples (survivorship ×3, look-ahead ×3, adjustment drift ×2, missing evidence ×2) against the real imm / lf / padj binaries — all 10 are blocked, zero false passes — plus a clean control that must release.

Evidence semantics (done)

Two attestation failures found by a production evidence grill (2026-09) are machine-checkable on the manifest: same-source evidence counted as independent and a claimed data cutoff later than the run that produced it (a now - lag stamp pretending to be a historical section).

Gate entries may attest two optional fields (gov attach, CLI or MCP):

gov attach --manifest research/artifact.json --gate pit_integrity \
  --status pass --tool padj --report-ref sha256:... \
  --evidence-source feed-a --data-cutoff 2026-09-06T14:59:00+08:00

  • evidence_source — which data lineage the gate evidence rests on;

  • data_cutoff — the data time the evidence reflects (not the run time).

Attestation is opt-in: an artifact that declares neither field anywhere keeps the legacy semantics (a placeholder report_ref may legitimately back several gate entries in manual-attach flows). Once any gate attests, gov check / gov report enforce these rules fail-closed on the whole artifact:

  • duplicate_evidence_content — two gate entries with the same report_ref: the same bytes counted twice;

  • data_cutoff_after_run_at — declared data time later than the run that produced the evidence: impossible by construction;

  • same_source_slice_not_independent — two pass gates declaring the same evidence_source and the same data_cutoff: one time slice cannot confirm twice, whatever the byte content (volatile stamps differ, the information does not);

  • warnings (not blockers): same_source_multiple (sequential slices from one source are complementary, not independent) and attestation_incomplete (a cutoff without a source cannot be checked).

Run the same check standalone (read-only):

gov evidence --manifest research/artifact.json

Scenarios 2 & 3 (done, M2)

  • strategy_advice — must carry backtest evidence: backtest_report and robustness_report attachments, plus the statistical_quality gate (real qc run). A backtest whose n_trials is not declared is a refusal, not a failure: qc refuses to judge → review_needed. A real overfitting blocker (DSR/PBO/haircut/MinTRL) → block. Acceptance suite: tests/test_m2_scenario23.py.

  • public_copy — must carry sources; when the copy declares return figures (declarations.contains_returns), limitations becomes required (conditional attachment, expressed in policy.yml, not code). A passing gov report prints the attachments and serves as the publication note.

Contract extension (frozen): artifact.declarations (boolean flags) and policy conditional_attachments (when/require).

Release integration (done, M3)

  • CI — .github/workflows/ci.yml: pytest matrix (3.11/3.12) + a fail-closed smoke (scaffold → gov check must exit 2 without evidence).

  • Reusable action — .github/actions/gov-check: composite action that runs gov check on any artifact in your workflows.

  • pre-commit hook — .pre-commit-hooks.yaml: gov check --manifest on every artifact.json; a block refuses the commit. Wire it with:

    # .pre-commit-config.yaml
repos:
  - repo: https://github.com/holdout-labs/holdout-governance
    rev: v0.5.0
    hooks:
      - id: gov-check

  • Agent interface — two ways to call gov from code/agents:

    • gov api --port 8000 — stdlib HTTP JSON API: GET /health, POST /check / /report / /init (no extra dependencies).

    • gov mcp — MCP stdio server (pip install 'holdout-governance[mcp]') exposing gov_check, gov_report, gov_init, gov_attach tools for Claude / Cursor / any MCP client.

Fit

Use this package as the wrapper above the existing Holdout tools:

Need

Existing tool

Data quality and snapshots

ashare-data-immunity

Point-in-time price meaning

pit-adjuster

Timing and future-data leakage

lookahead-free

Backtest quality

factor-qc

Claims and evidence trail

falsification-ledger

Past mistakes and reminders

lesson-book

This package records that the checks passed. It does not authorize execution.

Verify a PyPI release

Each of the seven package repositories now has verify-pypi.yml. A published GitHub release verifies its tag version on PyPI; the same workflow can be started manually with an explicit version after a separate upload. It uses a pinned pypi-verify-action, installs the exact version into a fresh environment and checks packaged modules, package/CLI version consistency and help output. It never uploads anything. A GitHub release made before the PyPI upload will fail verification until the version is available; rerun after uploading rather than treating the release page alone as proof of a usable package.

Development

python -m pip install -e .[test] pytest
python -m pytest

Available Tools

4 tools
gov_attachA

Attach gate evidence, attachments, declarations or a human review to an artifact (decision resets to pending — run gov_check afterwards). Record gate + status + tool + report_ref after a gate tool produced evidence, or review + reviewer after a human reviewed. Attachments/declarations are name=value strings; repeated calls merge, never drop existing entries. Do NOT claim a gate passed when no tool ran — gov_check trusts the journaled status. Returns the updated gates, attachments, declarations and review.

NameRequiredDescriptionDefault
gateNogate_id to record together with status, e.g. data_integrity or temporal_integrity.
toolNoName of the tool that produced the evidence, e.g. imm, padj, lf, fl, qc.
reviewNoHuman review outcome: approved or not_recorded.
statusNoGate outcome: pass, fail, warn or not_run.
manifestYesPath to artifact.json (v0.2) that will receive the evidence.
reviewerNoName of the human reviewer (auditable declaration, not authentication).
attachmentNoAttachment as name=value, e.g. sources=docs/sources.md. Repeat by calling again.
report_refNoEvidence reference for the report file, e.g. sha256:<hex> or a relative path.
data_cutoffNoAttest the data time the evidence reflects, ISO-8601 (distinct from run time). A cutoff later than the attach run is impossible and will block.
declarationNoDeclaration as name=true|false, e.g. contains_returns=true.
evidence_sourceNoAttest which data lineage the gate evidence rests on (e.g. feed id or input file). Two pass gates on the SAME source and SAME data_cutoff are NOT independent evidence and will block.

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well: it discloses that the decision resets to pending, that repeated calls merge and 'never drop existing entries', and that gov_check trusts the journaled status rather than the caller's claim. It omits permissions/auth requirements and any explicit error or blocking behavior (the two block conditions live only in the schema), which keeps it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four sentences, front-loaded with purpose before usage branches, constraints, and return shape. Only the closing 'Returns the updated gates, attachments, declarations and review' is largely redundant given an output schema exists.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex 11-parameter mutation tool with no annotations, the description covers purpose, the two recording modes, merge semantics, the reset-to-pending side effect, and the required gov_check follow-up; return values are covered by the output schema. Nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% (11 params fully described), so the baseline would be 3, but the description adds real semantic value by grouping parameters into co-occurring sets (gate+status+tool+report_ref versus review+reviewer) and by clarifying that attachments/declarations are repeatable name=value merges. Those pairings are not derivable from reading the schema properties individually.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence states a specific verb (Attach) and resource (gate evidence, attachments, declarations, human review) against an artifact, and further distinguishes itself by naming what it is not for and what must follow ('run gov_check afterwards'). An agent can separate it from gov_check and gov_report without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit when-to-use branches: 'Record gate + status + tool + report_ref after a gate tool produced evidence, or review + reviewer after a human reviewed.' It also states an exclusion — 'Do NOT claim a gate passed when no tool ran' — and gives the required follow-up call, so both the trigger and the anti-pattern are covered.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

gov_checkA

Run the full holdout gate chain on a research artifact and return the verdict (release / review_needed / block) as JSON. Executes every required gate of the artifact kind with the real tools (fail-closed: missing tool, crash or timeout records not_run and blocks) and writes the result back. Use as the final step before publishing any AI-assisted research claim — after gov_init created the project and gov_attach recorded the evidence. Do NOT call it before evidence is attached: missing required gates produce review_needed or block by design. Returns decision, exit_code, missing, gates and policy_ref_ok; treat exit_code 2 (block) as do-not-publish.

NameRequiredDescriptionDefault
kindNoOptional kind override, e.g. research_conclusion. Only use when the manifest's declared kind is wrong; otherwise leave unset.
policyNoOptional path to policy.yml. Defaults to policy.yml next to the manifest.
manifestYesPath to artifact.json (holdout v0.2 manifest) that records the research run.
gate_inputsNoOptional path to gate-inputs.json. Defaults to gate-inputs.json next to the manifest.

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden and meets it well. It discloses fail-closed behavior ('missing tool, crash or timeout records not_run and blocks'), the side effect of writing the result back, and the operational meaning of exit_code 2 as do-not-publish. This gives the agent a clear picture of safety and side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is four dense sentences with no filler: purpose and verdicts come first, then usage constraints, then return payload and error handling. Every sentence adds operational value, and critical information is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a gate-checking tool with an output schema, the description covers return fields (decision, exit_code, missing, gates, policy_ref_ok), failure semantics, prerequisites, and the do-not-publish signal. Combined with the fully described schema and output schema, an agent has everything needed to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the input schema already documents all four parameters including defaults and when kind override should be used. The description reinforces the manifest and evidence-sequencing context but does not add parameter-level detail beyond the schema, so the baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: 'Run the full holdout gate chain on a research artifact' and lists the exact verdict values (release / review_needed / block). Its lifecycle framing ('final step before publishing') distinguishes it from the related gov_init and gov_attach tools, so an agent can identify what it does without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit usage timing ('Use as the final step before publishing'), prerequisite ordering ('after gov_init created the project and gov_attach recorded the evidence'), and a clear don't-call condition ('Do NOT call it before evidence is attached'). It does not explicitly name gov_report as the alternative for reporting, so it stops just short of fully contrasting all siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

gov_initA

Scaffold a holdout governance project in a directory (created if missing): writes policy.yml and gate-inputs.json only if absent, and creates a new artifact.json with decision=pending. Never overwrites an existing artifact.json — if present it returns an error and writes nothing. Use once at the start of a research run, then attach evidence with gov_attach and decide with gov_check. Returns the created files and the policy_ref SHA.

NameRequiredDescriptionDefault
kindNoArtifact kind, which selects the required gates: research_conclusion, strategy_advice, public_copy or code.research_conclusion
nameNoOptional human-readable artifact id. Defaults to a timestamped id.
directoryYesDirectory to scaffold into. Created if missing (like mkdir -p).

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and does so well: it discloses that files are created, conditionally written only if absent, artifact.json is never overwritten, and an error is returned with no writes on conflict. It also states the return value (created files and policy_ref SHA).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but every sentence earns its place: what is written, the no-overwrite guarantee, the usage workflow, and the return value. It is front-loaded with the primary action and files.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity, an output schema, and full schema param coverage, the description provides enough context for correct invocation: setup workflow, safety behavior, failure condition, and return payload. Nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents kind, name, and directory. The description adds little beyond the schema's existing notes, though it does reinforce that directory is created if missing. This matches the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('scaffold'), resource ('holdout governance project'), and exactly what is written (policy.yml, gate-inputs.json, artifact.json). It also distinguishes itself from siblings by referencing gov_attach and gov_check in the stated workflow.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says to use it once at the start of a research run and then use gov_attach and gov_check, which gives an agent clear sequencing guidance. It also states the error condition where the tool should not be used (if artifact.json already exists).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

gov_reportA

Read-only governance assessment of a research artifact, returned as JSON: current decision state, missing required gates, pass/fail per recorded gate, and policy reference check — WITHOUT running any gate or changing any file. Use to inspect an artifact before attaching evidence or to understand why a previous check did not release. Prefer gov_check for fresh gate execution; gov_report never mutates and never spawns tools.

NameRequiredDescriptionDefault
policyNoOptional path to policy.yml. Defaults to policy.yml next to the manifest.
manifestYesPath to artifact.json (v0.2) or a v1 manifest to assess.

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description fully carries the behavioral disclosure burden. It clearly states the tool is read-only, runs no gates, changes no files, and never spawns tools. This gives an agent accurate expectations about side effects and safety.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact yet information-dense. Every sentence earns its place: the first defines behavior and output, the second gives use cases, and the third names the sibling alternative and reinforces non-mutation. Key constraints are front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is complete for a read-only inspection tool. It covers purpose, output shape, usage timing, alternatives, and side-effect guarantees. An output schema exists, so return-value details need not be repeated, and the sibling context makes the tool's role in the workflow clear.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents both parameters (manifest path and optional policy path). The description adds no additional parameter-level detail beyond what the schema provides, so the baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: a read-only governance assessment of a research artifact, returned as JSON. It enumerates the exact contents (decision state, missing gates, pass/fail per gate, policy reference check) and clearly distinguishes itself from gov_check by emphasizing it does not run gates.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use guidance: inspect an artifact before attaching evidence or understand why a previous check did not release. It also names the alternative (gov_check) and states the preference for fresh gate execution, plus exclusions (never mutates, never spawns tools).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 1 tool updatev0.5.0
    • Changedgov_attach2 fields changed
      • addedInput schema / properties / data_cutoff
        Added value: +{
+  "anyOf": [
+    {
+      "type": "string"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "description": "Attest the data time the evidence reflects, ISO-8601 (distinct from run time). A cutoff later than the attach run is impossible and will block.",
+  "title": "Data Cutoff"
+}
      • addedInput schema / properties / evidence_source
        Added value: +{
+  "anyOf": [
+    {
+      "type": "string"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "description": "Attest which data lineage the gate evidence rests on (e.g. feed id or input file). Two pass gates on the SAME source and SAME data_cutoff are NOT independent evidence and will block.",
+  "title": "Evidence Source"
+}
  2. 4 tool updatesv0.4.0
    • Changedgov_attach9 fields changed
      • addedInput schema / properties / attachment / description
        Added value: +"Attachment as name=value, e.g. sources=docs/sources.md. Repeat by calling again."
      • addedInput schema / properties / declaration / description
        Added value: +"Declaration as name=true|false, e.g. contains_returns=true."
      • addedInput schema / properties / gate / description
        Added value: +"gate_id to record together with status, e.g. data_integrity or temporal_integrity."
      • addedInput schema / properties / manifest / description
        Added value: +"Path to artifact.json (v0.2) that will receive the evidence."
      • addedInput schema / properties / report_ref / description
        Added value: +"Evidence reference for the report file, e.g. sha256:<hex> or a relative path."
      • addedInput schema / properties / review / description
        Added value: +"Human review outcome: approved or not_recorded."
      • addedInput schema / properties / reviewer / description
        Added value: +"Name of the human reviewer (auditable declaration, not authentication)."
      • addedInput schema / properties / status / description
        Added value: +"Gate outcome: pass, fail, warn or not_run."
      • addedInput schema / properties / tool / description
        Added value: +"Name of the tool that produced the evidence, e.g. imm, padj, lf, fl, qc."
    • Changedgov_check4 fields changed
      • addedInput schema / properties / gate_inputs / description
        Added value: +"Optional path to gate-inputs.json. Defaults to gate-inputs.json next to the manifest."
      • addedInput schema / properties / kind / description
        Added value: +"Optional kind override, e.g. research_conclusion. Only use when the manifest's declared kind is wrong; otherwise leave unset."
      • addedInput schema / properties / manifest / description
        Added value: +"Path to artifact.json (holdout v0.2 manifest) that records the research run."
      • addedInput schema / properties / policy / description
        Added value: +"Optional path to policy.yml. Defaults to policy.yml next to the manifest."
    • Changedgov_init3 fields changed
      • addedInput schema / properties / directory / description
        Added value: +"Directory to scaffold into. Created if missing (like mkdir -p)."
      • addedInput schema / properties / kind / description
        Added value: +"Artifact kind, which selects the required gates: research_conclusion, strategy_advice, public_copy or code."
      • addedInput schema / properties / name / description
        Added value: +"Optional human-readable artifact id. Defaults to a timestamped id."
    • Changedgov_report2 fields changed
      • addedInput schema / properties / manifest / description
        Added value: +"Path to artifact.json (v0.2) or a v1 manifest to assess."
      • addedInput schema / properties / policy / description
        Added value: +"Optional path to policy.yml. Defaults to policy.yml next to the manifest."
  3. 4 tool updatesv0.1.0
    • First observedgov_attach
    • First observedgov_check
    • First observedgov_init
    • First observedgov_report
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.7/5.0

Scored across 4 tools

Disambiguation5/5

Each tool has a clearly distinct role: gov_init scaffolds, gov_attach records evidence, gov_check executes gates and returns a verdict, and gov_report reads state without mutating. Descriptions explicitly contrast gov_report (read-only) against gov_check (fresh execution), removing the main risk of confusion. No overlapping or redundant tools.

Naming Consistency5/5

All four tools follow the same gov_ prefix with a consistent short verb (init, attach, check, report). The pattern is perfectly predictable across the set.

Tool Count5/5

Four tools map cleanly onto a linear workflow (scaffold → attach → decide → inspect), each earning its place with no filler. The count is well-scoped for the domain.

Completeness4/5

The toolset covers the full lifecycle init → attach → check → report, with no dead ends in the primary flow. Minor gaps exist (e.g. no explicit tool to list/inspect multiple artifacts or revoke a prior decision), but core workflows are covered.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • sigmodx-mcp

    Sigmodx
    A
    license
    A
    quality
    D
    maintenance
    Audit infrastructure for AI agents to log consequential decisions (invoice, GL, anomaly) and verify attestations via MCP tools.
    6
    MIT

  • audit-ledger-mcp

    shahidh68
    A
    license
    A
    quality
    D
    maintenance
    Tamper-evident audit logging for AI decisions. Three tools (record_decision, verify_decision, list_decisions) write to a regulator-grade ledger built on AWS S3 Object Lock with 7-year retention. Designed for EU AI Act Article 12 and FCA SS1/23 evidence requirements. Try zero-config: npx audit-ledger-mcp boots in sandbox mode against a public hosted tenant.
    3
    31 npm
    1
    Apache 2.0
  • @vaibot/mcp-server by vaibot-io

    @vaibot/mcp-serverofficial

    vaibot-io
    A
    license
    A
    quality
    D
    maintenance
    Governance circuit-breaker MCP server that enables AI agents to request risk-based decisions, approve or deny actions, and finalize outcomes with full audit receipts.
    4
    MIT