Skip to main content
Glama
evidence-readiness

io.github.BrienBounded/evr

by evidence-readiness
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

evidence-readiness

Checks whether an agent execution trace contains the evidence a post-hoc causal reconstruction would require.

When an AI agent causes a loss — a bad config change, a cross-session data leak, contaminated output cascading between agents, a prompt injection — someone eventually has to reconstruct why from the trace. Traces from general-purpose agent instrumentation cannot support that reconstruction: the evidence was never recorded. This repository contains the evidence-readiness specification (SPEC.md), a checker that evaluates a trace against it, and reference fixtures showing what conformant instrumentation looks like.

What this is — and is not

  • It checks that evidence is present, not that it is true. A fabricated-but-internally-consistent trace passes. This tool authenticates nothing, and its result must never be relayed as "this agent is safe."

  • It never determines whether a failure occurred. It answers "does this trace contain what a reconstruction of class X would require," never "did class X occur." It contains no detection or scoring logic.

  • This repository does not contain the reconstruction engine. The specification is published so the method is assessable; the engine is available for inspection under the terms in SOURCE_AVAILABILITY.md.

  • Four failure classes are covered: config_drift, session_isolation_failure, cascade_contamination, prompt_injection. A fifth (tool_misuse) was investigated and declined for stated structural reasons — a documented negative result, not a gap (SPEC.md §8).

  • The spec is a profile of the OTel GenAI semantic conventions: core OTel plus documented extensions (SPEC.md §2). It carries no standards-body endorsement.

Related MCP server: ProofGate

Who it is for

  • Engineers instrumenting agents who want incidents to be reconstructable after the fact — diff your trace against a conformant fixture and see what to add.

  • Reviewers (insurers, auditors, counterparties) evaluating whether an agent system's telemetry would support retrospective loss adjudication.

  • Coding agents: see llms.txt, spec.json (the machine-readable spec export, generated — never hand-maintained), and the MCP server in src/evidence_readiness/mcp_server.py (configuration below).

Quickstart (60 seconds)

# from a clone of this repository
pip install .
evr check path/to/trace.json        # or a directory of traces
evr check --json path/to/traces/    # machine-readable report
evr check --require-class prompt_injection path/to/trace.json

The package has zero runtime dependencies; pip install . in any Python ≥3.10 environment is the whole setup. Equivalent uvx forms, both exercised: uvx --from . evr check ./traces from a clone, and uvx --from evidence-readiness evr check ./traces from the package index.

Both need --from, and the trailing evr is not optional. A bare uvx evidence-readiness ... runs the console script named for the distribution, and that script is the MCP server (below), not the checker — it reads standard input, ignores the arguments, and exits successfully without checking anything.

Exit codes carry no reconstruction vocabulary: 0 all requirements met · 1 one or more requirements partially met · 2 one or more requirements unmet · 3 malformed input (SPEC.md §9 is the source of record). Universal requirements (EVR-R1..R6) drive the exit code; per-class evidence readiness is reported separately, and --require-class folds a named class's readiness into the exit code. For every failed requirement the output names the specific missing field and the span kind it belongs on.

Every run ends with the same statement this README opened with: presence, not truth — evidence being present does not make it authentic, and a passing trace is not a safe agent.

CI: check every PR (GitHub Action)

A spec is read once; a check that runs on every PR changes instrumentation behavior. Add this to a workflow in the repository that produces your traces (evidence-readiness is this repository's owner once published; from within this repository itself, uses: ./.github/actions/readiness-check):

steps:
  - uses: actions/checkout@v4
  - uses: evidence-readiness/spec/.github/actions/readiness-check@main
    with:
      trace-path: traces/

That is the whole setup — the default GitHub runner's python3 is sufficient, and the Action installs the (zero-dependency) checker itself. The step fails when the checker's exit code exceeds minimum-passing-exit-level (default 0; exit-code semantics as in the Quickstart above). Optional required-classes folds named classes' evidence requirements into the exit code. Outputs: exit-code, exit-meaning, report-path (full JSON report), and per-class-readiness. PR annotations name each incomplete requirement's specific missing field and span kind. Every annotation is a statement about the trace telemetry, never about the agent: an annotated PR has incomplete evidence, not a detected failure — and a clean run is not a safe agent. This repository's own CI (.github/workflows/ci.yml) dogfoods the Action against fixtures/, asserting the expected — deliberately nonzero — exit codes per bucket.

MCP server (agent surface)

src/evidence_readiness/mcp_server.py exposes the checker and the requirement data to agents over the Model Context Protocol (stdio). Three tools: check_trace_readiness (file, directory, or inline JSON — inline content is processed in memory and never written anywhere), get_requirement, and list_class_requirements.

Installing the package delivers three console scripts, and it is worth being precise about which is which before you copy anything below:

script

what it runs

evr

the checker CLI — this is the one a human wants

evr-mcp

the MCP server, on stdio

evidence-readiness

the MCP server, on stdio — same target as evr-mcp, named for the distribution so that per-invocation runners which resolve <package> to a like-named script find it

So evidence-readiness is not the checker. Typing it at a shell gets a server waiting for JSON-RPC on stdin, which looks like a hang. Use evr.

If the package is installed, the evr-mcp console script is the whole configuration:

{
  "mcpServers": {
    "evidence-readiness": {
      "command": "evr-mcp"
    }
  }
}

From a clone, with nothing installed, run the module by path instead:

{
  "mcpServers": {
    "evidence-readiness": {
      "command": "python3",
      "args": [
        "/absolute/path/to/evidence-readiness/src/evidence_readiness/mcp_server.py"
      ]
    }
  }
}

Either way it needs no dependencies, and the second form needs no installation. It is local-only: no network calls, no phone-home, nothing persisted. It deliberately has no submission tool — the server never transmits anything; submitting a trace anywhere is a human decision gated on the CONTRIBUTING.md disclaimer and a redaction judgment an agent cannot make. Every tool description carries the presence-not-truth statement, so a readiness result cannot reasonably be relayed as "this agent is safe."

Status

The specification is versioned (SPEC.md carries its current version and correction log) and the checker is implemented against it, with reference fixtures (fixtures/), the machine-readable export (spec.json), the agent surface (llms.txt, the MCP server), and the CI action (.github/actions/readiness-check/) in place. The methodology and limitations paper is drafted (PAPER.md); its front-matter banner records which sections have had an adversarial read and which have not. The standing disclosure policy is SOURCE_AVAILABILITY.md; reporting and trace-submission terms are CONTRIBUTING.md, with redaction guidance in REDACTION.md and release verification in RELEASING.md. Corrections are logged in CHANGELOG.md. The spec is descriptive: every requirement is backed by an observed outcome in a validation record of 240 reconstruction attempts, 222 of them synthetic — backing is per-requirement, from the specific cases that exercised it, not a property measured across all 240 (18 real-world Who&When cases validate the abstention direction only; the split is stated wherever the count appears — SPEC.md §1).

Attribution

Brien Buckman

Licensed under Apache-2.0 (see LICENSE).

The substance of this repository's development record is public: corrections that affect published content are logged in CHANGELOG.md, and the development defect record, as it bears on the method, is summarized in PAPER.md §8.6. The commit history in which these files were drafted and revised is not published.

Available Tools

3 tools
check_trace_readinessA

Check an agent execution trace against the evidence-readiness specification and report, per requirement, whether the evidence a post-hoc causal reconstruction would need is met, partially met, or unmet — naming the specific missing field and the span kind it belongs on. Accepts a file/directory path OR inline JSON content (exactly one). Runs entirely locally in this process; nothing is transmitted or persisted, and inline content is processed in memory only. This tool NEVER determines whether a failure occurred: it answers "does this trace contain what a reconstruction of class X would require," never "did class X occur." A fabricated-but-internally-consistent trace passes. This tool verifies evidence is present, not that it is true, and its result must never be relayed as "this agent is safe".

ParametersJSON Schema
NameRequiredDescriptionDefault
trace_pathNoPath to a trace .json file or a directory of traces (*manifest.json excluded).
require_classNoFold these classes' evidence requirements into the exit code (choices: config_drift, session_isolation_failure, cascade_contamination, prompt_injection).
trace_contentNoA trace document as a JSON string, checked in memory and never written to disk.

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden, and it does so richly: local-only execution, nothing transmitted or persisted, inline content memory-only, and an explicit statement that it never determines whether a failure occurred. It also warns that a fabricated-but-consistent trace passes and that results must not be relayed as a safety verdict.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action, then inputs, then scope limits. Every sentence is substantive, though the final two sentences about fabricated traces and non-relay are dense with overlapping caveats that could be tightened.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the return-value burden and does it: per-requirement status plus the specific missing field and its span kind. For a local, non-mutating analysis tool, an agent has everything needed to call it and interpret the result correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds the meaningful mutual-exclusivity constraint ('exactly one' of path or inline content) that the schema does not encode. It also implies the reporting granularity tied to require_class, though it doesn't restate the enum values.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource (check a trace against the evidence-readiness specification) with a precise reporting scope: per-requirement met/partially met/unmet. It is clearly distinguishable from siblings get_requirement and list_class_requirements, which retrieve requirements rather than evaluate an artifact.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear operating context: accepts a path OR inline JSON, 'exactly one', and runs locally in-process. It does not explicitly name the sibling tools or say when to call this versus fetching requirements first, leaving that routing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_requirementA

Return one evidence-readiness requirement by ID (for example EVR-R3 or EVR-PI-7): its title, the published field forms, the span kind the evidence belongs on, its spec section, and — for prompt_injection — which attribution tiers include it. Requirements state what must be PRESENT in a trace; the spec is not an enumeration of everything the reconstruction engine reads, so no inference can be drawn from a field's absence. This tool verifies evidence is present, not that it is true, and its result must never be relayed as "this agent is safe".

ParametersJSON Schema
NameRequiredDescriptionDefault
requirement_idYesA requirement ID, e.g. EVR-R1, EVR-CD-4.

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well: it discloses result contents, warns that the spec is not an exhaustive enumeration of what the engine reads, and explicitly bounds the semantics ('verifies evidence is present, not that it is true'). It stops short of noting whether a missing ID errors or returns empty, which is the main unstated behavioral edge case.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the return-value summary in sentence one, then progressively adds the semantic caveats. The em-dash parenthetical makes the opening sentence dense, but every clause earns its place and there is no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description rightly enumerates the returned fields, and it adds the interpretation guardrails an agent needs to avoid misrelaying results. Only the behavior on an unknown/invalid ID is left uncovered.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for the single parameter, so the schema already documents requirement_id and its example format. The description's own ID examples (EVR-R3, EVR-PI-7) echo rather than extend the schema, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Return one ... requirement by ID') plus the resource and even enumerates the returned fields (title, published field forms, span kind, spec section, attribution tiers). The 'one ... by ID' framing implicitly distinguishes it from the sibling list_class_requirements, which is a list-style tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives important interpretation rules (no inference from a field's absence; result verifies presence, not truth; never relay as 'this agent is safe'), but it never states when to pick this tool over check_trace_readiness or list_class_requirements. Usage context is implied by the ID lookup, not explicitly routed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_class_requirementsA

List the evidence requirements for one failure class (config_drift, session_isolation_failure, cascade_contamination, prompt_injection), or pass "universal" for the requirements every trace must meet (EVR-R1..R6). States which items are required versus supporting, and for prompt_injection the per-tier evidence sets. tool_misuse returns its documented negative result (investigated and declined), not an error. Requirements state what must be PRESENT; nothing may be inferred about fields the spec does not name. This tool verifies evidence is present, not that it is true, and its result must never be relayed as "this agent is safe".

ParametersJSON Schema
NameRequiredDescriptionDefault
class_nameYesOne of: config_drift, session_isolation_failure, cascade_contamination, prompt_injection, universal, tool_misuse.

TDQS

A4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so richly: it states that requirements describe only what must be PRESENT, that nothing may be inferred about unnamed fields, that the tool verifies presence and not truth, that tool_misuse returns a documented negative result rather than an error, and that results must never be relayed as a safety guarantee.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core purpose is front-loaded in the first clause, and subsequent sentences each add distinct constraints (required vs supporting, per-tier sets, negative result, presence-vs-truth caveat). It is dense and slightly long, but no sentence is filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter, no-annotation, no-output-schema tool, the description covers selector semantics, what the result contains (required vs supporting, per-tier sets), and critical interpretive limits. An agent has enough to call it correctly, though return shape specifics are left implicit.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single parameter is fully documented in the schema, so the baseline is 3. The description adds real meaning beyond it by explaining the 'universal' option and the special tool_misuse return behavior.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

It names a specific verb ('List') and resource ('evidence requirements for one failure class') and enumerates the exact valid class values, including the 'universal' special case. The purpose is unambiguous, but it never distinguishes itself from the sibling 'get_requirement', which could plausibly be chosen instead.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It clarifies the 'universal' selector and explains tool_misuse's negative-result behavior, which is useful usage context. However, it gives no explicit when-to-use-this-vs-'get_requirement'/'check_trace_readiness' guidance and no exclusions, so routing remains implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 3 tool updatesv0.1.0
    • First observedcheck_trace_readiness
    • First observedget_requirement
    • First observedlist_class_requirements
Deploy Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.2/5.0

Scored across 3 tools

Disambiguation4/5

The three tools are largely distinct: check_trace_readiness evaluates a trace, get_requirement fetches a single requirement by ID, and list_class_requirements enumerates requirements for a class. The two retrieval tools overlap slightly in purpose, but their ID-vs-class parameters clearly separate them.

Naming Consistency5/5

All three names follow a clean snake_case verb_noun pattern (check_trace_readiness, get_requirement, list_class_requirements), with verbs matching the operation. No mixed conventions.

Tool Count4/5

Three tools is a bit lean but well-matched to a narrow read-only spec-inspection purpose. Each tool has a distinct role, though a broader server might warrant a tool to enumerate classes or list all requirements.

Completeness4/5

The surface covers the core lifecycle: evaluate a trace, look up a requirement, and list requirements per class (including 'universal'). Since the domain is inherently read-only, no create/update/delete is needed; only minor gaps like a global enumeration exist.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • ProofGate

    yuzhiyang1
    A
    license
    Not graded
    quality
    C
    maintenance
    Evidence-first delivery audit MCP server that evaluates task requirements against delivery evidence and returns a reproducible pass/needs_review/fail decision with a deterministic receipt.
    MIT

  • flameox

    morluto
    A
    license
    C
    quality
    A
    maintenance
    Enables coding agents to query, compare, and audit local profiler traces, benchmarks, memory captures, and execution evidence without uploading code or data, using CLI and MCP interfaces.
    111
    121
    MIT

  • CodexPro Runtime

    Menglook
    A
    license
    B
    quality
    B
    maintenance
    A local, evidence-driven MCP runtime and control plane for open-source maintainers that provides workspace-bounded tools including controlled file operations, command execution, validation primitives, durable execution records, and human review workflows via stdio and Streamable HTTP transports.
    33
    MIT