Skip to main content
Glama

evidence-readiness

Checks whether an agent execution trace contains the evidence a post-hoc causal reconstruction would require.

When an AI agent causes a loss — a bad config change, a cross-session data leak, contaminated output cascading between agents, a prompt injection — someone eventually has to reconstruct why from the trace. Traces from general-purpose agent instrumentation cannot support that reconstruction: the evidence was never recorded. This repository contains the evidence-readiness specification (SPEC.md), a checker that evaluates a trace against it, and reference fixtures showing what conformant instrumentation looks like.

What this is — and is not

  • It checks that evidence is present, not that it is true. A fabricated-but-internally-consistent trace passes. This tool authenticates nothing, and its result must never be relayed as "this agent is safe."

  • It never determines whether a failure occurred. It answers "does this trace contain what a reconstruction of class X would require," never "did class X occur." It contains no detection or scoring logic.

  • This repository does not contain the reconstruction engine. The specification is published so the method is assessable; the engine is available for inspection under the terms in SOURCE_AVAILABILITY.md.

  • Four failure classes are covered: config_drift, session_isolation_failure, cascade_contamination, prompt_injection. A fifth (tool_misuse) was investigated and declined for stated structural reasons — a documented negative result, not a gap (SPEC.md §8).

  • The spec is a profile of the OTel GenAI semantic conventions: core OTel plus documented extensions (SPEC.md §2). It carries no standards-body endorsement.

Related MCP server: flameox

Who it is for

  • Engineers instrumenting agents who want incidents to be reconstructable after the fact — diff your trace against a conformant fixture and see what to add.

  • Reviewers (insurers, auditors, counterparties) evaluating whether an agent system's telemetry would support retrospective loss adjudication.

  • Coding agents: see llms.txt, spec.json (the machine-readable spec export, generated — never hand-maintained), and the MCP server in src/evidence_readiness/mcp_server.py (configuration below).

Quickstart (60 seconds)

# from a clone of this repository
pip install .
evr check path/to/trace.json        # or a directory of traces
evr check --json path/to/traces/    # machine-readable report
evr check --require-class prompt_injection path/to/trace.json

The package has zero runtime dependencies; pip install . in any Python ≥3.10 environment is the whole setup. Equivalent uvx forms, both exercised: uvx --from . evr check ./traces from a clone, and uvx --from evidence-readiness evr check ./traces from the package index.

Both need --from, and the trailing evr is not optional. A bare uvx evidence-readiness ... runs the console script named for the distribution, and that script is the MCP server (below), not the checker — it reads standard input, ignores the arguments, and exits successfully without checking anything.

Exit codes carry no reconstruction vocabulary: 0 all requirements met · 1 one or more requirements partially met · 2 one or more requirements unmet · 3 malformed input (SPEC.md §9 is the source of record). Universal requirements (EVR-R1..R6) drive the exit code; per-class evidence readiness is reported separately, and --require-class folds a named class's readiness into the exit code. For every failed requirement the output names the specific missing field and the span kind it belongs on.

Every run ends with the same statement this README opened with: presence, not truth — evidence being present does not make it authentic, and a passing trace is not a safe agent.

CI: check every PR (GitHub Action)

A spec is read once; a check that runs on every PR changes instrumentation behavior. Add this to a workflow in the repository that produces your traces (evidence-readiness is this repository's owner once published; from within this repository itself, uses: ./.github/actions/readiness-check):

steps:
  - uses: actions/checkout@v4
  - uses: evidence-readiness/spec/.github/actions/readiness-check@main
    with:
      trace-path: traces/

That is the whole setup — the default GitHub runner's python3 is sufficient, and the Action installs the (zero-dependency) checker itself. The step fails when the checker's exit code exceeds minimum-passing-exit-level (default 0; exit-code semantics as in the Quickstart above). Optional required-classes folds named classes' evidence requirements into the exit code. Outputs: exit-code, exit-meaning, report-path (full JSON report), and per-class-readiness. PR annotations name each incomplete requirement's specific missing field and span kind. Every annotation is a statement about the trace telemetry, never about the agent: an annotated PR has incomplete evidence, not a detected failure — and a clean run is not a safe agent. This repository's own CI (.github/workflows/ci.yml) dogfoods the Action against fixtures/, asserting the expected — deliberately nonzero — exit codes per bucket.

MCP server (agent surface)

src/evidence_readiness/mcp_server.py exposes the checker and the requirement data to agents over the Model Context Protocol (stdio). Three tools: check_trace_readiness (file, directory, or inline JSON — inline content is processed in memory and never written anywhere), get_requirement, and list_class_requirements.

Installing the package delivers three console scripts, and it is worth being precise about which is which before you copy anything below:

script

what it runs

evr

the checker CLI — this is the one a human wants

evr-mcp

the MCP server, on stdio

evidence-readiness

the MCP server, on stdio — same target as evr-mcp, named for the distribution so that per-invocation runners which resolve <package> to a like-named script find it

So evidence-readiness is not the checker. Typing it at a shell gets a server waiting for JSON-RPC on stdin, which looks like a hang. Use evr.

If the package is installed, the evr-mcp console script is the whole configuration:

{
  "mcpServers": {
    "evidence-readiness": {
      "command": "evr-mcp"
    }
  }
}

From a clone, with nothing installed, run the module by path instead:

{
  "mcpServers": {
    "evidence-readiness": {
      "command": "python3",
      "args": [
        "/absolute/path/to/evidence-readiness/src/evidence_readiness/mcp_server.py"
      ]
    }
  }
}

Either way it needs no dependencies, and the second form needs no installation. It is local-only: no network calls, no phone-home, nothing persisted. It deliberately has no submission tool — the server never transmits anything; submitting a trace anywhere is a human decision gated on the CONTRIBUTING.md disclaimer and a redaction judgment an agent cannot make. Every tool description carries the presence-not-truth statement, so a readiness result cannot reasonably be relayed as "this agent is safe."

Status

The specification is versioned (SPEC.md carries its current version and correction log) and the checker is implemented against it, with reference fixtures (fixtures/), the machine-readable export (spec.json), the agent surface (llms.txt, the MCP server), and the CI action (.github/actions/readiness-check/) in place. The methodology and limitations paper is drafted (PAPER.md); its front-matter banner records which sections have had an adversarial read and which have not. The standing disclosure policy is SOURCE_AVAILABILITY.md; reporting and trace-submission terms are CONTRIBUTING.md, with redaction guidance in REDACTION.md and release verification in RELEASING.md. Corrections are logged in CHANGELOG.md. The spec is descriptive: every requirement is backed by an observed outcome in a validation record of 240 reconstruction attempts, 222 of them synthetic — backing is per-requirement, from the specific cases that exercised it, not a property measured across all 240 (18 real-world Who&When cases validate the abstention direction only; the split is stated wherever the count appears — SPEC.md §1).

Attribution

Brien Buckman

Licensed under Apache-2.0 (see LICENSE).

The substance of this repository's development record is public: corrections that affect published content are logged in CHANGELOG.md, and the development defect record, as it bears on the method, is summarized in PAPER.md §8.6. The commit history in which these files were drafted and revised is not published.

Available Tools

3 tools
check_trace_readinessA

Check an agent execution trace against the evidence-readiness specification and report, per requirement, whether the evidence a post-hoc causal reconstruction would need is met, partially met, or unmet — naming the specific missing field and the span kind it belongs on. Accepts a file/directory path OR inline JSON content (exactly one). Runs entirely locally in this process; nothing is transmitted or persisted, and inline content is processed in memory only. This tool NEVER determines whether a failure occurred: it answers "does this trace contain what a reconstruction of class X would require," never "did class X occur." A fabricated-but-internally-consistent trace passes. This tool verifies evidence is present, not that it is true, and its result must never be relayed as "this agent is safe".

Inputs: trace_path, require_class, trace_content.

get_requirementA

Return one evidence-readiness requirement by ID (for example EVR-R3 or EVR-PI-7): its title, the published field forms, the span kind the evidence belongs on, its spec section, and — for prompt_injection — which attribution tiers include it. Requirements state what must be PRESENT in a trace; the spec is not an enumeration of everything the reconstruction engine reads, so no inference can be drawn from a field's absence. This tool verifies evidence is present, not that it is true, and its result must never be relayed as "this agent is safe".

Inputs: requirement_id (required).

list_class_requirementsA

List the evidence requirements for one failure class (config_drift, session_isolation_failure, cascade_contamination, prompt_injection), or pass "universal" for the requirements every trace must meet (EVR-R1..R6). States which items are required versus supporting, and for prompt_injection the per-tier evidence sets. tool_misuse returns its documented negative result (investigated and declined), not an error. Requirements state what must be PRESENT; nothing may be inferred about fields the spec does not name. This tool verifies evidence is present, not that it is true, and its result must never be relayed as "this agent is safe".

Inputs: class_name (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.2/5.0

    Scored across 3 tools

    Disambiguation4/5

    The three tools are largely distinct: check_trace_readiness evaluates a trace, get_requirement fetches a single requirement by ID, and list_class_requirements enumerates requirements for a class. The two retrieval tools overlap slightly in purpose, but their ID-vs-class parameters clearly separate them.

    Naming Consistency5/5

    All three names follow a clean snake_case verb_noun pattern (check_trace_readiness, get_requirement, list_class_requirements), with verbs matching the operation. No mixed conventions.

    Tool Count4/5

    Three tools is a bit lean but well-matched to a narrow read-only spec-inspection purpose. Each tool has a distinct role, though a broader server might warrant a tool to enumerate classes or list all requirements.

    Completeness4/5

    The surface covers the core lifecycle: evaluate a trace, look up a requirement, and list requirements per class (including 'universal'). Since the domain is inherently read-only, no create/update/delete is needed; only minor gaps like a global enumeration exist.

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      A
      maintenance
      Enables coding agents to coordinate local profilers, benchmark tools, trace processors, and explicit targets into bounded runtime evidence, including observed inputs, execution provenance, typed findings, coverage, and limitations. It supports analysis and capture through named MCP tools without workspace setup or project configuration.
      -
    • F
      license
      Not graded
      quality
      A
      maintenance
      Enables agents to gather bounded, reproducible runtime evidence by coordinating local profilers, benchmark tools, and trace processors against explicit artifacts or captured commands. It exposes typed analysis and capture operations over MCP that return provenance-tracked results inline, with optional durable preservation and replay of prior evidence.
      -
    • F
      license
      Not graded
      quality
      B
      maintenance
      Enables agents to run bounded local profiling, benchmarking, and trace analysis on explicit artifacts or commands, returning typed evidence with provenance, coverage, limitations, and recovery actions. It also exposes optional evidence preservation and inspection through MCP tools.
      -
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables MCP clients to check acceptance evidence against a specification and JUnit results, and to review structured AI proposals for current contract hash, coverage, and in-scope paths. Both tools are read-only, idempotent, content-based, and require no model credentials or filesystem access.
      MIT