Skip to main content
Glama

TraceCheck: a reliability check for agent traces (MCP + A2A)

Status: test facility v0.2. Free, no API key, deterministic and offline (keyword heuristics, no LLM).

For agents: after a multi-step task, send goal + trace + outcome. You get back a score from 0 to 100, four sub-scores (goal_clarity, observability, error_recovery, goal_alignment), the issues found and up to 5 concrete fixes.

The loop: run task → evaluate_trace → apply suggestions → re-run → evaluate_trace again and compare.

Tools

Tool

Purpose

evaluate_trace(goal, trace, outcome, agent_name?)

Scores the trace. Returns {score, dimensions, issues, suggestions, summary, version}

get_trace_schema()

Returns the input format, dimension weights and a copyable example

Related MCP server: Pisama MCP Server

Quickstart

pip install -r requirements.txt          # mcp>=2.3 (the MCPServer API)
python server.py                         # stdio (local MCP clients)
python server.py --http                  # streamable HTTP on $PORT (default 8000)
#   MCP:        http://localhost:8000/mcp
#   Agent card: http://localhost:8000/.well-known/agent-card.json  (also /.well-known/agent.json)
#   A2A:        POST http://localhost:8000/a2a/message:send
#   Health:     http://localhost:8000/health
python examples/run_examples.py          # scores the sample traces

Env: TRACECHECK_TRANSPORT=http, PORT, HOST (default 0.0.0.0), PUBLIC_URL (put into the agent card), TRACECHECK_LOG_FILE (default /workspace/artifacts/tracecheck/calls.jsonl, falls back to ~/.tracecheck/calls.jsonl).

Docker: docker build -t tracecheck . && docker run -p 8000:8000 tracecheck

MCP client config

Remote (once hosted):

{ "mcpServers": { "tracecheck": { "type": "streamable-http", "url": "https://tracecheck-mcp.onrender.com/mcp" } } }

Local stdio:

{ "mcpServers": { "tracecheck": { "command": "python", "args": ["/path/to/tracecheck/server.py"] } } }

Trace format

Plain strings. trace = chronological steps, one per line: what you called, what it returned/observed, any error and the retry/fallback, and a final Verified: ... line.

{
  "goal": "Find the cheapest direct flight from Amsterdam to Lisbon on 12 March and report price and flight number.",
  "trace": "1. Called search_flights(AMS->LIS, 2027-03-12, direct)\n2. Observed: returned 4 flights; cheapest KL1693 EUR 89\n3. get_fare_details -> error: timeout\n4. Retried -> returned EUR 89 incl. taxes\n5. Verified: direct, correct date, cheapest; matches goal",
  "outcome": "KL1693 AMS->LIS on 12 March, EUR 89 incl. taxes.",
  "agent_name": "my-agent"
}

A2A: send the same object as a data part: {"message":{"role":"ROLE_USER","messageId":"1","parts":[{"data":{...}}]}}.

Every call is logged to calls.jsonl (timestamp, agent_name, goal preview, trace length, score, dimensions).

Available Tools

2 tools
evaluate_traceAInspect

Get an independent, deterministic reliability check of a multi-step task you (an agent) just ran. Returns a 0-100 score, the issues found, and concrete fixes to apply on your next attempt.

WHEN TO CALL: after you finish (or give up on) a multi-step task that used tools, before you report "done" to the user, or after a failure you want to learn from. Free, no API key, fast and offline (keyword heuristics, no LLM). Call get_trace_schema first if unsure of the format.

INPUT (all plain strings):

  • goal: the original task, stated specifically (what success looks like). Vague goals under 20 chars are penalised.

  • trace: your chronological steps, one per line, e.g. "1. Called search_flights(AMS->LIS) -> returned 3 results" / "2. Observed: ..." . Write what each tool RETURNED ("observed", "returned", "result:"), any error and the retry/fallback you used, and a final "Verified: ..." line.

  • outcome: the concrete final result you claim, and how it meets the goal.

  • agent_name: optional identifier for yourself (logging only).

RETURNS JSON: score (int 0-100), dimensions {goal_clarity, observability, error_recovery, goal_alignment}, issues (list), suggestions (list, max 5), summary, version.

THE LOOP: run task -> evaluate_trace -> apply the suggestions to your process -> re-run -> evaluate_trace again and compare scores/dimensions. Scores below ~60 usually mean missing observations, unrecovered errors, or no verification step.

Inputs: goal (required), trace (required), outcome (required), agent_name.

get_trace_schemaAInspect

Return the exact input format TraceCheck's evaluate_trace expects, the scored dimensions and weights, and a complete example call you can copy. Call this once before your first evaluate_trace. No arguments, no side effects.

Inputs: no named parameters.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.6/5.0

    Scored across 2 tools

    Disambiguation5/5

    The two tools have clearly distinct roles: one returns format/schema information, the other performs the actual trace evaluation. The descriptions explicitly instruct when to use each, leaving no realistic ambiguity.

    Naming Consistency5/5

    Both names use a consistent snake_case verb_noun pattern: get_trace_schema and evaluate_trace. The convention is predictable and readable.

    Tool Count3/5

    Two tools is thin for most servers, but here one is a schema/introspection helper and the other is the core evaluator. It is borderline minimal rather than clearly wrong.

    Completeness4/5

    The surface covers the core workflow: learn the input format, then evaluate a trace. Minor gaps exist around comparing multiple evaluations or retrieving history, but agents can work around these by recording scores themselves.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers