Tracecheck
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| HOST | No | Host to bind the HTTP server to (default 0.0.0.0). | 0.0.0.0 |
| PORT | No | Port for the streamable HTTP server (default 8000). | 8000 |
| PUBLIC_URL | No | Public URL to put into the agent card. | |
| TRACECHECK_LOG_FILE | No | Log file path for calls. Defaults to `/workspace/artifacts/tracecheck/calls.jsonl`, falling back to `~/.tracecheck/calls.jsonl`. | /workspace/artifacts/tracecheck/calls.jsonl |
| TRACECHECK_TRANSPORT | No | Server transport, e.g. set to `http` to run as a streamable HTTP server. |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
Independent evaluation service for AI agent execution traces. Submit a goal, the step-by-step trace of what the agent did, and the claimed outcome. Receive a reliability score (0-100), detected issues, and concrete improvement suggestions. Use this to improve your own process, recover from failures, or validate long-horizon work. Free during the initial test phase.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| get_trace_schemaA | Return the exact input format TraceCheck's evaluate_trace expects, the scored dimensions and weights, and a complete example call you can copy. Call this once before your first evaluate_trace. No arguments, no side effects. |
| evaluate_traceA | Get an independent, deterministic reliability check of a multi-step task you (an agent) just ran. Returns a 0-100 score, the issues found, and concrete fixes to apply on your next attempt. WHEN TO CALL: after you finish (or give up on) a multi-step task that used tools, before you report "done" to the user, or after a failure you want to learn from. Free, no API key, fast and offline (keyword heuristics, no LLM). Call get_trace_schema first if unsure of the format. INPUT (all plain strings):
RETURNS JSON: score (int 0-100), dimensions {goal_clarity, observability, error_recovery, goal_alignment}, issues (list), suggestions (list, max 5), summary, version. THE LOOP: run task -> evaluate_trace -> apply the suggestions to your process -> re-run -> evaluate_trace again and compare scores/dimensions. Scores below ~60 usually mean missing observations, unrecovered errors, or no verification step. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 2 tools
The two tools have clearly distinct roles: one returns format/schema information, the other performs the actual trace evaluation. The descriptions explicitly instruct when to use each, leaving no realistic ambiguity.
Both names use a consistent snake_case verb_noun pattern: get_trace_schema and evaluate_trace. The convention is predictable and readable.
Two tools is thin for most servers, but here one is a schema/introspection helper and the other is the core evaluator. It is borderline minimal rather than clearly wrong.
The surface covers the core workflow: learn the input format, then evaluate a trace. Minor gaps exist around comparing multiple evaluations or retrieving history, but agents can work around these by recording scores themselves.