agent-tracedown

Find the step that caused your AI agent's failure.

Your tracing tool shows what your agent did. Tracedown tells you which step broke it, shows the evidence, suggests a fix and proves whether it works. Open source, and it works on top of the traces you already have.

Works with LangGraph · CrewAI · OpenAI Agents · Claude Agent SDK · Google ADK · Pydantic AI · Vercel AI SDK · MCP · Langfuse · Arize Phoenix · OpenTelemetry

Try the live demo · Quick start · How it works · Integrations · Benchmarks · Comparison · Docs

A real run from tracedown_server/demo/agents.py. Every verdict shown is Tracedown's actual output. Watch in HD, or click through it yourself (the free host can take a minute to wake up).

Why Tracedown

AI agents rarely crash. They fail quietly: a model writes "Friday" where a date belongs, a retriever returns last year's policy, a loop burns forty tool calls. Tracing tools record every step, and then leave you to work out which one broke the run.

Tracedown is the flight recorder that also reads the recording. It walks back from the symptom to the step that caused it, and shows the evidence.

• Works with what you have. Import failed runs from Langfuse, Arize Phoenix or any OpenTelemetry exporter with no code changes, or add one line to 13 frameworks.

• Explains the failure. The root cause, the evidence, and a verdict on who was at fault: the agent, or the user.

• Replays with your fix. Earlier steps come from the recording; from the broken step on, the agent runs live with your change. Needs a run recorded with the SDK.

• Proves the fix works. Tracedown suggests a fix, replays the run with it three times, and checks that the problem is gone and nothing new broke: tdn.try_fixes(trace_id, agent=...). No tool runs for real during the check. Python SDK for now.

• Runs without an LLM. Diagnosis is rules first: instant, free and private. An inexpensive LLM judge is optional.

• Measured. Accuracy on a public, human-labelled benchmark is published, losses included.

• Free and self-hosted. Apache-2.0, every feature, including SSO, roles, audit log and scale-out storage.

Related MCP server: Moth

Quick start

See it work. One command starts the server and the web app, loads eight agents that fail in realistic ways (from a stale document to a prompt injection), and opens your browser. No API key, Node or Docker needed.

pip install agent-tracedown-server
tracedown up                                      # http://127.0.0.1:7070

If 7070 is in use, tracedown up tries 7071 through 7079 and prints and opens the chosen URL. Use that port for your SDK endpoint too. Set --port or TDN_PORT to require a particular port; an occupied explicit port still stops startup.

Needs Python 3.10+. A Mac's built-in Python is 3.9; with uv instead: uvx --python 3.12 --from agent-tracedown-server tracedown up.

Prefer the terminal? tracedown demo prints every diagnosis instead.

Diagnose the runs you already have. No code changes: paste an export into Diagnose a trace in the web app, or import from the command line. A 5-minute tutorial walks through it.

export LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-...
tracedown-import langfuse --errors-since 24h      # failed runs from the last day
tracedown-import otlp traces.json --failed        # or any OpenTelemetry export

Imported runs get the full diagnosis. Replay needs a run recorded with one of the SDKs below, because it runs your agent's own code.

Instrument your own agent. The SDKs have no dependencies.

import agent_tracedown as tdn                     # pip install agent-tracedown
tdn.init(project="support-bot", endpoint="http://localhost:7070")

with tdn.run("CalendarAssistant", input=message) as run:
    reply = tdn.llm_call(call_model, model="gpt-4o-mini", prompt=messages)
    tdn.tool_call("create_event", args, create_event, schema=SCHEMA)
    run.output(reply)

import * as tdn from "agent-tracedown";           // npm install agent-tracedown
tdn.init({ project: "support-bot", endpoint: "http://localhost:7070" });

await tdn.run("CalendarAssistant", { input: message }, async run => {
  const reply = await tdn.llmCall(callModel, { model: "gpt-4o-mini", prompt: messages });
  await tdn.toolCall("create_event", args, createEvent, { schema: SCHEMA });
  run.output(reply);
});

The TypeScript and Python SDKs record the same data, and the same run gets the same diagnosis from either.

Run it for a team. tracedown up is one process on SQLite. For Postgres, several workers, Docker Compose or Kubernetes, see Deployment:

git clone https://github.com/nbn07/agent-tracedown
cd agent-tracedown/deploy && docker compose up -d   # web app on :3000, API on :7070

How it works

How Tracedown decides walks through each stage, with its limits. Every failed run gets an AI stack trace you can paste into an issue:

AgentFailureException: Invalid tool arguments
  at        step 5  PlannerAgent → execute_tool calendar_create_event
              start must be ISO-8601 date-time, got 'Friday'
  caused by step 4  PlannerAgent → chat gpt-4o-mini
              PlannerAgent produced 'Friday' at step 4, which calendar_create_event() could not use
  root cause: Bad LLM output (top suspect; check the evidence)
  evidence:   step 4 output 'Friday' is used verbatim in step 5 arguments
  next:       replay from step 4 with your fix: tdn.replay(bundle, mode="from-step", from_step=4)

Beyond single runs, Tracedown checks each claim in an answer against its sources, compares runs across prompt and model versions, groups failures into incidents with Slack, PagerDuty or GitHub alerts, and shows cost per run, step, model and agent.

Integrations

Each adapter is tested in CI against the real library: LangGraph 1.2, openai-agents 0.24, claude-agent-sdk 0.2, google-adk 2.13, pydantic-ai 2.64, mcp 1.30 and 2.3, CrewAI 1.15, AutoGen 0.7, Semantic Kernel 1.46, LlamaIndex 0.14, Vercel AI SDK 5 to 7, and @openai/agents 0.21. Python adapters live in agent_tracedown.integrations.

Tracedown also exports: OTLP to Jaeger, Tempo, Datadog or any collector, and Prometheus metrics at /metrics.

Ask your coding assistant. tracedown mcp lets Claude Code, Cursor or any MCP client ask "why did this run fail, and what fix works?" (setup):

claude mcp add tracedown -- uvx --from "agent-tracedown-server[mcp]" tracedown mcp

Benchmarks

Who&When (ICML 2025) asks which step caused each of 184 failed multi-agent runs, labelled by people. Exact match, one run, no tuning on this data:

The first guess is wrong more often than it is right, so treat it as the first place to look and check the evidence. On the algorithm-generated set, rules alone do no better than a random pick within the top three; the judge is what lifts it. How Tracedown decides explains the method and where it fails.

On τ-bench failures, Tracedown's first guess ties a simple rule ("blame the first database change") at 74%, and its top three contain the right step 81% of the time. It used to lose 68% to 74%; the fix was found on the development split and scored once on the test split. The method and every change after the first scoring are in LABELING.md. The judge's answers are saved in the repository, so re-scoring needs no API key.

Comparison

Tracedown is a root-cause layer that works alongside your tracing tool, not a replacement for it.

AgentDebugX is the closest project. Scored the same way on Who&When (agent and step both right, all 184 runs), Tracedown reaches 36.4% with a Claude Sonnet 5.5 judge and 31.5% with Claude Haiku 5.5; AgentDebugX with Qwen 3.5 9B reaches 28.8%. The models differ, so this compares setups, not methods.

Based on each project's documentation, the AgentDebugX paper and Arize's 2026 comparison, October 2026. Something out of date? Open an issue.

Architecture

flowchart TB
    subgraph sources["Your agents and traces"]
        direction LR
        py["Python SDK"] ~~~ ts["TypeScript SDK"] ~~~ imp["tracedown-import<br/>Langfuse · Phoenix · OTLP"]
    end
    subgraph server["Tracedown server"]
        direction LR
        api["Ingest API"] --> q[("Kafka<br/>optional")] --> pipe["Analysis<br/>root cause · fault · claims<br/>drift · incidents · cost"]
    end
    subgraph outputs["Results"]
        direction LR
        store[("SQLite or Postgres<br/>ClickHouse at scale")] --> web["Web app"]
        alerts["Slack · PagerDuty · GitHub<br/>OTLP · Prometheus"]
    end
    sources --> api
    pipe --> store
    pipe --> alerts

It starts as one process on SQLite and scales out with Postgres, Redis, Kafka and ClickHouse. A Helm chart is included.

helm install tracedown deploy/helm/tracedown

Documentation: Wiki · How it decides · Deployment · Configuration · Design · Langfuse tutorial · Python SDK · TypeScript SDK · Telemetry spec

Roadmap

• Proven fixes in TypeScript, and suggestions worded by the LLM judge when one is turned on

• GitHub Action that runs your agent's test conversations on every pull request

• Write-back to Langfuse and Phoenix, so the verdict appears in the tool you already use

Ideas are welcome in Discussions; requests with a real failing run behind them get priority.

Using Tracedown?

If Tracedown found a bug for you, or you run it at work, say so in Show and tell. With your permission your team's name goes here. It is the most useful thing you can do for the project after reporting a wrong diagnosis.

Contributing

The most valuable contribution is a run Tracedown diagnoses wrongly: report a wrong diagnosis and it becomes a benchmark case that every later version is tested against. Detectors, framework adapters and documentation are welcome too. Start with CONTRIBUTING.md or an issue labelled good first issue: each one names the files to change and how to test. Questions go to Discussions. The wiki explains how the code fits together, with step-by-step guides for a first pull request, a new detector or a new framework integration.

Nothing to install: open a ready-made development environment in your browser, with every dependency set up and the tests passing.

pip install -e ./sdk/python -e "./server[dev]" && pytest

Contributors

Thank you to everyone who has made Tracedown better:

Your name goes here with your first merged pull request.

License

Apache-2.0, all of it: the SDKs, the server, the root-cause engine, the web app and the enterprise features. There is no paid edition.