afr
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@afrHave we hit this ModuleNotFoundError before? Search recorded fixes"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
agent-flight-recorder
A flight recorder and experience-retrieval engine for coding agents. It records what agents observably do, reconstructs task attempts, separates failed attempts from objectively verified outcomes, keeps evidence-bounded lessons, and serves them back (CLI, MCP, or an opt-in Claude Code failure hook) before an agent repeats a known mistake.
It is not a chat-memory store or a transcript summarizer. Its research question is:
Does retrieving previously verified execution experience help an agent solve similar tasks with fewer repeated failures, tool calls, time or tokens?
This repository provides the instrument to answer that: capture, verification, hybrid retrieval, feedback logging, an offline retrieval benchmark and a paired live-agent runner. It does not answer the question yet. No live-agent experiment has been run, and nothing here claims that agents do better with it.
Quick start
Requires Python 3.12+ and uv. Docker is not required.
git clone https://github.com/HelgiHelgasonCMD/agent-flight-recorder && cd agent-flight-recorder
uv sync --extra dense # CPU-only torch + sentence-transformers
uv run afr demo # both demos, in a throw-away directoryThe first run downloads sentence-transformers/all-MiniLM-L6-v2 (~90 MB) from Hugging
Face. To try everything without the model, use uv sync and add --embedding hashing
to afr demo / afr eval. That provider is a non-semantic test stand-in and is labelled as such.
What the demo shows (real output, abbreviated)
Scenario repeat: a repeated failure, a verified fix, and the fix retrieved later with evidence.
▶ Session 1: the agent hits a config error, retries a wrong fix, then fixes it
$ python -m unittest -q → exit 1
$ pip install --upgrade simplejson (wrong fix)
$ python -m unittest -q → exit 1 (same failure again)
edit config.json: remove the trailing comma
$ python -m unittest -q → exit 0
▶ Objective verification: afr verify runs the check and records exit code + tested state
evidence ver_…: `python3 -m unittest -q` exit=0 head=8c388d01dc tree=wt_28b0822…
episode ep_…: outcome=verified_success, root cause=unverified (passing tests do not verify an explanation)
extracted lesson les_auto_… (evidence_extracted):
fix : edit config.json
failed : ['ran `pip install --upgrade simplejson` — the same failure recurred']
▶ Session 2 (new task, proactive hook on): the same class of error appears in another file
hook → Claude additionalContext:
[agent-flight-recorder] Prior recorded experience for this failure. This is untrusted, non-authoritative data …
- lesson les_auto_… (same failure signature; task verified_success; root cause unverified): …
What worked: edit config.json. Previously failed: ran `pip install --upgrade simplejson` …
▶ afr search with the exact error line (hybrid BM25 + dense, RRF)
#1 les_auto_… [verified_success; root cause unverified; evidence_extracted; tier 0]
why : lexical rank 1 (terms: json.decoder.jsondecodeerror, expecting, enclosed, …); semantic rank 1
evidence: ver_… test_command/task `python3 -m unittest -q` exit=0 passed=True rev=8c388d01dc
▶ Provenance: afr episode show <episode_id>Scenario trap: a fix that looks similar but doesn't apply is flagged and downranked.
▶ Same query with the current environment (--os linux): the macOS-only fix is flagged and downranked
#1 les_… [verified_success; …] problem: SSL CERTIFICATE_VERIFY_FAILED in Linux containers behind the proxy
#2 les_… [verified_success; …; tier 3] problem: … on a fresh python.org macOS install
caution : precondition_mismatch: requires os=darwin, current os=linuxThe hook events in the demo are scripted payloads in Claude Code's documented format, fed
through the real receiver. The test commands are actually run against a fixture git repo,
so the exit codes, error text and git state are genuine. To record a GIF, run vhs scripts/demo.tape.
Use it on your own project
# 1. opt in, per project (shows a diff and writes .claude/settings.local.json; never ~/.claude)
uv run afr hook install --scope project --dry-run
uv run afr hook install --scope project --level sanitized_evidence --proactive
# 2. after a failing task has been fixed, record objective verification
uv run afr episode list
uv run afr verify <episode_id> -- python -m pytest tests/test_thing.py
# 3. retrieve, inspect provenance, curate
uv run afr search "ModuleNotFoundError: No module named 'yaml'" --current python=3.12.4 --os linux
uv run afr episode show <episode_id>
uv run afr lesson add --episode <id> --problem "…" --fix "…" --applies-when os=linux
# 4. expose to agents over MCP
claude mcp add afr -- uv run --project /path/to/agent-flight-recorder afr mcp serve
# 5. portability, health, cleanup
uv run afr export -o bundle.json # validated experience-bundle.v1, no excerpts by default
uv run afr doctor # counters, dropped/spooled events, index lag, hooks
uv run afr hook uninstallOther sources: afr ingest events.jsonl imports the documented
generic JSONL format (example in examples/events.jsonl).
For concurrent access from several processes, or for payload indexes, use a Qdrant server:
run docker compose up -d, then set AFR_QDRANT_URL=http://localhost:6333.
Related MCP server: twiceshy
Architecture
Claude Code hooks / generic JSONL / bundles
│ fail-open receiver + spool redaction BEFORE persistence
▼
ingest pipeline: dedupe → capture level → redact/limit → failure signature → episode → observed re-run
▼
SQLite (WAL, Alembic) ── authoritative ──▶ idempotent sync ──▶ Qdrant (dense + BM25 sparse, derived)
▲ │
afr verify (exit code, git HEAD, working-tree hash) ▼
mandatory workspace filter → bm25 | dense | hybrid (RRF) → applicability policy
│
afr search · MCP tools · proactive PostToolUseFailure context · exportDetails are in docs/architecture.md, including the module map, design decisions and acceptance checklist. Data flow and security are in docs/security.md. Claude Code field mapping is in docs/claude-code-hooks.md, and the formats are in docs/formats.md.
Evidence model in one paragraph
An episode is a task attempt. Failures are normalized into stable failure signatures,
which make "the same error again" detectable across sessions. Outcomes are unknown,
claimed_success (the agent said so), verified_success or verified_failure. Only
objective evidence produces a verified status: afr verify exit codes, imported CI/check
results, or a previously failing command later succeeding in the same episode. Root cause
is tracked separately and is never upgraded by passing tests. Lessons restate observed
actions only. At metadata capture level the fix isn't visible, so it stays empty with a
reason rather than being invented. Every lesson links to its episodes, verifications and
signatures, and can be superseded or invalidated.
Why hybrid retrieval
Agent failures mix two kinds of signal:
Exact tokens:
ERR_REQUIRE_ESM,error[E0502],[import-untyped],--legacy-peer-deps,src/app/config.py,numpy==2.0.1. Embedding models blur these together. BM25 with a tokenizer that keeps compound tokens and their parts matches them exactly, and IDF rewards rare codes.Intent and symptoms: "my user can't talk to the docker daemon" vs
permission denied … /var/run/docker.sock. Lexical retrieval misses these, and dense retrieval catches them.
So afr runs both over the same documents and fuses them with reciprocal rank fusion inside Qdrant. RRF uses ranks only, so BM25 scores and cosine similarities, which are on incomparable scales, are never added together. An applicability policy then adjusts ranks using evidence: verified vs claimed, staleness, version and precondition match, and superseded or contradicted status. It is a bounded rank penalty, not a score blend.
On this repository's synthetic corpus, the measured case for hybrid is modest. See the next section.
Benchmark (synthetic retrieval only)
uv run afr eval loads benchmarks/corpus/ into an isolated store and runs 26 labelled
queries in each mode. The corpus is 31 synthetic but technically plausible episodes,
generated by benchmarks/build_corpus.py, with graded relevance
judgments and trap labels. The six case types are exact repeat, paraphrase, stale fix,
misleading prior failure, version mismatch, and similar symptom with a different root cause.
Leakage is controlled two ways:
time split: each query only sees records created before its
as_of; a run fails if a future record is returned;task-family split: dev and test are reported separately.
Real run on 2026-10-30: all-MiniLM-L6-v2, embedded Qdrant 1.20, Linux x86_64, CPU.
Full report: docs/benchmark.md. The server-mode run produced identical
quality numbers: docs/benchmark-server.md.
mode | Recall@1 | Recall@3 | MRR@10 | nDCG@5 | p50 ms | p95 ms | traps flagged | traps ranked below correct |
bm25 | 0.865 | 0.981 | 0.955 | 0.963 | 6.3 | 7.7 | 8/8 | 7/7 |
dense | 0.827 | 0.923 | 0.931 | 0.943 | 16.6 | 19.8 | 8/8 | 7/7 |
hybrid | 0.865 | 0.962 | 0.955 | 0.962 | 20.8 | 27.0 | 8/8 | 7/7 |
hybrid, no policy | 0.865 | 1.000 | 0.904 | 0.948 | 17.8 | 20.6 | 0/8 | 2/7 |
How to read it honestly:
The corpus is small (26 queries; one query moves MRR by about 0.04) and lexically easy, so BM25 alone matches hybrid overall. Dense alone is best on paraphrases (MRR 1.000 vs 0.917), and hybrid ties BM25 there. This corpus doesn't show hybrid beating BM25. Hybrid stays as the default because it doesn't lose to BM25 here while also covering paraphrased symptoms.
The applicability policy accounts for most of the measurable effect. It raises MRR@10 from 0.904 to 0.955, and trap ordering from 2/7 to 7/7. It also has costs:
a correct answer whose evidence is just past the 180-day staleness threshold loses a tie (q07);
a fix that is both stale and version-mismatched, and is the only answer, drops to rank 3 (q14).
These are retrieval metrics on synthetic fixtures. They are not agent-effectiveness results.
Live-agent effectiveness: not measured
afr eval live runs paired tasks: control vs retrieval arm, same model, prompt, budget,
timeout and tools. It records task success from the check command's exit code, tool calls and
repeated failures from afr's own hooks, elapsed time, and turns/cost/tokens only when the agent's
JSON output reports them. It refuses to run without --i-understand-cost. It has only been run
as --dry-run and with a fake agent script that tests the plumbing. There are no live
results in this repository.
Privacy and safety defaults
Opt-in, project-scoped hooks. A diff is shown before install, other hooks are preserved,
uninstallremoves only afr's handlers, and user-global config is never touched.metadatacapture by default: no prompts, transcripts, file contents, diffs or full tool inputs. Prompts and the final message are reduced to a hash and a boolean.Redaction (keys, tokens, auth headers, private keys, URL credentials, secret-named env values) runs before persistence, and again at indexing, retrieval and export. Each boundary has a test.
The workspace/tenant boundary is a mandatory pre-retrieval filter, not a ranking penalty.
Retrieved text is labelled untrusted and never executed. MCP and hook outputs say so explicitly.
The receiver is fail-open but observable: counters, a bounded spool, drop counts,
afr doctorand--replay-spool.
Development
uv sync --extra dense
uv run ruff check . && uv run ruff format --check . && uv run mypy
uv run pytest -q # 68 tests, embedded Qdrant, hashing embeddings
uv run pytest -q -m dense # real sentence-transformers model
AFR_TEST_QDRANT_URL=http://localhost:6333 uv run pytest -q -m "qdrant_server or not dense" # needs a server
uv run python benchmarks/build_corpus.py # regenerate the corpus (deterministic)CI (.github/workflows/ci.yml) runs lint, type checks, the tests, the
dense-model tests, the demo and the benchmark, plus a separate job against a qdrant/qdrant:v1.20.0
service container.
Limitations
Claude Code delivery not exercised live. Payload handling is tested with the documented example payloads (Claude Code 2.1.314 docs); hook delivery from a live, credentialed session wasn't run here.
Bash success has no exit code.
PostToolUsecarries none. afr recordssucceeded=truewithout inventingexit_code=0. Observed re-run verification uses this documented behaviour.Embedded Qdrant is single-process. It holds a file lock, ignores payload indexes, and its latency isn't representative of a server. Use a server for concurrent hooks + MCP + CLI.
BM25
avgdlis fixed per collection. It's set at first build;afr index rebuildrefreshes it.The policy only knows recorded preconditions.
applies_when(os, version specifiers), recorded versions, signatures and status. Two causes that share an error code and have no recorded precondition (e.g. two different TS2307 causes) aren't told apart beyond lexical relevance.Prompts are never stored, so Claude Code episodes have no task summary. Add one with
afr episode annotateor a reviewed lesson.The benchmark is small and synthetic. It is useful for regression and ablation, not for absolute claims.
Backlog
A Codex CLI importer. It should be built only after verifying the event schema of the installed Codex version (
codex exec --json). It isn't implemented, and no Codex format is assumed anywhere.Run the paired live evaluation at a meaningful sample size, and publish the raw run records and analysis.
A larger, partly real-world corpus (opt-in, redacted bundles) and harder paraphrase sets.
Optional reranker stage (cross-encoder) behind the same rank-only contract; measure with
afr eval.OpenTelemetry spans for ingest, sync and search; an optional FastAPI read API.
An end-to-end hook smoke test with a real Claude Code session in CI, behind a secret.
License
MIT. See LICENSE.
Available Tools
3 toolsget_episode_evidenceA
Return the recorded evidence for an episode: outcome, verification records (command, exit code, git revision, working-tree fingerprint), failure signatures, tool-call metadata and lessons citing it.
| Name | Required | Description | Default |
|---|---|---|---|
| episode_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. "Return the recorded evidence" implies a non-mutating read and it details the evidence payload categories, but it says nothing about behavior when the episode is unknown, whether evidence can be partially missing, permissions required, or cost/latency. Adequate but leaves real behavioral gaps for a zero-annotation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with the action first and the returned categories enumerated after the colon. Every listed item is substantive evidence content, not filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple (one required id) and an output schema exists, so return values need not be spelled out, yet the description still summarizes them helpfully. The only notable omission is how to obtain a valid episode_id and what happens on a missing episode.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Only one parameter, and the phrase "for an episode" ties episode_id to an episode identifier, so the description adds a small amount of framing. However it does not say where the identifier comes from (e.g., from report_task_outcome) or its format, and schema coverage is 0%.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ("Return") and resource ("the recorded evidence for an episode") and enumerates the evidence categories returned (outcome, verification records, failure signatures, tool-call metadata, lessons). It does not explicitly contrast itself with search_experience or report_task_outcome, but the retrieval-by-id framing is distinct from a search or a write.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: the required episode_id makes clear this is a point lookup for a known episode, and the sibling names suggest search_experience is for discovery rather than direct fetch. The description never states when to prefer this over search_experience, nor any preconditions such as the episode having been recorded first.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
report_task_outcomeA
Record which retrieved lessons you actually consulted and how the task ended.
reported_outcome is your own account (succeeded/failed/abandoned/unknown) and is
stored as a self-report. The objective outcome is taken only from verification
evidence on the linked episode (if episode_id is given), never from this report.
| Name | Required | Description | Default |
|---|---|---|---|
| note | No | ||
| task_ref | Yes | ||
| episode_id | No | ||
| retrieved_ids | No | ||
| reported_outcome | Yes | ||
| consulted_lesson_ids | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it discloses the most important behavioral trait: reported_outcome is a self-report that is never treated as the objective outcome, which comes only from episode verification evidence. That is genuinely non-obvious semantics an agent could not get from the schema. It still does not say whether repeat reports overwrite prior ones or what permissions are required.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the purpose and followed by the one caveat that matters. No filler, though the second sentence's line-broken formatting is slightly awkward rather than harmful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need no explanation, and the self-report caveat is well covered. However, for a 6-parameter write tool with no annotations and zero schema descriptions, leaving task_ref and the retrieved_ids vs consulted_lesson_ids distinction unexplained is a real gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it partly does — it enumerates the accepted values for reported_outcome and explains that episode_id links to verification evidence. But task_ref, retrieved_ids, and note are never explained, and the distinction between consulted_lesson_ids and retrieved_ids is left unstated.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb and object pair — recording which retrieved lessons were consulted and how the task ended — which is more than a restatement of the name. It does not name either sibling (search_experience, get_episode_evidence), so it only implicitly marks itself as the write-back counterpart to those read tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: the tool is meant to be called at task end, after lessons have been retrieved and consulted. There is no explicit 'use this when / not when' clause and no reference to the sibling tools that produce the ids this tool consumes, so the agent must infer the workflow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_experienceB
Search recorded experience (lessons and failed episodes) for an error or task.
Paste the exact error line for best lexical matching. Each hit carries
episode/lesson ids, verification status, evidence references, why it matched,
prerequisites and cautions. Provide current_os/current_versions (e.g.
{"python": "3.12.4"}) so inapplicable fixes are flagged.
| Name | Required | Description | Default |
|---|---|---|---|
| k | No | ||
| mode | No | hybrid | |
| tool | No | ||
| query | Yes | ||
| repo_id | No | ||
| language | No | ||
| current_os | No | ||
| error_type | No | ||
| verified_only | No | ||
| current_versions | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the behavioral burden and does disclose meaningful traits: lexical matching behavior, the contents of each hit, verification status, evidence references, why it matched, prerequisites, cautions, and applicability flagging via version context. It still does not explicitly confirm read-only behavior or mention pagination, rate limits, or permissions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the main purpose and gives practical query advice early. It is reasonably concise, though the list of hit contents is somewhat verbose given that an output schema exists.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the purpose and some key input guidance, and an output schema exists so return values need not be fully explained. But for a 10-parameter tool with no annotations and 0% schema description coverage, leaving most parameter semantics and all sibling routing unaddressed is a clear completeness gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 10 parameters, so the description must explain parameter meaning, but it only addresses query, current_os, and current_versions. It leaves k, mode, tool, repo_id, language, error_type, and verified_only unexplained, which is a significant gap for a 10-parameter search tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: searching recorded experience, specifically lessons and failed episodes, for an error or task. It is clear what the tool does, but it does not explicitly distinguish itself from sibling tools such as get_episode_evidence or report_task_outcome.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives useful usage advice for the query parameter by recommending pasting the exact error line, and it explains why current_os/current_versions should be supplied. However, it does not say when to use this tool versus sibling alternatives, nor does it give conditions for when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
3 tool updates
v0.1.0- First observed
get_episode_evidence - First observed
report_task_outcome - First observed
search_experience
TDQS
Scored across 3 tools
Each tool has a clearly distinct role: search retrieves experiences, get_episode_evidence fetches evidence by episode id, and report_task_outcome records an outcome. The descriptions make the read/search/write boundaries explicit, so an agent can reliably select the right tool.
All tools follow a consistent snake_case verb_noun pattern: search_experience, get_episode_evidence, report_task_outcome. There are no mixed conventions or vague names.
Three tools is appropriately minimal for this focused experience-retrieval and reporting server. Each tool serves a distinct, necessary function without redundant operations.
The set covers searching experiences, retrieving evidence, and reporting outcomes, but lacks any explicit way to create or record new episodes/lessons or add verification evidence. Those omissions are notable for a system centered on recorded experience, even if ingestion may happen outside the MCP surface.
Maintenance
Related MCP Connectors
Agent-native MCP server over the public saagarpatel.dev corpus. Read-only, stateless.
Team memory for AI coding agents and code reviews, reviewed by humans and served over MCP.
Capability registry for the agentic economy. Semantic search over verified MCP server listings.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceAn MCP server enabling AI agents to record, query, and share structured problem-solving experiences with human review and confidence decay.2 npmMIT
- AlicenseNot gradedqualityDmaintenanceA self-hosted MCP server that retrieves git-backed engineering experience records (issues, fixes) to inform LLM coding agents, preventing repeated mistakes with a relevance floor.AGPL 3.0
- FlicenseNot gradedqualityDmaintenanceMCP server for a shared Postgres-backed knowledge base with hybrid retrieval and agentic RAG, enabling coding agents to upload, search, and ask questions over documents with cited answers.-
- AlicenseAqualityBmaintenanceMCP server that gives coding agents persistent, verified memory of codebase decisions, conventions, and skills, with evidence-based claims that are re-checked via git hooks and human-gated review. Enables memory search, propose/approve, chat harvesting, and critique across MCP-compatible tools.21112 npm1MIT