agentarch

A retrieval-grounded multi-agent system that you can replay exactly from a trace, exposed over MCP.

The goal is not another RAG demo. It is the engineering that separates a demo from a system: retrieval that is measured on its own before any generation is judged, an orchestrator whose state is explicit and whose runs are byte-for-byte reproducible, every generated claim tied to a retrieved chunk, and a check that fails when a claim has no support.

Status: the offline system is complete. Eight slices, built thin and vertical, each leaving the quickstart working: the determinism kernel (effect boundary, trace format, strict replay), the retrieval layer with its own standalone evaluation, the dense arm plus hybrid fusion, measured against that baseline, the multi-agent orchestrator — which runs end-to-end offline and replays exactly from its own trace — the MCP surface, in both directions (a stdio server any MCP client can drive, and an agent whose retrieval is answered by calling a tool on another MCP server), the observability layer and its replaying trace viewer, the Tier-1 grounding gate with the corrupting runner it is measured against, and the judge-validation layer, wired to judgecheck (the judge-reliability project), whose negative result killed Tier 2.

Four things are deliberately absent, each with its reason stated where it is felt: a live model path (the default path is offline by design; a provider is opt-in and none is wired), Tier-2 entailment (every offline candidate failed its own audit), a reranker, and end-to-end task-success evaluation (against an extractive stub it would measure the stub). The slices say where each part stands, DESIGN.md says how it is put together, and nothing in this README describes behaviour that is not in the repo.

The shape of the system

flowchart LR
    MCPC[any MCP client] --> SRV[MCP server]
    SRV --> ORCH
    subgraph ORCH [orchestrator]
        direction LR
        P[planner] --> W[retrieval workers] --> S[synthesizer] --> C[critic]
    end
    W --> RET["retrieval: BM25 + LSA + fusion"]
    ORCH -. every model, retrieval and clock call .-> EB[effect boundary]
    RET -.-> EB
    EB --> TR[("trace, JSONL")]
    TR --> REP[strict replay]
    TR --> VIEW[trace viewer]
    TR --> GATE[grounding gate]

One invariant carries the design: nothing above the effect boundary touches the outside world except through it, and everything that crosses it is journalled. That is why a run replays exactly with nothing running, why the viewer can render a run it cannot re-execute, and why the grounding gate can refuse a months-old run from its trace alone. The gate and the viewer hang off the trace, not off the orchestrator — a check the model can argue with is not a gate. The fuller diagram, with the failure modes and the state each hand-off carries, is DESIGN.md §2.

If you read one section. agentarch is a retrieval-grounded multi-agent system in which every model, retrieval and clock call crosses one journalled effect boundary, so a recorded run replays byte-for-byte with nothing running and refuses to replay once the code's behaviour has changed (replay_demo.py). Retrieval was measured before anything generated: BM25 over overlapping 100-word chunks reaches R@2000w 0.629 [0.522, 0.731] on 56 queries, the baseline everything later had to beat (retrieval-baseline.md). Adding a dense arm and fusion did not beat it by more than the noise: the best fusion configuration's paired gain is 0.031 [0.000, 0.076], an interval that includes zero (hybrid.md). The grounding gate catches five injected lies, passes two controls, and flags 0 of 13,899 claims in a false-alarm sweep over the corpus (grounding_demo.py). No offline entailment judge was fit to ship as Tier 2: the best judge that answers every item reaches κ = 0.8176 [0.7965, 0.8393], and every candidate scores 0/198 on claims whose one edit is a negation (judge.md).

Each layer ships with the check that proves it, and every one runs offline from a clean clone — with one disclosed exception: the judge row needs judgecheck, the judge-reliability project, installed from its own repository, and Python 3.12 (see the Quickstart):

Related MCP server: mcp-production-mastery

Why the determinism kernel came first

Every other feature is easier to build on top of a run you can reproduce, and impossible to debug without one. So the first slice is the constraint that makes reproduction possible:

All non-determinism — model calls, tool calls, the clock, the RNG — goes through one interface.

from agentarch import EffectRequest, RecordingRunner, Trace, RunHeader, verify_replay

rec = RecordingRunner(live_runner)
answer = my_agent(rec)                       # the agent only talks to the runner
trace = Trace(header=RunHeader(run_id="r1", config_digest=cfg), records=rec.records)

# later, on another machine, with no network and no API key:
assert verify_replay(trace, my_agent) == answer

verify_replay is strict on purpose. It fails if the code requests a different effect, requests effects in a different order within a lane, requests an extra one, or stops early. A replay that tolerates a changed request proves nothing; a replay that fails loudly turns any recorded run into a regression test for agent behaviour — which is what an agent system actually lacks.

Retrieval, measured before anything generates

A generator cannot cite what was never retrieved, so retrieval is evaluated on its own first. The corpus is the MCP specification and the Python SDK documentation, vendored with per-file provenance (corpus card); the gold set is 56 questions labelled with the corpus sections that answer them (datasheet).

.venv/bin/python tools/eval_retrieval.py --out eval/results/retrieval-baseline.md

Everything below regenerates from a clean clone with no network and no key. The full report, including the whole sweep, is in eval/results/retrieval-baseline.md.

The finding worth reading. Sweeping chunk size and ranking by recall@10 says bigger chunks are better, monotonically. Holding the context budget fixed instead says the opposite:

At fixed k, a 1000-word configuration is handed roughly nine times more text than a 100-word one, so R@k mostly rediscovers that more text retrieves more answers. Both orderings are separable from zero under a paired bootstrap, in opposite directions. Chunking papers and blog posts that report recall@k across chunk sizes without a matched budget are measuring this artefact.

Other honest results from the same run:

• BM25's k1/b grid is flat on this corpus — the 20-cell grid spans 0.507 to 0.537 on R@10, a range far smaller than any single cell's confidence interval. The parameters most tutorials fuss over do not matter here; the analyzer choices do, slightly.

• Heading-aware chunking loses to fixed windows, which was not the expected result.

• Retrieval is much worse on questions needing more than one section (R@10 0.42) or spanning both sources (0.45) than on single-section ones (0.65).

Numbers only mean something with their instrument characterised, so: 56 queries is small; the labels were generated with a language model and checked for internal consistency, and nobody independent checked whether a label is the right answer. A re-labelling of 12 queries agreed on the primary answer 12/12 but had a mean Jaccard of only 0.44 across the full label sets. This is a self-consistency check, not an independent validation. The original labels and the re-labelling were generated with language models of the same family, from the same corpus and a similar brief, so correlated blind spots will not show up in it. The datasheet says what that does and does not license you to conclude.

Hybrid search, and the result I did not want

The dense arm is latent semantic indexing computed from the shipped corpus — a truncated SVD of the tf-idf term-chunk matrix, numpy and nothing else. Not a neural embedding, and this repo never calls it one. It exists because the design constraint is that a reviewer can clone and run with no key and no download, and precomputed embeddings do not cover a query typed at runtime. The full argument, including what LSA cannot do, is in lsa.py.

Two fusion rules are implemented — reciprocal rank fusion and normalised score fusion — and swept over pool depth and arm weighting, 32 configurations. Then measured against the BM25 baseline that was published before any of this existed:

The best fusion beats BM25 by 0.031 [0.000, 0.076] paired — an interval that includes zero. On this corpus and this gold set, hybrid search was not shown to help. That is reported as the headline of eval/results/hybrid.md rather than buried, because a repo whose whole argument is measurement discipline does not get to only publish the wins.

Three things the measurement did surface:

• Recall and ranking quality come apart. The dense arm matches BM25 on recall at a matched context budget (Δ 0.001 [-0.076, 0.077]) while scoring worse at ordering (ΔnDCG@10 -0.038 [-0.074, -0.001] — an interval that excludes zero, but only just, and it is one of ~40 uncorrected comparisons on this page, so read it as suggestive). It gets as much of the answer into the window and then puts it lower down. That asymmetry is the argument for fusing rather than replacing.

• The ceiling is not the fusion rule. An oracle that picks the better arm per query scores 0.693 — 0.03 above the fusion. (That bounds arm selection, not fusion: interleaving can beat both arms on a query. It does not do so on any of these 56, so here it is an empirical ceiling rather than a theorem, and the report says it that way.) Meanwhile both arms score zero on 9 of 56 queries — 0.16 of recall locked behind questions neither arm can see, five times the 0.03 sitting in the gap. The next move is a better arm or a reranker, not more fusion tuning.

• The LSA rank curve is flat. Only two of the other six ranks are separable from the one selected. Picking rank 192 was choosing the top row of a noisy curve, and the report says so.

The orchestrator: four named ways for a run to end

Planner → retrieval workers → synthesizer → critic, with typed artifacts on every hand-off and a bounded revision loop. Run it:

.venv/bin/python examples/agent_demo.py

1. asked: How do I register a tool and handle its errors?
   outcome  : complete — critic accepted the draft
   rounds   : 1, steps: 5
   effects  : {'main': 3, 'worker/0/0': 1, 'worker/0/1': 1}
   partial  : False

   ## How do I handle its errors

   - **Protocol Errors** indicate issues with the request structure itself that models are
     less likely to be able to fix: [3]
   ...
   [3] spec/server/tools — `spec/server/tools::32`

   (that bullet is wrapped here; the demo prints it on one line)

The planner split that question in two, so two workers ran on two separate lanes — replay is order-strict within a lane and makes no claim across them. Stated precisely: the workers run sequentially today and each issues one retrieval, so each lane holds exactly one record. What lanes buy right now is that reordering the workers in a round is not a divergence; the property is designed for a parallelism that does not exist yet, and DESIGN.md §3 says so.

A run ends complete, incomplete, stalled or failed, never ambiguously, and partial is a field on the result so a caller that ignores the outcome still cannot mistake a capped answer for a finished one. Three of the four are reachable on the shipped corpus — failed needs an effect to raise, which the offline handlers cannot do, so it is exercised by tests with injected failures:

4. asked: What is the zzzqqx protocol and how do I frobnicate a widget?
   outcome : stalled — planner returned no sub-questions in round 2
   rounds  : 2 (it re-planned, then ran out of new questions)
   gaps    : ['retrieval returned no chunks for this sub-question', ...]

5. the same answerable question under a 3-step budget
   outcome : incomplete — step budget exhausted after 3 steps
   partial : True

The run replays from its trace file with nothing else present — no corpus, no index, no agents, no clock:

2. wrote run-6de99385d564.jsonl (32226 bytes) and replayed it
   no index was built, no agent ran, no clock was read
   identical RunResult: True

3. replay the same trace under a different config (k=5 -> k=8)
   caught: replay diverged on lane 'main' at seq 0: config mismatch …

Three design points behind that. The trace carries each retrieved chunk's text, not just its id — larger files, in exchange for a trace that is self-sufficient. Every model request carries the digest of that role's task specification, so editing what an agent is asked to do breaks every trace recorded under the old wording rather than replaying green; there is a test for exactly that. And the one case that does not replay is named rather than glossed: a run that died at the effect boundary leaves a truncated journal, because failed effects are deliberately not recorded, so replay() refuses it with UnreplayableRun instead of reporting a divergence that would blame the code for a provider outage. A run that failed on a malformed hand-off does replay — the bad reply is in the journal — and both halves have tests.

The agents on the offline path are not a language model. The planner splits compound questions syntactically, the synthesizer is extractive — it quotes sentences that already exist in retrieved chunks — and the critic runs deterministic checks. They exist so the architecture is exercisable with no API key, and so a live path has an honest floor to be measured against. The answers they produce are correspondingly shallow; see Limitations.

MCP, in both directions

The system is an MCP server, and it is also an MCP client — the same protocol running both ways, built on the official Python SDK.

Outward. python -m agentarch.mcp speaks stdio, which is how MCP clients launch servers. The config names the venv's interpreter, because the client launches the server from its own working directory and a bare python there may not be the one agentarch is installed into:

{ "mcpServers": { "agentarch": { "command": ".venv/bin/python", "args": ["-m", "agentarch.mcp"] } } }

Every run is also a resource at trace://<run_id>, and a client that opened a subscriptions/listen stream gets a ResourcesListChanged event when one appears — that is the 2026-07-28 replacement for the old notifications/resources/list_changed, and there is a test that subscribes and waits for it rather than assuming the call did something.

answer_question returns partial and outcome beside the prose, because a transport that shipped only the answer text would let a client present a budget-capped run as a finished one — the exact failure the orchestrator's budgets exist to prevent.

Inward. The orchestrator's retrieval:search effect can be answered by calling search_corpus on another MCP server instead of an in-process index. The property that makes this more than symmetry:

8. now the other direction: the agent's retrieval served over MCP
   in-process retrieval :     9.4 ms   complete
   over stdio JSON-RPC  :  1808.8 ms   complete   (includes launching the server and building its index)
   mean per search call, 50 calls each after 5 warm-up:
     in-process           :    0.81 ms
     over the open session:    5.41 ms
   byte-identical trace : True

   The transport is below the effect boundary, so it is not in the journal.
   That is why a run retrieved over MCP replays with no server present:
   replayed the MCP run with no session, no server, no index: True

Real output of examples/mcp_demo.py, one of the workflow's demo steps — one run of it; the millisecond figures are wall-clock and vary, the last line does not. The trace of a run whose retrieval crossed a subprocess boundary and JSON-RPC framing is byte-identical to the same run done in-process, and replays with nothing running. That is the effect boundary doing its job: a retrieval:search effect is a query and a set of hits, and where the hits came from is below it. The overhead is printed rather than hidden. The whole run over stdio takes well over a second because it includes launching the server and building its index; once the session is open, the mean per search call lines put one search at a few milliseconds over JSON-RPC against about one in-process. That is a bad trade for this corpus and the right one when the corpus belongs to somebody else.

The SDK is an optional extra (pip install -e ".[mcp]") so the core stays a three-dependency install. The workflow installs it and sets AGENTARCH_REQUIRE_MCP=1, which turns the MCP suite's importorskip into a failure — otherwise "optional" quietly becomes "untested".

Seeing a run: the trace viewer

If you cannot see what the agent did, you cannot debug it. The trace has been the substrate since the first slice; agentarch.observe is the render that lets a reader see a run without reading JSONL:

.venv/bin/python -m agentarch.observe path/to/run-6de99385d564.jsonl              # text
.venv/bin/python -m agentarch.observe path/to/run-6de99385d564.jsonl --html run.html

The HTML render is one self-contained file — no JavaScript, no external assets — with the per-lane/per-effect roll-up, every effect's request and response, and each claim's citation shown as its exact character span highlighted inside the full text of the chunk the run retrieved. The chunk text comes from the trace itself, so the viewer needs no corpus and no index.

Two decisions worth knowing about before trusting it:

• The answer section is produced by replaying the trace, not by reading its labels. If the code has changed since the run was recorded, the viewer says the run no longer reproduces instead of rendering an answer the current code would not produce. The timeline and roll-up render regardless — a truncated journal from a run that died at the effect boundary still gets its timeline, because a broken run is the one you most need to see.

• Token counts always, currency almost never. The cost report prices usage only from a table (src/agentarch/observe/prices.json) whose every entry must carry a source URL and a retrieval date — and that table ships empty, so out of the box you get token counts and an itemised explanation instead of a dollar figure. The offline path reports no usage at all, and the report says so rather than printing $0.00: "no model was involved" is not "this run was free".

Per-effect durations exist when a run is recorded with effects.TimingRunner — deliberately not the default, because a wall-clock stamp would break the byte-identical-traces property the MCP demo checks. examples/observe_demo.py shows a timed run, both renders, and the cost report's refusals, and every step asserts.

Grounding: the check that fails the run

Every claim carries a citation to a span inside a chunk the run actually retrieved, and a deterministic gate refuses the answer when one does not:

.venv/bin/python -m agentarch.grounding path/to/run-6de99385d564.jsonl
# 0 = grounded · 1 = a claim failed the gate · 2 = could not check at all

Three checks, all mechanical, no model involved: a claim must have a citation; every citation must name a chunk this run retrieved and point at a non-empty span inside it; and every quoted span, number and code-like identifier in the claim must appear verbatim in a cited chunk. It runs over a stored trace with nothing else present — no corpus, no index, no agents — because it replays the journal to get the claims. That is what makes it a check someone who does not trust the run can re-run months later. What it proves is consistency, not provenance: the trace is unsigned and every digest in it is a plain sha256, so a producer who rewrites the whole file — chunk texts, claims and digests together — gets a trace this gate passes. Against a malicious producer the trace itself has to be trusted, or the chunk texts re-verified against corpus/manifest.json where the corpus is present, which this gate does not yet do.

It is deliberately not a step in the orchestrator's loop. The critic already reviews the draft, and on the live path the critic is the model; a check the model can influence is not a gate.

What it actually catches — measured, not asserted

The offline synthesizer is extractive: its claims are quotations, so they are grounded by construction and a clean pass proves nothing about hallucination. So the gate is measured against agentarch.agents.adversarial, which damages the synthesizer's reply inside the effect boundary — one property at a time, so the corrupted claim is journalled, replays, and is caught from the trace alone. Verbatim from examples/grounding_demo.py, a workflow demo step:

2. one property damaged per run — 5 lies, 2 controls
   drop_citations     lie     outcome=incomplete violations=1 ['no_citation']
   invent_chunk       lie     outcome=incomplete violations=1 ['unresolved_chunk']
   stretch_span       lie     outcome=incomplete violations=1 ['bad_span']
   invent_number      lie     outcome=complete   violations=1 ['unsupported_quote']
   invent_identifier  lie     outcome=complete   violations=1 ['unsupported_quote']
   reflow             control outcome=complete   violations=0
   quote_evidence     control outcome=complete   violations=0

Read the outcome column. The three citation-shaped lies end incomplete — the critic saw them. The two content lies end complete: the critic accepted an answer containing a number that appears nowhere in the evidence. That gap is the gate's entire job. The two controls — a claim reflowed to different whitespace, and a claim that quotes a token out of its own cited span — must produce nothing, because a gate that flags those is not strict, it is broken.

And the number the honest run reports, which is the one worth being uncomfortable about (same demo, step 1, two lines elided):

1. the honest run
   grounding (tier 1, scope=chunk) — run run-6de99385d564
     claims            : 6
     with a citation   : 6 (100%)
     tokens checked    : 2 (in 2 of 6 claims; 4 contained nothing checkable)
     violations        : 0

The other half of "does this gate work" is how often it cries wolf, and that is measured too. Every quotable sentence in the shipped corpus becomes a claim citing its own span — verbatim, and then rewritten the way a writer quoting it correctly would rewrite it (a value backticked, a function name written with its parentheses, the sentence's full stop moved inside the quotation marks). Every token in every one of those claims comes from the evidence it cites, so a flag would be a false alarm by construction:

6. the false-alarm sweep: every quotable sentence in the corpus, checked
   scope=chunk 13899 claims → 0 flagged   verbatim 0/6812, quote with the stop inside 0/5011, number in backticks 0/308, code with parens 0/1768
   scope=span  13899 claims → 0 flagged   verbatim 0/6812, quote with the stop inside 0/5011, number in backticks 0/308, code with parens 0/1768

The rewrites are the half of the sweep that can actually fail — a verbatim copy can only surface an asymmetric normalisation bug — and one rewrite family is honestly absent: regrouping a number's digit commas needs a standalone number of five or more digits, and the corpus's quotable prose contains none, so that rewrite never fires here and the comma normalisation is pinned by unit tests instead.

Most corpus sentences contain no number, quotation or identifier, so on four of those six claims the verbatim tier had nothing to look for and only the citation checks ran. The report carries checkless_claims for exactly that reason: "0 violations" over claims nothing could be checked in is a weaker statement than it looks, and the tool says so instead of letting a reader assume otherwise. Entailment — the check that would cover those four claims — is Tier 2, and the next section is the reason it does not exist.

Tier 2, and the judge that failed its own audit

Entailment needs a judge, and an unmeasured judge is a coin flip with a rubric. This repo measures its candidates with judgecheck, the judge-reliability project in this portfolio — its decision table, Krippendorff's α, Cohen's κ, Gwet's AC1 and its cluster bootstrap. Nothing statistical is reimplemented here. That is the one claim in this portfolio worth making twice: I built the system, and I built the instrument that told me a part of it was not fit to ship.

The items cannot be labelled by a judge — that would be circular — so they are built by transformation. Take a prose sentence out of a chunk and either preserve its meaning or damage it in exactly one named way; the label follows from the transformation rather than from anyone's reading. 4,765 items, split by document into 1,951 dev / 2,814 test, so the one tuned threshold never sees its test items.

python tools/eval_judge.py --out eval/results/judge.md      # ~45s on an idle machine, offline, no key

Read the coefficients and it looks survivable: the panel reaches κ = 0.82. Read the families and it does not.

Every candidate scores zero on the family whose single edit is is → is not. A verbatim-token gate cannot see it (every token is still there), and neither can a lexical-overlap judge (not is a stopword, and counted it would be one token in twenty). A Tier 2 built from these would import a κ near the conventional bar while being blind to polarity — it would pass "the client must not send the token" citing text that says it must. So there is no Tier 2, and that is the reason.

Three more things the page says out loud, because each is a way a number like this gets misread:

• tier1_checkable has the highest κ on the page and is the same code as tier1_gate — one reading answers everything, the other abstains where it has no mechanism. Abstention turned 0.4853 into 0.8394 by answering 71% of the time. A coefficient without its coverage is a judge grading its own syllabus.

• always_supported scores κ = 0.0000 and AC1 = 0.3220. Same data, same judge, two chance corrections: AC1 declines to charge a judge for lopsided marginals, and a constant judge is the limit of that leniency. Quoting the second alone is the mistake both tools exist to prevent.

• κ against a constructed truth is chance-corrected accuracy, not reliability. truth is the transformation, not an annotator, so no human ceiling is in the picture and none is implied.

Full method, the per-family construction rules and the limitations: eval/results/judge.md and DESIGN.md §10. The workflow regenerates and diffs the report, so once it runs the numbers above cannot drift away from the code that made them; until then the same --check is a command in the Quickstart.

Quickstart

Needs Python 3.11+. No API key, no network, no services. Timings below are wall-clock on an idle machine; a loaded host can be several times slower.

"The workflow", wherever this README or DESIGN.md says it, is .github/workflows/ci.yml. It is configured to run two jobs on every push to main and every pull request. check, on Python 3.11 and 3.12, installs the package, runs ruff, mypy and the suite with the MCP extra required, runs the five demos, and regenerates and diffs the retrieval and hybrid reports. judge, on 3.12, installs judgecheck from the repository URL configured as the JUDGECHECK_GIT_URL repository variable, runs the judge suite with that dependency required, and regenerates and diffs the judge report; where the variable is unset the job is skipped under a name that says so, never passed. Everything both jobs run is also the sequence below, run by hand from a clean clone.

git clone <this repository> agentarch && cd agentarch
python3 -m venv .venv && .venv/bin/pip install -e ".[dev,mcp]"   # drop `,mcp` to skip the MCP surface
.venv/bin/pytest
.venv/bin/ruff check . && .venv/bin/mypy

Reproduce the retrieval numbers (about a minute and a few minutes respectively, no network; add --check eval/results/<file>.md to fail if the committed report has drifted, which is what the workflow does):

.venv/bin/python tools/eval_retrieval.py     # chunking, analyzer, BM25 grid
.venv/bin/python tools/eval_hybrid.py        # dense rank sweep, 32-configuration fusion grid

Run the full system end-to-end and watch it replay (~1s, no network):

.venv/bin/python examples/agent_demo.py

Drive it over MCP — a real client, then the agent consuming a subprocess server (~5s, no network):

.venv/bin/python examples/mcp_demo.py

See a timed run through the viewer — both renders, and the cost report refusing to invent a dollar figure (~1s, no network):

.venv/bin/python examples/observe_demo.py

Watch the grounding gate catch five lies and let two controls through (~6s, no network):

.venv/bin/python examples/grounding_demo.py

Reproduce the judge-validation page. This is the one command that needs a second repository — judgecheck, the judge-reliability project, which is not on PyPI and requires Python 3.12, so it is installed from its own repository (substitute that repository's URL below, or the path of a local checkout with pip install -e). Without it, everything above still runs and only this step is unavailable (~45s on an idle machine, no network):

.venv/bin/pip install "git+<judgecheck repository URL>"
.venv/bin/python tools/eval_judge.py --check eval/results/judge.md

Then watch replay catch a behaviour change in the kernel itself:

.venv/bin/python examples/replay_demo.py

It records a run, writes the trace, replays it with no model available, then edits the agent twice — once to drop a call, once to change an input — and shows the replay refusing both:

4. now change the agent — drop its follow-up call — and replay again
   caught: replay diverged on lane 'main' at seq 1: 1 recorded effect(s) were never
           requested (unconsumed lanes: {'main': 1})

5. and change the question instead of the control flow
   caught: replay diverged on lane 'main' at seq 0: same call model:stub, different
           payload — 'question': 'what is an MCP resource?' -> 'something else'

The slices, and where they stand

Built in order; each one left the quickstart working. ✅ is shipped; 🟡 is shipped with a named part deliberately not built, for the reason given.

1. ✅ Determinism kernel — effect boundary, canonical digests, trace format, strict replay.

2. ✅ Retrieval + its own evaluation — corpus vendored with provenance, 56-query gold set, chunking sweep, BM25 with recall@k / nDCG@k / MRR and bootstrap intervals. 2b. 🟡 Dense arm + fusion — offline LSA, RRF and score fusion, swept and measured against the baseline. Done, with a negative result. Reranking: not yet.

3. ✅ Orchestration — planner / worker / synthesizer / critic with typed hand-offs, bounded recursion, four named terminal outcomes, and real failure handling (tool error, malformed hand-off, timeout, empty retrieval, no-progress loop, contradiction). A whole run replays from its trace.

4. ✅ MCP server — four tools and a resource per run over stdio, driven by a real client in examples/mcp_demo.py; and MCP consumed internally, with the trace proving the transport stays below the effect boundary.

5. ✅ Observability — per-lane/per-effect roll-up, a price table that ships empty by design, and a trace viewer (text + self-contained HTML) whose answer section is produced by replaying the trace rather than trusting its labels.

6. 🟡 Grounding enforcement — Tier 1 done: citations required, citations resolved, quoted numbers and identifiers checked verbatim against the retrieved evidence, with a CLI whose exit code fails the workflow and a corrupting runner that proves the gate catches what it claims to. Tier 2 (entailment) is not shipping, for the measured reason in item 7.

7. 🟡 Evaluation, wired to judgecheck — done for the judge layer: a 4,765-item labelled set whose truth is by construction, five candidate judges, and every coefficient computed by the judge-reliability toolkit. The result is negative and it killed Tier 2. Not done: end-to-end task success with cost, which needs a live model path to mean anything.

Limitations

Stated up front, and updated as the repo grows.

• The determinism kernel, the retrieval layer (lexical, dense and fused), the orchestrator, the MCP surface, the observability layer, the Tier-1 grounding gate and the judge-validation layer are implemented. Tier-2 entailment does not exist, and now has a measured reason not to; nor does end-to-end task-success evaluation.

• The judge-validation numbers describe constructed damage, not model output. Every item was made by transforming a corpus sentence, so the coefficients are a sensitivity profile per named family — evidence about is → is not, a swapped number, a foreign sentence, and about nothing else. They are not a benchmark and no judge is certified by them; what they support is the refusal to ship Tier 2.

• judgecheck (judge-reliability) is installed from its repository, not declared. It is not on PyPI and needs Python 3.12, so tools/eval_judge.py needs it installed by hand (Quickstart). Committed results stay readable without it, the code raises a message naming the install rather than an ImportError, and the workflow runs its suite in a separate job that is skipped under a name saying so when no repository URL is configured — never passed quietly.

• The grounding gate is weak by construction, and thin on this corpus. It compares claims against evidence mechanically: it cannot tell that a correctly-quoted number is being misinterpreted, it says nothing about relevance, and a claim containing no number, quotation or identifier receives only the citation checks — 4 of 6 claims in the demo run. Its measured strength is exactly the seven corruptions in examples/grounding_demo.py and no more.

• The MCP server streams nothing and reports no progress. It does not need to: an offline run answers in single-digit milliseconds once the index is built, so there is nothing to stream. That is a property of the stub agents rather than of the design, and it is the first thing a live model path would have to change.

• The two error channels are this repo's, not the whole surface's. InvalidRequest becomes a JSON-RPC INVALID_PARAMS and ExecutionFailed is re-raised as the SDK's ToolError, which becomes an isError result carrying the message — the one exception class every mcp 2.x in the declared range forwards the text of. From 2.1 the SDK treats any other exception as a crash and sends the client only Error executing tool <name>; the flag survives that, the reason does not, so the test and the demo assert the message rather than the flag. The SDK's own argument validation, on the other hand, reports an out-of-schema value as an isError result rather than as a protocol error. A test pins that too, including the half that contradicts what the design would prefer.

• One retrieval arm per server process. The arm is fixed at construction rather than chosen per call, because three indexes is three builds and a second of startup for a knob whose measured effect on this corpus is not distinguishable from zero.

• The offline agents are deterministic stand-ins, not a model. The synthesizer selects sentences from retrieved chunks, so answers read as a list of quotations and cannot paraphrase, aggregate or reason across chunks. Its output is therefore grounded by construction, which means the stub path proves nothing about whether the grounding gate (item 6) catches hallucination — that is what agentarch.agents.adversarial and the eventual live path are for. A "100% citation rate" measured here would be a meaningless number and is not reported as one.

• No token or cost figures on the offline path. No tokenizer is involved, so any count would be a number this repo cannot reproduce. max_tokens is enforced against whatever usage a runner reports and does nothing when nothing reports usage. Money is not reported at all — see DESIGN.md §8.

• Durations are opt-in and partial. TimingRunner is not installed on the default path (a wall-clock stamp breaks byte-identical traces across transports, which is a tested property), and what it measures is time inside the effect boundary only — records carry no timestamps, so time between effects is neither recorded nor estimated, and the viewer presents no wall-clock total.

• The critic's contradiction detector is a lexical heuristic (shared identifier, disjoint numeric literals). It will miss disagreements phrased in prose and flag pairs that merely quote the same rule with different examples. It reports; it never arbitrates and never sets the verdict.

• A run that fails at the effect boundary is not replayable, by design — see above. "Any recorded run is a regression test" is true of complete, incomplete, stalled and malformed-hand-off failed runs, and false of boundary failures.

• Retrieval quality bounds everything above it. Asked about a made-up "zzzqqx protocol", BM25 matches the one real word in the query and the extractive synthesizer dutifully quotes sentences about protocols. The run is marked partial and the unanswerable half is reported as a gap, but the answered half is still weak — a reranker or a relevance floor is the fix, and neither exists yet.

• The dense arm is LSA, not a learned embedding. Its notion of "related" is corpus co-occurrence only; a query term absent from the corpus vocabulary contributes nothing; word order is invisible. A sentence-transformer arm would very likely score better, and would make the quickstart download weights. The hybrid result above is a statement about this dense arm and is not evidence about hybrid search in general.

• No reranking. A cross-encoder is the standard next stage and would plausibly move the numbers more than the fusion rule does. Nothing here is claimed about it.

• The hybrid page selects three times on one 56-query set — rank, then vocabulary floor, then fusion configuration, each conditional on the last — and corrects for multiplicity nowhere. Across ~40 uncorrected intervals, roughly two would exclude zero by chance. One more reason to read the hybrid headline as "not shown to help", and to hold the nDCG result loosely.

• The retrieval numbers describe this corpus and this 56-query set. They are not a claim about retrieval in general, and the corpus deliberately excludes the near-duplicate historical spec revisions that make real documentation retrieval hard — see the corpus card.

• Recall is a lower bound: no judgement pooling was done, so a section that answers a query but was never labelled counts as a miss.

• Replay reproduces effects, not the outside world. It proves the agent's decision sequence is unchanged given the same responses; it does not prove the model would answer the same way again.

• Order-strict replay per lane is the right check for this design, but it means a genuinely nondeterministic scheduler within a lane would show up as divergence. Lanes exist so that parallel workers do not have to reproduce a global interleaving.

Licence

MIT — see LICENSE.