agentarch MCP Server
by adhammomen
README.md
# agentarch
A retrieval-grounded multi-agent system that you can **replay exactly** from a trace, exposed over MCP.
The goal is not another RAG demo. It is the engineering that separates a demo from a system:
retrieval that is measured on its own before any generation is judged, an orchestrator whose state
is explicit and whose runs are byte-for-byte reproducible, every generated claim tied to a retrieved
chunk, and a check that fails when a claim has no support.
> **Status: the offline system is complete.** Eight slices, built thin and vertical, each leaving
> the quickstart working: the **determinism kernel** (effect boundary, trace format, strict
> replay), the **retrieval layer with its own standalone evaluation**, the **dense arm plus hybrid
> fusion, measured against that baseline**, the **multi-agent orchestrator** — which runs
> end-to-end offline and replays exactly from its own trace — the **MCP surface**, in both
> directions (a stdio server any MCP client can drive, and an agent whose retrieval is answered by
> calling a tool on another MCP server), the **observability layer** and its replaying trace
> viewer, the **Tier-1 grounding gate** with the corrupting runner it is measured against, and the
> **judge-validation layer**, wired to `judgecheck` (the `judge-reliability` project), whose
> negative result killed Tier 2.
>
> Four things are deliberately absent, each with its reason stated where it is felt: a **live
> model path** (the default path is offline by design; a provider is opt-in and none is wired),
> **Tier-2 entailment** (every offline candidate failed [its own audit](#tier-2-and-the-judge-that-failed-its-own-audit)),
> a **reranker**, and **end-to-end task-success evaluation** (against an extractive stub it would
> measure the stub). [The slices](#the-slices-and-where-they-stand) say where each part stands,
> [DESIGN.md](DESIGN.md) says how it is put together, and nothing in this README describes
> behaviour that is not in the repo.
## The shape of the system
```mermaid
flowchart LR
MCPC[any MCP client] --> SRV[MCP server]
SRV --> ORCH
subgraph ORCH [orchestrator]
direction LR
P[planner] --> W[retrieval workers] --> S[synthesizer] --> C[critic]
end
W --> RET["retrieval: BM25 + LSA + fusion"]
ORCH -. every model, retrieval and clock call .-> EB[effect boundary]
RET -.-> EB
EB --> TR[("trace, JSONL")]
TR --> REP[strict replay]
TR --> VIEW[trace viewer]
TR --> GATE[grounding gate]
```
One invariant carries the design: **nothing above the effect boundary touches the outside world
except through it, and everything that crosses it is journalled.** That is why a run replays
exactly with nothing running, why the viewer can render a run it cannot re-execute, and why the
grounding gate can refuse a months-old run from its trace alone. The gate and the viewer hang off
the trace, not off the orchestrator — a check the model can argue with is not a gate. The fuller
diagram, with the failure modes and the state each hand-off carries, is [DESIGN.md](DESIGN.md) §2.
**If you read one section.** `agentarch` is a retrieval-grounded multi-agent system in which every
model, retrieval and clock call crosses one journalled effect boundary, so a recorded run replays
byte-for-byte with nothing running and refuses to replay once the code's behaviour has changed
([`replay_demo.py`](examples/replay_demo.py)). Retrieval was measured before anything generated:
BM25 over overlapping 100-word chunks reaches R@2000w 0.629 [0.522, 0.731] on 56 queries, the
baseline everything later had to beat ([retrieval-baseline.md](eval/results/retrieval-baseline.md)).
Adding a dense arm and fusion did not beat it by more than the noise: the best fusion
configuration's paired gain is 0.031 [0.000, 0.076], an interval that includes zero
([hybrid.md](eval/results/hybrid.md)). The grounding gate catches five injected lies, passes two
controls, and flags 0 of 13,899 claims in a false-alarm sweep over the corpus
([`grounding_demo.py`](examples/grounding_demo.py)). No offline entailment judge was fit to ship
as Tier 2: the best judge that answers every item reaches κ = 0.8176 [0.7965, 0.8393], and every
candidate scores 0/198 on claims whose one edit is a negation ([judge.md](eval/results/judge.md)).
Each layer ships with the check that proves it, and every one runs offline from a clean clone —
with one disclosed exception: the judge row needs `judgecheck`, the `judge-reliability` project,
installed from its own repository, and Python 3.12 (see the [Quickstart](#quickstart)):
| layer | code | the proof |
|---|---|---|
| determinism kernel | [`effects.py`](src/agentarch/effects.py), [`trace.py`](src/agentarch/trace.py) | [`replay_demo.py`](examples/replay_demo.py) — a recorded run refuses two behaviour edits |
| retrieval | [`retrieval/`](src/agentarch/retrieval) | [`eval_retrieval.py`](tools/eval_retrieval.py), [`eval_hybrid.py`](tools/eval_hybrid.py) — both reports regenerated and diffed by the workflow |
| orchestration | [`agents/`](src/agentarch/agents), [`orchestrator.py`](src/agentarch/orchestrator.py) | [`agent_demo.py`](examples/agent_demo.py) — a full run replays from its trace; stalled and budget-capped outcomes asserted |
| MCP, both directions | [`mcp/`](src/agentarch/mcp) | [`mcp_demo.py`](examples/mcp_demo.py) — the trace recorded over stdio JSON-RPC is byte-identical to the in-process one |
| observability | [`observe/`](src/agentarch/observe) | [`observe_demo.py`](examples/observe_demo.py) — the viewer replays the journal rather than trusting its labels |
| grounding gate | [`grounding/`](src/agentarch/grounding) | [`grounding_demo.py`](examples/grounding_demo.py) — five injected lies caught, two controls pass, 13,899-claim false-alarm sweep |
| judge validation | [`evaluation/`](src/agentarch/evaluation) | [`eval_judge.py`](tools/eval_judge.py) — every coefficient computed by `judgecheck` (the `judge-reliability` project), report diffed by the workflow's judge job |
## Why the determinism kernel came first
Every other feature is easier to build on top of a run you can reproduce, and impossible to debug
without one. So the first slice is the constraint that makes reproduction possible:
**All non-determinism — model calls, tool calls, the clock, the RNG — goes through one interface.**
```python
from agentarch import EffectRequest, RecordingRunner, Trace, RunHeader, verify_replay
rec = RecordingRunner(live_runner)
answer = my_agent(rec) # the agent only talks to the runner
trace = Trace(header=RunHeader(run_id="r1", config_digest=cfg), records=rec.records)
# later, on another machine, with no network and no API key:
assert verify_replay(trace, my_agent) == answer
```
`verify_replay` is strict on purpose. It fails if the code requests a *different* effect, requests
effects in a *different order* within a lane, requests an *extra* one, or *stops early*. A replay
that tolerates a changed request proves nothing; a replay that fails loudly turns any recorded run
into a regression test for agent behaviour — which is what an agent system actually lacks.
## Retrieval, measured before anything generates
A generator cannot cite what was never retrieved, so retrieval is evaluated on its own first. The
corpus is the MCP specification and the Python SDK documentation, vendored with per-file provenance
([corpus card](corpus/CORPUS_CARD.md)); the gold set is 56 questions labelled with the corpus
sections that answer them ([datasheet](eval/gold/DATASHEET.md)).
```bash
.venv/bin/python tools/eval_retrieval.py --out eval/results/retrieval-baseline.md
```
Everything below regenerates from a clean clone with no network and no key. The full report,
including the whole sweep, is in [eval/results/retrieval-baseline.md](eval/results/retrieval-baseline.md).
**The finding worth reading.** Sweeping chunk size and ranking by `recall@10` says bigger chunks are
better, monotonically. Holding the *context budget* fixed instead says the opposite:
| chunk size | R@10 | words put in the window @10 | R@2000 words |
|---|---|---:|---|
| 100 words, 20 overlap | 0.537 [0.426, 0.646] | 988 | **0.629 [0.522, 0.731]** |
| 250 words | 0.618 [0.500, 0.726] | 2 417 | 0.573 [0.455, 0.685] |
| 500 words | 0.674 [0.568, 0.781] | 4 743 | 0.493 [0.376, 0.607] |
| 1000 words | 0.683 [0.576, 0.792] | 8 948 | 0.417 [0.304, 0.536] |
At fixed `k`, a 1000-word configuration is handed roughly nine times more text than a 100-word one,
so `R@k` mostly rediscovers that more text retrieves more answers. Both orderings are separable from
zero under a paired bootstrap, in opposite directions. Chunking papers and blog posts that report
`recall@k` across chunk sizes without a matched budget are measuring this artefact.
Other honest results from the same run:
- **BM25's `k1`/`b` grid is flat** on this corpus — the 20-cell grid spans 0.507 to 0.537 on R@10,
a range far smaller than any single cell's confidence interval. The parameters most tutorials fuss over do not matter
here; the analyzer choices do, slightly.
- **Heading-aware chunking loses to fixed windows**, which was not the expected result.
- Retrieval is much worse on questions needing more than one section (R@10 0.42) or spanning both
sources (0.45) than on single-section ones (0.65).
Numbers only mean something with their instrument characterised, so: 56 queries is small; the
labels were generated with a language model and checked for internal consistency,
and nobody independent checked whether a label is the right answer. A re-labelling of 12 queries
agreed on the primary answer 12/12 but had a mean Jaccard of only 0.44 across the full label sets.
**This is a self-consistency check, not an independent validation.** The original labels and the
re-labelling were generated with language models of the same family, from the same corpus and a
similar brief, so correlated blind spots will not show up in it. The
[datasheet](eval/gold/DATASHEET.md) says what that does and does not license you to conclude.
## Hybrid search, and the result I did not want
The dense arm is **latent semantic indexing computed from the shipped corpus** — a truncated SVD of
the tf-idf term-chunk matrix, `numpy` and nothing else. Not a neural embedding, and this repo never
calls it one. It exists because the design constraint is that a reviewer can clone and run with no
key and no download, and precomputed embeddings do not cover a query typed at runtime. The full
argument, including what LSA cannot do, is in [`lsa.py`](src/agentarch/retrieval/lsa.py).
Two fusion rules are implemented — reciprocal rank fusion and normalised score fusion — and swept
over pool depth and arm weighting, 32 configurations. Then measured against the BM25 baseline that
was published *before* any of this existed:
| retriever | R@10 | nDCG@10 | R@2000 words |
|---|---|---|---|
| BM25 (baseline) | 0.537 [0.426, 0.646] | 0.385 [0.299, 0.470] | 0.629 [0.522, 0.731] |
| LSA, rank 192 | 0.490 [0.378, 0.601] | 0.347 [0.264, 0.429] | 0.631 [0.524, 0.738] |
| best fusion (`score`, depth 100, 1:1) | 0.537 [0.423, 0.643] | 0.379 [0.291, 0.464] | **0.661 [0.560, 0.756]** |
**The best fusion beats BM25 by 0.031 [0.000, 0.076] paired — an interval that includes zero. On
this corpus and this gold set, hybrid search was not shown to help.** That is reported as the
headline of [eval/results/hybrid.md](eval/results/hybrid.md) rather than buried, because a repo
whose whole argument is measurement discipline does not get to only publish the wins.
Three things the measurement did surface:
- **Recall and ranking quality come apart.** The dense arm matches BM25 on recall at a matched
context budget (Δ 0.001 [-0.076, 0.077]) while scoring worse at ordering
(ΔnDCG@10 -0.038 [-0.074, -0.001] — an interval that excludes zero, but only just, and it is one
of ~40 uncorrected comparisons on this page, so read it as suggestive). It gets as much of the
answer into the window and then puts it lower down. That asymmetry is the argument for fusing
rather than replacing.
- **The ceiling is not the fusion rule.** An oracle that picks the better arm per query scores
0.693 — 0.03 above the fusion. (That bounds *arm selection*, not fusion: interleaving can beat
both arms on a query. It does not do so on any of these 56, so here it is an empirical ceiling
rather than a theorem, and the report says it that way.) Meanwhile **both arms score zero on 9
of 56 queries** — 0.16 of recall locked behind questions neither arm can see, five times the
0.03 sitting in the gap. The next move is a better arm or a reranker, not more fusion tuning.
- **The LSA rank curve is flat.** Only two of the other six ranks are separable from the one
selected. Picking rank 192 was choosing the top row of a noisy curve, and the report says so.
## The orchestrator: four named ways for a run to end
Planner → retrieval workers → synthesizer → critic, with typed artifacts on every hand-off and a
bounded revision loop. Run it:
```bash
.venv/bin/python examples/agent_demo.py
```
```
1. asked: How do I register a tool and handle its errors?
outcome : complete — critic accepted the draft
rounds : 1, steps: 5
effects : {'main': 3, 'worker/0/0': 1, 'worker/0/1': 1}
partial : False
## How do I handle its errors
- **Protocol Errors** indicate issues with the request structure itself that models are
less likely to be able to fix: [3]
...
[3] spec/server/tools — `spec/server/tools::32`
(that bullet is wrapped here; the demo prints it on one line)
```
The planner split that question in two, so two workers ran on **two separate lanes** — replay is
order-strict within a lane and makes no claim across them. Stated precisely: the workers run
sequentially today and each issues one retrieval, so each lane holds exactly one record. What lanes
buy right now is that reordering the workers in a round is not a divergence; the property is
designed for a parallelism that does not exist yet, and [DESIGN.md](DESIGN.md) §3 says so.
A run ends `complete`, `incomplete`, `stalled` or `failed`, never ambiguously, and `partial` is a
field on the result so a caller that ignores the outcome still cannot mistake a capped answer for a
finished one. Three of the four are reachable on the shipped corpus — `failed` needs an effect to
raise, which the offline handlers cannot do, so it is exercised by tests with injected failures:
```
4. asked: What is the zzzqqx protocol and how do I frobnicate a widget?
outcome : stalled — planner returned no sub-questions in round 2
rounds : 2 (it re-planned, then ran out of new questions)
gaps : ['retrieval returned no chunks for this sub-question', ...]
5. the same answerable question under a 3-step budget
outcome : incomplete — step budget exhausted after 3 steps
partial : True
```
**The run replays from its trace file with nothing else present** — no corpus, no index, no agents,
no clock:
```
2. wrote run-6de99385d564.jsonl (32226 bytes) and replayed it
no index was built, no agent ran, no clock was read
identical RunResult: True
3. replay the same trace under a different config (k=5 -> k=8)
caught: replay diverged on lane 'main' at seq 0: config mismatch …
```
Three design points behind that. The trace carries each retrieved chunk's **text**, not just its id
— larger files, in exchange for a trace that is self-sufficient. Every `model` request carries the
digest of that role's task specification, so **editing what an agent is asked to do breaks every
trace recorded under the old wording** rather than replaying green; there is a test for exactly
that. And the one case that does *not* replay is named rather than glossed: a run that died at the
effect boundary leaves a truncated journal, because failed effects are deliberately not recorded, so
`replay()` refuses it with `UnreplayableRun` instead of reporting a divergence that would blame the
code for a provider outage. A run that failed on a *malformed hand-off* does replay — the bad reply
is in the journal — and both halves have tests.
**The agents on the offline path are not a language model.** The planner splits compound questions
syntactically, the synthesizer is *extractive* — it quotes sentences that already exist in retrieved
chunks — and the critic runs deterministic checks. They exist so the architecture is exercisable
with no API key, and so a live path has an honest floor to be measured against. The answers they
produce are correspondingly shallow; see [Limitations](#limitations).
## MCP, in both directions
The system is an MCP server, and it is also an MCP *client* — the same protocol running both ways,
built on the official Python SDK.
**Outward.** `python -m agentarch.mcp` speaks stdio, which is how MCP clients launch servers. The
config names the venv's interpreter, because the client launches the server from its own working
directory and a bare `python` there may not be the one `agentarch` is installed into:
```json
{ "mcpServers": { "agentarch": { "command": ".venv/bin/python", "args": ["-m", "agentarch.mcp"] } } }
```
| tool | what it is for |
|---|---|
| `search_corpus(query, k)` | retrieval with no generation, so the layer this repo *measures* is drivable on its own |
| `answer_question(question, …)` | the full loop; each claim's citation carries the **quoted span**, so grounding is checkable from the response alone |
| `replay_run(run_id)` | replay as a tool — a client can ask the system to prove a past run reproduces, and gets the divergence, plus how far it got, when it does not |
| `list_runs()` | the runs this server has traces for; a reconnecting client holds no run ids and `replay_run` needs one |
Every run is also a resource at `trace://<run_id>`, and a client that opened a
`subscriptions/listen` stream gets a `ResourcesListChanged` event when one appears — that is the
2026-07-28 replacement for the old `notifications/resources/list_changed`, and there is a test that
subscribes and waits for it rather than assuming the call did something.
`answer_question` returns `partial` and `outcome` beside the prose, because a transport that shipped
only the answer text would let a client present a budget-capped run as a finished one — the exact
failure the orchestrator's budgets exist to prevent.
**Inward.** The orchestrator's `retrieval:search` effect can be answered by calling `search_corpus`
on another MCP server instead of an in-process index. The property that makes this more than
symmetry:
```
8. now the other direction: the agent's retrieval served over MCP
in-process retrieval : 9.4 ms complete
over stdio JSON-RPC : 1808.8 ms complete (includes launching the server and building its index)
mean per search call, 50 calls each after 5 warm-up:
in-process : 0.81 ms
over the open session: 5.41 ms
byte-identical trace : True
The transport is below the effect boundary, so it is not in the journal.
That is why a run retrieved over MCP replays with no server present:
replayed the MCP run with no session, no server, no index: True
```
Real output of `examples/mcp_demo.py`, one of the workflow's demo steps — one run of it; the millisecond figures are
wall-clock and vary, the last line does not. The trace of a run whose retrieval crossed a
subprocess boundary and JSON-RPC framing is **byte-identical** to the same run done in-process, and
replays with nothing running. That is the effect boundary doing its job: a `retrieval:search` effect
is a query and a set of hits, and where the hits came from is below it. The overhead is printed
rather than hidden. The whole run over stdio takes well over a second because it includes launching
the server and building its index; once the session is open, the `mean per search call` lines put
one search at a few milliseconds over JSON-RPC against about one in-process. That is a bad trade for
this corpus and the right one when the corpus belongs to somebody else.
The SDK is an optional extra (`pip install -e ".[mcp]"`) so the core stays a three-dependency
install. The workflow installs it and sets `AGENTARCH_REQUIRE_MCP=1`, which turns the MCP suite's
`importorskip` into a failure — otherwise "optional" quietly becomes "untested".
## Seeing a run: the trace viewer
If you cannot see what the agent did, you cannot debug it. The trace has been the substrate since
the first slice; `agentarch.observe` is the render that lets a reader see a run without reading
JSONL:
```bash
.venv/bin/python -m agentarch.observe path/to/run-6de99385d564.jsonl # text
.venv/bin/python -m agentarch.observe path/to/run-6de99385d564.jsonl --html run.html
```
The HTML render is one self-contained file — no JavaScript, no external assets — with the
per-lane/per-effect roll-up, every effect's request and response, and each claim's citation shown
as its exact character span highlighted inside the full text of the chunk the run retrieved. The
chunk text comes from the trace itself, so the viewer needs no corpus and no index.
Two decisions worth knowing about before trusting it:
- **The answer section is produced by replaying the trace, not by reading its labels.** If the
code has changed since the run was recorded, the viewer says the run no longer reproduces
instead of rendering an answer the current code would not produce. The timeline and roll-up
render regardless — a truncated journal from a run that died at the effect boundary still gets
its timeline, because a broken run is the one you most need to see.
- **Token counts always, currency almost never.** The cost report prices usage only from a table
(`src/agentarch/observe/prices.json`) whose every entry must carry a source URL and a retrieval
date — and that table ships empty, so out of the box you get token counts and an itemised
explanation instead of a dollar figure. The offline path reports no usage at all, and the report
says so rather than printing $0.00: "no model was involved" is not "this run was free".
Per-effect durations exist when a run is recorded with `effects.TimingRunner` — deliberately not
the default, because a wall-clock stamp would break the byte-identical-traces property the MCP
demo checks. `examples/observe_demo.py` shows a timed run, both renders, and the cost report's
refusals, and every step asserts.
## Grounding: the check that fails the run
Every claim carries a citation to a span inside a chunk the run actually retrieved, and a
deterministic gate refuses the answer when one does not:
```bash
.venv/bin/python -m agentarch.grounding path/to/run-6de99385d564.jsonl
# 0 = grounded · 1 = a claim failed the gate · 2 = could not check at all
```
Three checks, all mechanical, no model involved: a claim must have a citation; every citation must
name a chunk this run retrieved and point at a non-empty span inside it; and every quoted span,
number and code-like identifier in the claim must appear **verbatim** in a cited chunk. It runs over
a stored trace with nothing else present — no corpus, no index, no agents — because it replays the
journal to get the claims. That is what makes it a check someone who does not trust the run can
re-run months later. What it proves is consistency, not provenance: the trace is unsigned and
every digest in it is a plain sha256, so a producer who rewrites the whole file — chunk texts,
claims and digests together — gets a trace this gate passes. Against a malicious producer the trace
itself has to be trusted, or the chunk texts re-verified against `corpus/manifest.json` where the
corpus is present, which this gate does not yet do.
**It is deliberately not a step in the orchestrator's loop.** The critic already reviews the draft,
and on the live path the critic *is* the model; a check the model can influence is not a gate.
### What it actually catches — measured, not asserted
The offline synthesizer is extractive: its claims are quotations, so they are grounded by
construction and a clean pass proves nothing about hallucination. So the gate is measured against
`agentarch.agents.adversarial`, which damages the synthesizer's reply **inside the effect boundary**
— one property at a time, so the corrupted claim is journalled, replays, and is caught from the
trace alone. Verbatim from `examples/grounding_demo.py`, a workflow demo step:
```
2. one property damaged per run — 5 lies, 2 controls
drop_citations lie outcome=incomplete violations=1 ['no_citation']
invent_chunk lie outcome=incomplete violations=1 ['unresolved_chunk']
stretch_span lie outcome=incomplete violations=1 ['bad_span']
invent_number lie outcome=complete violations=1 ['unsupported_quote']
invent_identifier lie outcome=complete violations=1 ['unsupported_quote']
reflow control outcome=complete violations=0
quote_evidence control outcome=complete violations=0
```
Read the outcome column. The three citation-shaped lies end `incomplete` — the critic saw them. The
two content lies end `complete`: **the critic accepted an answer containing a number that appears
nowhere in the evidence.** That gap is the gate's entire job. The two controls — a claim reflowed to
different whitespace, and a claim that quotes a token out of its own cited span — must produce
nothing, because a gate that flags those is not strict, it is broken.
And the number the honest run reports, which is the one worth being uncomfortable about (same
demo, step 1, two lines elided):
```
1. the honest run
grounding (tier 1, scope=chunk) — run run-6de99385d564
claims : 6
with a citation : 6 (100%)
tokens checked : 2 (in 2 of 6 claims; 4 contained nothing checkable)
violations : 0
```
The other half of "does this gate work" is how often it cries wolf, and that is measured too. Every
quotable sentence in the shipped corpus becomes a claim citing its own span — verbatim, and then
rewritten the way a writer quoting it correctly would rewrite it (a value backticked, a function
name written with its parentheses, the sentence's full stop moved inside the quotation marks).
Every token in every one of those claims comes from the evidence it cites, so a flag would be a
false alarm by construction:
```
6. the false-alarm sweep: every quotable sentence in the corpus, checked
scope=chunk 13899 claims → 0 flagged verbatim 0/6812, quote with the stop inside 0/5011, number in backticks 0/308, code with parens 0/1768
scope=span 13899 claims → 0 flagged verbatim 0/6812, quote with the stop inside 0/5011, number in backticks 0/308, code with parens 0/1768
```
The rewrites are the half of the sweep that can actually fail — a verbatim copy can only surface an
asymmetric normalisation bug — and one rewrite family is honestly absent: regrouping a number's
digit commas needs a standalone number of five or more digits, and the corpus's quotable prose
contains none, so that rewrite never fires here and the comma normalisation is pinned by unit tests
instead.
Most corpus sentences contain no number, quotation or identifier, so on four of those six claims the
verbatim tier had nothing to look for and only the citation checks ran. The report carries
`checkless_claims` for exactly that reason: "0 violations" over claims nothing could be checked in
is a weaker statement than it looks, and the tool says so instead of letting a reader assume
otherwise. Entailment — the check that would cover those four claims — is Tier 2, and the next
section is the reason it does not exist.
## Tier 2, and the judge that failed its own audit
Entailment needs a judge, and an unmeasured judge is a coin flip with a rubric. This repo measures
its candidates with `judgecheck`, the `judge-reliability` project in this portfolio — its decision table, Krippendorff's α, Cohen's κ,
Gwet's AC1 and its cluster bootstrap. **Nothing statistical is reimplemented here.** That is the one
claim in this portfolio worth making twice: I built the system, and I built the instrument that told
me a part of it was not fit to ship.
The items cannot be labelled by a judge — that would be circular — so they are **built by
transformation**. Take a prose sentence out of a chunk and either preserve its meaning or damage it
in exactly one named way; the label follows from the transformation rather than from anyone's
reading. 4,765 items, split by *document* into 1,951 dev / 2,814 test, so the one tuned threshold
never sees its test items.
```
python tools/eval_judge.py --out eval/results/judge.md # ~45s on an idle machine, offline, no key
```
| judge | answered | accuracy | balanced | κ vs truth | 95% CI | P(κ < 0.8) | AC1 |
| --- | ---: | ---: | ---: | ---: | --- | ---: | ---: |
| `tier1_gate` | 2814/2814 | 76.0% | 72.9% | 0.4853 | [0.4572, 0.5142] | 1.000 | 0.5727 |
| `tier1_checkable` | 2000/2814 | 93.0% | 90.1% | 0.8394 | [0.8146, 0.8639] | 0.001 | 0.8626 |
| `overlap@0.40` | 2814/2814 | 82.4% | 80.4% | 0.6307 | [0.6036, 0.6578] | 1.000 | 0.6736 |
| `always_supported` | 2814/2814 | 55.6% | 50.0% | 0.0000 | [0.0000, 0.0000] | 1.000 | 0.3220 |
| `any_unsupported` | 2814/2814 | 91.2% | 90.2% | 0.8176 | [0.7965, 0.8393] | 0.045 | 0.8291 |
Read the coefficients and it looks survivable: the panel reaches κ = 0.82. Read the **families** and
it does not.
| judge | `verbatim` | `rewrite` | `negated` | `number_swapped` | `identifier_swapped` | `swapped_evidence` |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `tier1_gate` | 808/808 | 757/757 | **0/198** | 110/110 | 133/133 | 330/808 |
| `overlap@0.40` | 808/808 | 739/757 | **0/198** | 0/110 | 0/133 | 773/808 |
| `any_unsupported` | 808/808 | 739/757 | **0/198** | 110/110 | 133/133 | 775/808 |
Every candidate scores **zero** on the family whose single edit is `is` → `is not`. A verbatim-token
gate cannot see it (every token is still there), and neither can a lexical-overlap judge (`not` is a
stopword, and counted it would be one token in twenty). A Tier 2 built from these would import a κ
near the conventional bar while being blind to polarity — it would pass *"the client must **not**
send the token"* citing text that says it must. **So there is no Tier 2, and that is the reason.**
Three more things the page says out loud, because each is a way a number like this gets misread:
- **`tier1_checkable` has the highest κ on the page and is the same code as `tier1_gate`** — one
reading answers everything, the other abstains where it has no mechanism. Abstention turned
0.4853 into 0.8394 by answering 71% of the time. A coefficient without its coverage is a judge
grading its own syllabus.
- **`always_supported` scores κ = 0.0000 and AC1 = 0.3220.** Same data, same judge, two chance
corrections: AC1 declines to charge a judge for lopsided marginals, and a constant judge is the
limit of that leniency. Quoting the second alone is the mistake both tools exist to prevent.
- **κ against a constructed truth is chance-corrected accuracy, not reliability.** `truth` is the
transformation, not an annotator, so no human ceiling is in the picture and none is implied.
Full method, the per-family construction rules and the limitations:
[`eval/results/judge.md`](eval/results/judge.md) and [DESIGN.md](DESIGN.md) §10. The workflow
regenerates and diffs the report, so once it runs the numbers above cannot drift away from the
code that made them; until then the same `--check` is a command in the Quickstart.
## Quickstart
Needs Python 3.11+. No API key, no network, no services. Timings below are wall-clock on an idle
machine; a loaded host can be several times slower.
**"The workflow"**, wherever this README or DESIGN.md says it, is
[`.github/workflows/ci.yml`](.github/workflows/ci.yml). It is configured to run two jobs on every
push to `main` and every pull request. `check`, on Python 3.11 and 3.12, installs the package, runs `ruff`,
`mypy` and the suite with the MCP extra *required*, runs the five demos, and regenerates and diffs
the retrieval and hybrid reports. `judge`, on 3.12, installs `judgecheck` from the repository URL
configured as the `JUDGECHECK_GIT_URL` repository variable, runs the judge suite with that
dependency *required*, and regenerates and diffs the judge report; where the variable is unset the
job is skipped under a name that says so, never passed. Everything both jobs run is also the
sequence below, run by hand from a clean clone.
```bash
git clone <this repository> agentarch && cd agentarch
python3 -m venv .venv && .venv/bin/pip install -e ".[dev,mcp]" # drop `,mcp` to skip the MCP surface
.venv/bin/pytest
.venv/bin/ruff check . && .venv/bin/mypy
```
Reproduce the retrieval numbers (about a minute and a few minutes respectively, no network; add
`--check eval/results/<file>.md` to fail if the committed report has drifted, which is what the
workflow does):
```bash
.venv/bin/python tools/eval_retrieval.py # chunking, analyzer, BM25 grid
.venv/bin/python tools/eval_hybrid.py # dense rank sweep, 32-configuration fusion grid
```
Run the full system end-to-end and watch it replay (~1s, no network):
```bash
.venv/bin/python examples/agent_demo.py
```
Drive it over MCP — a real client, then the agent consuming a subprocess server (~5s, no network):
```bash
.venv/bin/python examples/mcp_demo.py
```
See a timed run through the viewer — both renders, and the cost report refusing to invent a
dollar figure (~1s, no network):
```bash
.venv/bin/python examples/observe_demo.py
```
Watch the grounding gate catch five lies and let two controls through (~6s, no network):
```bash
.venv/bin/python examples/grounding_demo.py
```
Reproduce the judge-validation page. This is the one command that needs a second repository —
`judgecheck`, the `judge-reliability` project, which is not on PyPI and requires Python 3.12, so it
is installed from its own repository (substitute that repository's URL below, or the path of a
local checkout with `pip install -e`). Without it, everything above still runs and only this step
is unavailable (~45s on an idle machine, no network):
```bash
.venv/bin/pip install "git+<judgecheck repository URL>"
.venv/bin/python tools/eval_judge.py --check eval/results/judge.md
```
Then watch replay catch a behaviour change in the kernel itself:
```bash
.venv/bin/python examples/replay_demo.py
```
It records a run, writes the trace, replays it with no model available, then edits the agent twice —
once to drop a call, once to change an input — and shows the replay refusing both:
```
4. now change the agent — drop its follow-up call — and replay again
caught: replay diverged on lane 'main' at seq 1: 1 recorded effect(s) were never
requested (unconsumed lanes: {'main': 1})
5. and change the question instead of the control flow
caught: replay diverged on lane 'main' at seq 0: same call model:stub, different
payload — 'question': 'what is an MCP resource?' -> 'something else'
```
## The slices, and where they stand
Built in order; each one left the quickstart working. ✅ is shipped; 🟡 is shipped with a named
part deliberately not built, for the reason given.
1. ✅ **Determinism kernel** — effect boundary, canonical digests, trace format, strict replay.
2. ✅ **Retrieval + its own evaluation** — corpus vendored with provenance, 56-query gold set,
chunking sweep, BM25 with recall@k / nDCG@k / MRR and bootstrap intervals.
2b. 🟡 **Dense arm + fusion** — offline LSA, RRF and score fusion, swept and measured against the
baseline. **Done, with a negative result.** Reranking: not yet.
3. ✅ **Orchestration** — planner / worker / synthesizer / critic with typed hand-offs, bounded
recursion, four named terminal outcomes, and real failure handling (tool error, malformed
hand-off, timeout, empty retrieval, no-progress loop, contradiction). A whole run replays from
its trace.
4. ✅ **MCP server** — four tools and a resource per run over stdio, driven by a real client in
`examples/mcp_demo.py`; and MCP consumed *internally*, with the trace proving the transport
stays below the effect boundary.
5. ✅ **Observability** — per-lane/per-effect roll-up, a price table that ships empty by design,
and a trace viewer (text + self-contained HTML) whose answer section is produced by replaying
the trace rather than trusting its labels.
6. 🟡 **Grounding enforcement** — Tier 1 done: citations required, citations resolved, quoted
numbers and identifiers checked verbatim against the retrieved evidence, with a CLI whose exit
code fails the workflow and a corrupting runner that proves the gate catches what it claims to. Tier 2
(entailment) is **not shipping**, for the measured reason in item 7.
7. 🟡 **Evaluation, wired to `judgecheck`** — done for the judge layer: a 4,765-item labelled
set whose truth is by construction, five candidate judges, and every coefficient computed by the
`judge-reliability` toolkit. The result is negative and it killed Tier 2. **Not done:** end-to-end task
success with cost, which needs a live model path to mean anything.
## Limitations
Stated up front, and updated as the repo grows.
- The determinism kernel, the retrieval layer (lexical, dense and fused), the orchestrator, the
MCP surface, the observability layer, the Tier-1 grounding gate and the judge-validation layer
are implemented. Tier-2 entailment does not exist, and now has a measured reason not to; nor
does end-to-end task-success evaluation.
- **The judge-validation numbers describe constructed damage, not model output.** Every item was
made by transforming a corpus sentence, so the coefficients are a sensitivity profile per named
family — evidence about `is` → `is not`, a swapped number, a foreign sentence, and about nothing
else. They are not a benchmark and no judge is certified by them; what they support is the
refusal to ship Tier 2.
- **`judgecheck` (`judge-reliability`) is installed from its repository, not declared.** It is not
on PyPI and needs Python 3.12, so `tools/eval_judge.py` needs it installed by hand (Quickstart).
Committed results stay readable without it, the code raises a message naming the install rather
than an `ImportError`, and the workflow runs its suite in a separate job that is skipped under a
name saying so when no repository URL is configured — never passed quietly.
- **The grounding gate is weak by construction, and thin on this corpus.** It compares claims
against evidence *mechanically*: it cannot tell that a correctly-quoted number is being
misinterpreted, it says nothing about relevance, and a claim containing no number, quotation or
identifier receives only the citation checks — 4 of 6 claims in the demo run. Its measured
strength is exactly the seven corruptions in `examples/grounding_demo.py` and no more.
- **The MCP server streams nothing and reports no progress.** It does not need to: an offline run
answers in single-digit milliseconds once the index is built, so there is nothing to stream. That
is a property of the stub agents rather than of the design, and it is the first thing a live model
path would have to change.
- **The two error channels are this repo's, not the whole surface's.** `InvalidRequest` becomes a
JSON-RPC `INVALID_PARAMS` and `ExecutionFailed` is re-raised as the SDK's `ToolError`, which
becomes an `isError` result *carrying the message* — the one exception class every `mcp` 2.x
in the declared range forwards the text of. From 2.1 the SDK treats any other exception as a
crash and sends the client only `Error executing tool <name>`; the flag survives that, the
reason does not, so the test and the demo assert the message rather than the flag. The SDK's
own argument validation, on the other hand, reports an out-of-schema value as an `isError`
result rather than as a protocol error. A test pins that too, including the half that
contradicts what the design would prefer.
- **One retrieval arm per server process.** The arm is fixed at construction rather than chosen per
call, because three indexes is three builds and a second of startup for a knob whose measured
effect on this corpus is not distinguishable from zero.
- **The offline agents are deterministic stand-ins, not a model.** The synthesizer selects sentences
from retrieved chunks, so answers read as a list of quotations and cannot paraphrase, aggregate or
reason across chunks. Its output is therefore **grounded by construction**, which means the stub
path proves nothing about whether the grounding gate (item 6) catches hallucination — that is what
`agentarch.agents.adversarial` and the eventual live path are for. A "100% citation rate" measured
here would be a meaningless number and is not reported as one.
- **No token or cost figures on the offline path.** No tokenizer is involved, so any count would be
a number this repo cannot reproduce. `max_tokens` is enforced against whatever usage a runner
reports and does nothing when nothing reports usage. Money is not reported at all — see
[DESIGN.md](DESIGN.md) §8.
- **Durations are opt-in and partial.** `TimingRunner` is not installed on the default path (a
wall-clock stamp breaks byte-identical traces across transports, which is a tested property), and
what it measures is time inside the effect boundary only — records carry no timestamps, so time
*between* effects is neither recorded nor estimated, and the viewer presents no wall-clock total.
- **The critic's contradiction detector is a lexical heuristic** (shared identifier, disjoint
numeric literals). It will miss disagreements phrased in prose and flag pairs that merely quote
the same rule with different examples. It reports; it never arbitrates and never sets the verdict.
- **A run that fails at the effect boundary is not replayable**, by design — see above. "Any
recorded run is a regression test" is true of `complete`, `incomplete`, `stalled` and
malformed-hand-off `failed` runs, and false of boundary failures.
- **Retrieval quality bounds everything above it.** Asked about a made-up "zzzqqx protocol", BM25
matches the one real word in the query and the extractive synthesizer dutifully quotes sentences
about protocols. The run is marked partial and the unanswerable half is reported as a gap, but the
answered half is still weak — a reranker or a relevance floor is the fix, and neither exists yet.
- **The dense arm is LSA, not a learned embedding.** Its notion of "related" is corpus
co-occurrence only; a query term absent from the corpus vocabulary contributes nothing; word
order is invisible. A sentence-transformer arm would very likely score better, and would make the
quickstart download weights. The hybrid result above is a statement about *this* dense arm and is
not evidence about hybrid search in general.
- **No reranking.** A cross-encoder is the standard next stage and would plausibly move the numbers
more than the fusion rule does. Nothing here is claimed about it.
- The hybrid page selects three times on one 56-query set — rank, then vocabulary floor, then
fusion configuration, each conditional on the last — and corrects for multiplicity nowhere.
Across ~40 uncorrected intervals, roughly two would exclude zero by chance. One more reason to
read the hybrid headline as "not shown to help", and to hold the nDCG result loosely.
- The retrieval numbers describe *this* corpus and *this* 56-query set. They are not a claim about
retrieval in general, and the corpus deliberately excludes the near-duplicate historical spec
revisions that make real documentation retrieval hard — see the
[corpus card](corpus/CORPUS_CARD.md).
- Recall is a **lower bound**: no judgement pooling was done, so a section that answers a query but
was never labelled counts as a miss.
- Replay reproduces *effects*, not the outside world. It proves the agent's decision sequence is
unchanged given the same responses; it does not prove the model would answer the same way again.
- Order-strict replay per lane is the right check for this design, but it means a genuinely
nondeterministic scheduler *within* a lane would show up as divergence. Lanes exist so that
parallel workers do not have to reproduce a global interleaving.
## Licence
MIT — see [LICENSE](LICENSE).