# agentic-rag

[![CI](https://github.com/trentmilam/agentic-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/trentmilam/agentic-rag/actions/workflows/ci.yml)

Cited, revision-aware retrieval over a **real** corpus drawn from the IETF RFC
ecosystem -- RFC full text, the RFC index's real Obsoletes/Obsoleted-by/Updates
supersession graph, real community-submitted RFC errata, and real IANA
protocol-parameter registries -- fetched over HTTPS (`corpus_fetch/`) and
ingested through [`ragpack`](packages/ragpack) into a searchable vector store
(321,124 real chunks), then routed and citation-gated through the
[`consilium`](packages/consilium) spine.

Ask a real question about an IETF protocol spec, an RFC's authorship/status, a
real errata correction, or an IANA registry entry, and get back an answer built
entirely out of **real cited chunks -- or an honest abstain**. The one exception
is an obsoletion question ("what replaced RFC 2616?"): there is no "current
revision" of RFC 2616's own text to retrieve (it was entirely superseded by six
later documents, RFC 7230-7235 -- IETF's own famous multi-way obsoletion case),
so that is answered by `SupersessionModule`, a deterministic graph lookup over
the real Obsoletes/Obsoleted-by graph, not retrieval. **No LLM anywhere in the
answer path.**

## Quickstart

```bat
git clone https://github.com/trentmilam/agentic-rag
cd agentic-rag
py -3.12 -m venv .venv
.venv\Scripts\python -m pip install -r requirements.txt
.venv\Scripts\python -m pip install -e .
.venv\Scripts\python scripts\verify.py
```

That is the whole bootstrap. The editable install is what makes `import consilium`,
`import ragpack`, `import linkgraph`, `import activerag` and `import chainrag` resolve --
`pyproject.toml`'s `package-dir` maps each name into `packages/`.

Answering a real question additionally needs the ingested corpus; see
[The corpus](#the-corpus) and [Verify](#verify).

## One repo, five merged tools

`agentic-rag` used to import `consilium` and `linkgraph` from sibling repositories by
inserting their roots on `sys.path`, and required `RAGpack` pip-installed editable from a
third. Cloning it meant cloning four repos at three commits pinned in three separate files
-- which had already drifted apart from each other. `linkgraph` also read this repo's data
file, so those two imported each other across a repository boundary.

They are one repo now. Each merged tool kept its whole tree under `packages/<name>/`, so
everything it resolves relative to its own root still resolves, and each kept its own
commit history rather than being squashed into an import commit:

| package | what it is | tests |
|---|---|---|
| [`packages/consilium`](packages/consilium) | the routing / citation-gating spine: Registry, Router, compose, integrity gate, ComputeModule | 5 eval suites |
| [`packages/ragpack`](packages/ragpack) | ingest, chunking, embedding, the Qdrant store wrapper | 20 |
| [`packages/linkgraph`](packages/linkgraph) | the cross-document relationship graph behind the MCP relationship tools | 50 |
| [`packages/activerag`](packages/activerag) | evidence-thinness detection and bounded hunt-and-retry | 54 |
| [`packages/chainrag`](packages/chainrag) | a second vertical over blockchain protocol docs, proving the spine is not corpus-specific | -- |

They are nested under `packages/` rather than sitting at the repo root for a concrete
reason: a top-level directory named `consilium` shadows the `consilium` package as an
implicit namespace package, because the working directory precedes the editable install on
`sys.path`. Imports then resolve to an empty namespace and `consilium.__file__` is `None`.
CI asserts every package resolves to a file inside this repo, so that class of silent
mis-binding fails the build instead of passing every test against the wrong copy.

`rag-reliability` is the one dependency still outside this repo. It supplies `graphrx`,
`headroom`, `vecstamp`, `chunkledger`, `plumbline` and `legigate` to three of the tools
above, and is resolved by path today; packaging it so it can be a declared dependency is
the outstanding follow-up. CI checks it out pinned and **fails if those integration tests
skip**, so the one remaining path dependency cannot rot unnoticed.

## The corpus

| source type | what it is | real chunks | current (non-obsoleted) |
|---|---|---:|---:|
| `rfc_text` | verbatim RFC full text | 306,939 | 225,809 |
| `rfc_index` | per-RFC index card (title/authors/date/status/obsoletes/updates) | 5,854 | 4,663 |
| `errata` | real community-submitted RFC corrections | 7,295 | 7,295 (n/a -- no revision concept) |
| `iana_registry` | 7 real IANA protocol-parameter registries, rendered as markdown tables | 1,036 | 1,036 (n/a -- no revision concept) |

321,124 total real chunks, embedded with the real `BAAI/bge-base-en-v1.5`
model. `rfc_text`/`rfc_index` are the only source types with a revision
concept -- `is_current` there is a real structural fact (see [the revision
guard](#the-revision-guard-proven)), not guessed.

## Five modules, one router

`agenticrag/bootstrap.py::build_registry` assembles a real 5-module consilium
`Registry`: the 4 retrieval modules above (each loaded straight from Qdrant,
`current_only=True` by default -- see `agenticrag/registry_loader.py`), plus
`SupersessionModule` (`agenticrag/supersession.py`), a `ComputeModule` that
parses `data/entities/revisions.json` -- the real Obsoletes/Obsoleted-by graph
for all 9,794 RFCs in the live index -- once at construction and answers
obsoletion questions with a deterministic, cycle-safe graph walk (bounded at 50
visited nodes -- a defensive cap, not an expected real limit; real IETF
obsoletion components are small).

`build_registry` also fails loud if the process's configured embedder does not
match the one that ingested the store (`verify_embedder_marker`, called first) --
otherwise cosine scores would be silently meaningless -- and closes any Qdrant
client it opened itself once the modules are loaded (retrieval is fully
in-memory afterward), so it does not hold the local-mode store's lock for the
caller's whole process lifetime.

`errata`'s `trust_tier` (0.55, well below `rfc_text`/`rfc_index`/`iana_registry`'s
0.9-0.95) is measured, not guessed: of 5,061 real errata records ingested,
**only 2,400 (47.4%) are `Verified`** by the RFC Editor -- 1,781 (35.2%) are
`Held for Document Update`, 679 (13.4%) are outright `Rejected`, and 201 (4.0%)
are still `Reported`. Fewer than half of real submitted corrections are
RFC-Editor-confirmed, so this module corrects the primary text without being
uniformly authoritative itself.

## MCP server

`agenticrag/mcp/server.py` exposes the answer path as a
[Model Context Protocol](https://modelcontextprotocol.io) server over stdio
(`mcp>=1.28.1`, FastMCP). Four tools:

- `search(query)` -- the full cited/abstain answer path (`consilium.compute.answer_v3`), passed through verbatim (an honest abstain stays an abstain);
- `get_obsoletion_chain(rfc_id)` / `get_corrections(rfc_id)` / `get_related(entity_id)` -- the relationship graph, via the `linkgraph` sibling (`agenticrag/relationships.py`); each returns a documented `{"ok": false, "fallback": ...}` envelope if that sibling is absent.

Run it: `.venv\Scripts\python -m agenticrag.mcp.server` (stdio transport -- an
MCP client launches it and speaks the protocol over stdin/stdout). The tool
*logic* is import-light and has no `mcp`/Qdrant dependency, so it unit-tests
without either (`agenticrag/mcp/test_server.py`).

## Verify

**Quick verify** (fast, corpus-free -- no ingested corpus needed):

```bat
verify.bat        :: or:  .venv\Scripts\python -m pytest -q
```

Runs the MCP tool-wrapper tests and the supersession graph-walk cycle-safety
suite. This is what CI runs on every push (see the badge above).

**Full verify** (needs the ingested 321k-chunk corpus from the Quickstart):

```bat
.venv\Scripts\python eval\eval_agenticrag.py
```

Deterministic given the already-ingested corpus; no re-ingestion. Uses the real
embedder -- the exact one that ingested the corpus -- so this proves genuine
semantic retrieval end-to-end, not wiring alone. Takes ~3.3 min on this
hardware (measured, full run: 196s): a one-time ~60s registry build (a single
scan of the corpus for the small poison-quarantine set -- not a load of all 321k
vectors), then the router/answer passes at ~12s each. Each pass runs routing and
retrieval as native Qdrant vector searches; Qdrant's embedded local mode is exact
brute-force (no ANN index), so a search still scans the filtered subset -- but in
native code, materializing only the top-k, ~16x faster than the old pure-Python
per-chunk scan. (Sub-second search would need Qdrant server mode's HNSW index;
local mode keeps the repo self-contained -- no server to run.) It
checks: one in-scope
query per source type (sensible module + >=1 real citation); the RFC 2616
obsoletion query (successor set is **exactly** `{7230, 7231, 7232, 7233, 7234,
7235}`); one out-of-scope query (honest abstain); the revision-guard structural
exclusion (below); a genuinely current RFC (791) resolving `current`; and an
absent RFC number (99999) resolving `not_found`, not a crash.

`eval/prove_revision_guard.py` is the same structural proof as a standalone,
narrated script. `eval/smoke_ingest_real.py` proves the fetch->ingest->Qdrant
wiring with the zero-cost `HashEmbedder` (needs `corpus_fetch.fetch_all` to have
run, but no GPU/embedding model).

<a name="the-revision-guard-proven"></a>
## The revision guard, proven

RFC 2616 (HTTP/1.1) is a real, single, whole document -- there is no "current
revision" of it; it was entirely superseded by six different documents. The
guard property here is two real, checkable facts, not one "corrected value":

1. **Structural exclusion.** RFC 2616's own `rfc_text` chunks exist in Qdrant
   (loading the `rfc_text` module a second time with `current_only=False`
   proves they're there) but are absent from the default `current_only=True`
   module every query actually uses -- because
   `data/entities/revisions.json["RFC2616"]["obsoleted_by"]` is real and
   non-empty. The data exists; it's structurally filtered, not accidentally
   missing.
2. **The correct path to the answer.** `SupersessionModule` is the explicit way
   to learn what happened to RFC 2616 -- a query naming it returns the real
   6-way successor list (RFC 7230-7235).

## Router calibration -- measured, not guessed

`consilium.router.Router`'s stated library defaults (`floor=0.11`,
`anchor_centroid=0.25`, `anchor_best_chunk=0.25`) assume a near-zero baseline
cosine between unrelated text -- true for a bag-of-words `HashEmbedder`, not
necessarily true for a real dense embedder over a 321k-chunk corpus (the same
gap shows up even at a much smaller ~1,100-chunk scale). `agenticrag/calibrate.py`
measures this directly against the real corpus + real embedder rather than
assuming it; see `agenticrag/bootstrap.py::ROUTER_KWARGS` for the resulting
per-instance kwargs and the real numbers that justified the decision.

```bat
.venv\Scripts\python agenticrag\calibrate.py
```

## GPU note (ingest only)

GPU matters for exactly one step: the corpus **ingest** (`ingest/run_ingest.py`),
which embeds ~321k chunks with `BAAI/bge-base-en-v1.5` via `onnxruntime`. At
**query time** you only embed the (short) query string, so the launchers run fine
on CPU with no GPU setup -- which is why they no longer touch any GPU config.

Expected-and-harmless warning: if `onnxruntime-gpu` is installed but its matching
CUDA runtime DLLs aren't on the search path, you'll see an alarming red
`CUDAExecutionProvider` / `Error loading ... cublasLt64_*.dll ... missing` block --
**on any run, ingest or query**, not just ingest. It is not a failure: onnxruntime
falls back to CPU and continues (for a query the embed is one short string, so the
CPU fallback is instant). Ignore it, or install the DLLs below to silence it.

For a fast ingest, `onnxruntime-gpu`'s CUDA execution provider needs those CUDA
runtime DLLs on the DLL search path. `pip install onnxruntime-gpu` alone does
**not** bundle them, and without them onnxruntime falls back to CPU (measured:
~6 chunks/sec -- computed: ≈14.9 hours for the full 321,124-chunk corpus) rather
than raising. To get real GPU execution (measured: ~650 chunks/sec on an RTX 5090,
i.e. ≈8 minutes for the full corpus), install the matching CUDA runtime wheels
into this repo's own venv, e.g.:

```bat
.venv\Scripts\python -m pip install nvidia-cublas-cu13 nvidia-cudnn-cu13
```

(match the `-cuNN` suffix to your `onnxruntime-gpu` build's CUDA major version),
or put any CUDA-enabled PyTorch install's `torch/lib` directory on `PATH` before
running the ingest. Either way this is an **optional acceleration of the one-time
ingest**, never required to run the demo or serve queries.

## Ingest re-runs (known limitation)

`ingest/run_ingest.py` supports a from-scratch `--recreate` rebuild and a fast
incremental path (only re-embeds raw files whose content hash changed). The
incremental path has two known limitations, deliberately not papered over: (1) it
keys "changed" purely on each raw file's byte hash, so if an RFC becomes newly
obsoleted in a later `rfc-index.txt` refresh *without its own text file
changing*, its `is_current` flag can go stale; and (2) it does not delete
orphaned Qdrant points for a document that re-ingests to fewer chunks. For a
guaranteed-consistent store, run `ingest/run_ingest.py --recreate` (a full
rebuild). Closing both limitations in the incremental path is future work.

## Layout

```
agenticrag/
  embed_config.py     shared Settings (model/qdrant path) + embedder-consistency guard
  registry_loader.py  loads a consilium Module's chunks straight from Qdrant (current_only guard)
  bootstrap.py        build_registry(embedder, client=None) -> Registry; the 5 Descriptors; ROUTER_KWARGS
  supersession.py     SupersessionModule -- real Obsoletes/Obsoleted-by graph walk, cycle-safe
  relationships.py    thin bridge into linkgraph (get_related / _obsoletion_chain / _corrections)
  calibrate.py        real router-score measurement script
  mcp/
    server.py         FastMCP server: search + the 3 relationship tools (stdio)
    test_server.py    fixture-only tests for the tool logic (no mcp package, no Qdrant)
corpus_fetch/         real HTTPS fetch of RFC full text / rfc-index.txt / errata / IANA registries
ingest/
  connectors/         per-source-type extract() -> ExtractedDoc (+ the revisions-index builder)
  run_ingest.py       raw files -> chunk -> embed -> Qdrant, real is_current currency check
eval/
  smoke_ingest_real.py               corpus_fetch -> ingest -> Qdrant wiring smoke (HashEmbedder)
  eval_agenticrag.py                 production eval (full verify; needs the ingested corpus)
  prove_revision_guard.py            standalone, narrated revision-guard proof
  test_supersession_cycle_safety.py  cycle-safety unit tests (synthetic graph; corpus-free)
tests/                unit tests for connectors / registry_loader / bootstrap (corpus-free)
packages/             the five merged tools, each keeping its own tree and history
  consilium/          routing / citation-gating spine (+ its 5 eval suites)
  ragpack/            ingest / chunk / embed / Qdrant store (src-layout)
  linkgraph/          cross-document relationship graph
  activerag/          evidence-thinness detection and bounded hunt-and-retry
  chainrag/           the blockchain-docs vertical
app.py                gr.ChatInterface chat UI
run_demo.py           scripted 3-question narrated transcript
```

## License

Code: [MIT](LICENSE) (c) 2026 Trent Milam.

The corpus is **not** included in this repo (`data/` is gitignored); it is
fetched at build time from rfc-editor.org and iana.org. IETF RFC/errata text is
subject to the [IETF Trust Legal Provisions](https://trustee.ietf.org/documents/trust-legal-provisions/)
(the fetch preserves each document's own copyright/Trust notice intact); IANA
registry data is published by IANA. This project redistributes none of it -- it
fetches it, locally, on your machine.