agentic-rag
by phense
README.md
# agentic-rag
### Provider-neutral long-term memory and compaction continuity — in a real database.
**Coding sessions end and long contexts compact. agentic-rag preserves both:
a canonical, searchable knowledge base in local PostgreSQL + pgvector, plus
bounded checkpoints that let Claude Code, Codex, and Antigravity resume after compaction. Hybrid vector +
full-text search, lifecycle hooks, and a provider CLI you control — without a
hosted RAG service.**
[](LICENSE)
[](pyproject.toml)
[](pyproject.toml)
[](https://github.com/pgvector/pgvector)
**0.6.34 dashboard refresh:** clearer grouped navigation, consistent cards, tables
and status badges, readable narrow-screen layouts and improved text contrast in
the sampled synthetic administrator views. Behavior,
permissions and stored data are unchanged; see the [dashboard reference](docs/dashboard-reference.md#layout-and-accessibility).
**0.6.33 embedding comparison:** optional task prefixes preserve model-bound
generation identities. The [local BGE/Jina CPU results](docs/benchmarks/2026-11-04-embedding-comparison/README.md)
recommend retaining BGE; Nano has faster model calls but mixed independent ranking,
and Small exceeds the frozen budgets. Updating code selects no model and rewrites
no vectors.
**0.6.29 signed webhooks:** explicitly subscribe to committed memory, review,
source, lifecycle or owned ingestion-job changes. A separate local sender posts
minimal signed notices with scoped authorization rechecks, bounded retries and
stable event IDs. Capture starts off; upgrades configure no receiver and send
nothing. Migration038 preserves existing knowledge and vectors.
Use the [webhook operations playbook](docs/webhooks.md).
**0.6.28 scoped feedback:** explicitly capture a search and label a result useful,
not useful, the wrong source or missing expected evidence. Writers can revise and
retract their own signals; feedback changes neither knowledge nor ranking. A local
operator can approve a safe query for private, read-only failure evaluation.
It is off by default and adds schema037 without rewriting existing data.
Follow [PB07101](docs/retrieval-feedback.md).
**0.6.25 source-backed images:** screenshots, receipts and scans can become
reviewable knowledge. agentic-rag keeps the exact image in PostgreSQL, reads its
text with a Tesseract you install yourself, and records where each word sits;
an optional local Ollama model can describe the picture. Nothing is searchable
before you confirm it against the original. It is off by default, adds
migration036 without touching existing data, and calls no image engine during
the upgrade. Follow the [image playbook](docs/image-ingestion.md).
**0.6.22 lexical search:** selective full-text queries run expensive truth and
lifecycle checks once per matching document, retaining the same scoped results
and ranking. Generic vector plans continue past filtered nearest neighbours on
pgvector0.8+, with an exact fallback when the scan cannot fill its candidate
pool. Supported0.7 installations use that exact path. Truth and source checks
remain before ranking and limits; retrieval settings restore on return.
Service embedding preparation also retains worker priority and
background timeout rules. Upgrade existing0.6.21 stores with the
[lexical-search upgrade procedure](docs/playbooks/agent-plugins.md#0621-to-0622-lexical-search-upgrade).
**0.6.21 native source continuity:** generated compaction summaries no longer
become original user evidence. A bounded, audited quarantine repairs old source
classification; migration033 preserves original records and ordinary no-revival
rules. Use the [source-origin upgrade gate](docs/playbooks/agent-plugins.md#0620-to-0621-native-source-origin-gate-acceptance-draft)
before restarting capture. Native synthetic observations are published under
[the acceptance fixtures](docs/benchmarks/2026-11-01-native-plugin-acceptance/).
**0.6.20 extraction policy:** an operator can decide in one versioned file which
mined knowledge is kept per domain or project: output categories, item and text
limits, plus advisory guidance for the provider. Admitted work keeps its policy
across retries, a change only affects new work, and without a file nothing
changes. Upgrade with the [operations playbook](docs/playbooks/system/rag-067-extraction-policy.md)
and read [extraction policy](docs/extraction-policy.md).
**0.6.19 memory lifecycle:** an operator can mark one memory as a duplicate,
extension, replaced version or contradiction of another after a read-only preview.
The decision is a separate audited record: the memory itself, its evidence and its
links stay untouched, search shows only the current one, and a correction makes it
visible again. Automatic curation now merges only copies with the same title and
type. Upgrade with the [operations playbook](docs/playbooks/system/rag-066-memory-lifecycle.md)
and read [memory lifecycle](docs/memory-lifecycle.md).
**0.6.18 pattern synthesis:** an operator can let a model suggest recurring
patterns across at least three independent user reports in one domain and project.
Each suggestion is stored as a reviewable inference with exact citations and
disappears from every read as soon as a cited source changes. It is off by default;
upgrade with the [operations playbook](docs/playbooks/system/rag-065-pattern-synthesis.md)
and read [pattern synthesis](docs/pattern-synthesis.md).
**0.6.17 temporal search:** searches can recognize yesterday/gestern, last
week/letzte Woche, next month/nächsten Monat and explicit dates, then order or
filter hits by their recorded event time without widening access or validity.
It is off by default and needs no migration; read [temporal search](docs/temporal-query.md).
**0.6.16 source evidence:** every client can inspect the exact stored quotes,
source trust, claim review and fact lifecycle behind a memory, and reviews are
bound to the evidence version that was inspected. That release introduced
schema029; follow its verified [upgrade playbook](docs/playbooks/system/rag-063-source-evidence-upgrade.md)
and read [claim evidence](docs/claim-evidence.md).
**0.6.15 reranker Settings:** administrators can select MiniLM L6 EN/DE,
mMARCO DE/EN (default) or Jina tiny (English only). Its additive schema028
is included in the 0.6.16 upgrade; use the verified [upgrade and model procedure](docs/reranker-settings.md).
Knowledge and existing embeddings are preserved.
**0.6.13 request tracing:** metadata-only request IDs and authorized job diagnostics
are available through the dashboard, REST and CLI. Read the
[tracing operations guide](docs/16-request-tracing.md). Existing service stores
introduced additive schema026; current startup requires030 as described above.
**0.6.12 dashboard:** authenticated setup, enrolled-client administration and scoped
knowledge/source browsing, served by the same engine. Read the
[dashboard operations guide](docs/15-dashboard.md) and [access/limits reference](docs/dashboard-reference.md).
That release introduced the verified additive schema025 upgrade; standalone CLI/stdio operation remains available.
> **New in v0.6.0:** ten retrieval and ingestion features, including original-backed
> entity aliases, bounded embedding reuse and offline confirmed-failure evaluation.
> Features 1–10 are adopted locally on schema 001–019. Read
> **[What’s New in 0.6.0](docs/00-whats-new-in-0.6.md)** and the
> [latest production acceptance](docs/verification/features9-10-production-adoption.md).
>
> **0.5.0:** Antigravity CLI (`agy`, Gemini) compaction continuity —
> `rag install --agy`, SessionStart/PreInvocation/Stop hooks, `/compact`
> handoff, automatic-compaction detection, Antigravity transcript mining. Read
> **[What’s New in 0.5.0](docs/00-whats-new-in-0.5.md)**.
>
> **0.4.0:** Claude compaction continuity — six Claude hooks,
> the managed 1M/500K `autoCompactWindow` policy, the `compact_summary`
> handoff, and `rag install --check`/`--restore`. Read
> **[What’s New in 0.4.0](docs/00-whats-new-in-0.4.md)**.
>
> **0.3.0:** Codex compaction continuity, native-memory policy,
> recoverable global installation, and provider-neutral mining. Read
> **[What’s New in 0.3.0](docs/00-whats-new-in-0.3.md)**.
Most "RAG memory" tools are a cloud retrieval layer you feed documents to:
you push, you query, you pay per call. agentic-rag flips both halves. It stores
knowledge in **local Postgres + pgvector** — real HNSW
approximate-nearest-neighbour search blended with bilingual full-text — and it
**populates itself from supported coding sessions**. It also stores compact,
audited continuation checkpoints at Claude Code, Codex, and Antigravity
compaction boundaries — on Claude including Claude's own compact summary as a bounded
handoff.
Every content write funnels through **one gateway**: it strips secret-shaped
tokens, chunks and embeds the text with a local model, resolves the document's
links into a **typed knowledge graph**, and logs the change — all in one
transaction. When a session ends, a single-writer worker uses the configured
extraction provider to turn the bounded transcript digest into durable,
findable memories. The core data model and provider seam are provider-neutral;
integrations adapt each coding agent's lifecycle and output contracts.
It runs on your machine and uses **your configured provider account** for
LLM-assisted mining, curation, and bounded checkpoint enrichment: Codex/Claude
CLI authentication or an explicitly configured compatible HTTP endpoint.
Mining prompts can also include all matching pin bodies; mining secret-strips
the provider-bound copies without mutating stored pin text.
Embeddings default to **local Ollama**. Selecting an external managed generation
sends future search queries and semantic duplicate candidate title/body to its
provider; explicit bounded builds send stored chunks. Default research skips structured assessment; its retrieval uses the selected embedding profile. [Bounded research](docs/bounded-research.md) follows eligible entity aliases and up to three graph hops, with per-facet literal/semantic coverage and original-source paths.
collects evidence for compound questions and reports disagreement or missing
support; `--provider` explicitly enables configured-provider assessment.
An optional [local multilingual reranker](docs/local-reranker.md)
can improve ambiguous-source ordering when its measured gains justify extra latency. The default installation adds no always-on daemon beyond Postgres and
Ollama. Optional neural reranking uses a separate local model process with its
[measured budgets and configuration choices](docs/benchmarks/2026-10-30-measured-reranker/README.md). And it's built
data-safety-first — it archives rather than deletes, writes through a
least-privilege role matrix, audits every change, and periodically
restore-tests its own backups.
Bounded profiles include [incremental thematic
summaries](docs/thematic-summaries.md): local original excerpts with source/version
drill-down, trust/temporal invalidation and audited reuse. The additive017 cache
preserves baseline reads on016. [Paired measurements](docs/benchmarks/2026-10-03-thematic-summaries/README.md)
record compression, coverage and additional original-corpus query cost.
[Entity identities and scoped aliases](docs/entity-identities.md) link confirmed alternate names to original facts.
`rag entity resolve` retrieves original facts under confirmed alternate names;
explicit source-backed links preserve scope/domain boundaries and support audited
revocation. Existing exact-name search and assertion keys remain unchanged.
[PB-5.8](docs/playbooks/entity-identities.md) records the supported 017→018 upgrade
and code recovery. [Incremental ingestion](docs/incremental-ingestion.md) reuses
exact sanitized embedding inputs under a verified local model identity, with bounded
preprocessing and ordered audited writes. [Confirmed-failure evaluation](docs/failure-evaluation.md)
compares offline routes and public mining prompts while retaining original evidence.
The supported 018→019 upgrade and recovery are in [PB-5.9](docs/playbooks/incremental-ingestion.md);
[PB-5.10](docs/playbooks/failure-evaluation.md) covers evaluation and code-only adoption.
Sealed benchmark candidates do not change production search or mining policy.
> **Your data stays under your control, with explicit provider calls.** This
> repository is **code only** — it ships no content. The canonical store lives
> in *your* PostgreSQL database, but the configured extraction provider intentionally sends
> these provider inputs: mining sends a bounded, secret-stripped transcript
> digest, live domain names, and secret-stripped copies of all matching pin
> bodies without mutating stored pin text; curation sends selected stored
> documents and contradiction evidence; checkpoint enrichment sends a
> secret-stripped transcript delta and validates the returned checkpoint
> content before persistence. Explicit research provider mode sends a redacted
> question and selected bounded source excerpts; default research skips structured assessment; retrieval still uses the selected embedding profile.
> Optional [managed provider profiles](docs/provider-indexes.md) can send search
> queries and explicitly selected rebuild batches to their configured HTTP
> destination. Selecting an external generation also sends future semantic
> duplicate candidate title/body there. Inspection is offline; probes use fixed synthetic inputs.
> Optional synced backups copy data only to a
> directory you configure. agentic-rag has no separate hosted RAG backend.
**[Why](#why-agentic-rag)** · **[Quick start](#quick-start)** · **[What's different](#what-makes-it-different)** · **[How it works](#how-it-works)** · **[Comparison](#comparison)** · **[Configuration](#configuration)** · **[📖 Handbook](#-documentation--handbook)** · **[Status](#status)** · **[Acknowledgments](#acknowledgments)** · **[License](#license)**
---
## Why agentic-rag
🔎 **Hybrid search that actually ranks.** Vector ANN over `pgvector` (HNSW, cosine) — multilingual by way of `bge-m3` embeddings — blended with GIN keyword full-text into one ranked query. Search in **any language**; not a file scan, not lexical-only.
🌱 **It turns sessions into durable knowledge and continuation state.** Mining
uses your configured extraction provider (Codex/Claude CLI or opt-in HTTP). On Claude Code and Codex,
`PreCompact` also captures a fast checkpoint so `SessionStart(source="compact")`
can restore the goal, blockers, next action, repository state, and evidence
references; on Claude the checkpoint also carries Claude's compact summary.
♻️ **It curates itself.** A near-duplicate gate stops the store from bloating; `rag review` surfaces duplicates, dangling links, and stale pins; refuting a fact **archives** it (with a reason and evidence), never hard-deletes it.
🔒 **Local-first, on your own account.** Canonical knowledge and checkpoints
live in your Postgres. Structured LLM work uses your configured CLI or opt-in
HTTP provider. Embeddings default to local Ollama; external managed profiles
receive queries, semantic duplicate candidates and explicit rebuild batches.
Default research skips structured assessment; its retrieval uses the selected embedding profile.
⚡ **RAM-lean.** A single-writer worker (flock singleton), no long-lived daemon of its own. Without the optional reranker runtime, the between-session footprint is essentially Postgres + Ollama idling.
---
## Quick start
**OpenCode / DeepSeek:** [install the native RAG adapter](docs/opencode.md) on the execution host with `uv run rag install --opencode`. It supports startup context, selective recall, checkpoint/handoff and idle mining. For T3 on Windows, install on the Mac execution host; canonical context and a real read-only RAG tool call were [verified through T3](docs/verification/opencode.md#production-t3-read-path-check). See the documented limits.
agentic-rag is a `rag` command-line tool with provider integrations. The
no-option install wires two MCP servers, six lifecycle hooks, and the managed
compaction window into Claude Code; the explicit Codex and Antigravity
(`--agy`) targets install continuity configuration and hooks.
**Prerequisites:**
- **PostgreSQL 17** with the [`pgvector`](https://github.com/pgvector/pgvector) extension (the schema uses `halfvec`, pgvector ≥ 0.7).
- **[Ollama](https://ollama.com)** with the embedding model pulled — `ollama pull bge-m3` (1024-dim, fixed to the schema).
- An authenticated LLM CLI: **Codex** (`codex login`) or **Claude** (`claude -p`). Claude/Haiku remains the package default for compatibility; select the provider in `[llm]`.
- **[`uv`](https://docs.astral.sh/uv/)** and **Python ≥ 3.13**.
**Install the common foundation, then choose an integration:**
```bash
uv sync
uv run rag init-db # creates the DB + schema + roles, seeds the 'general' domain
uv run rag domain add programming --description "Software engineering notes"
uv run rag install --check # preview the Claude settings merge; writes nothing
uv run rag install # Claude MCP/hooks + macOS backup schedule; omit for Codex-only
uv run rag install --agy # Antigravity CLI (agy) hooks.json target; --check previews
```
- `rag init-db` creates the database if needed, applies the migrations in `sql/`, creates the three least-privilege roles, and seeds the built-in `general` domain. **Run it first** — `rag install` does *not* create the database.
- `rag domain add <name>` adds any domains you want to organize documents under (`general` always exists; add more anytime).
- `rag install --check` previews the Claude merge (`managed:
autoCompactWindow=500000`, the would-change path, policy warnings) and
writes nothing.
- The no-option `rag install` is the Claude target: it registers the
`agentic-rag` (read-write) and `agentic-rag-ro` (read-only) MCP servers,
merges six hooks (`SessionStart`, `UserPromptSubmit`, `Stop`, `PreCompact`,
`PostCompact`, `SessionEnd`) plus `autoCompactWindow = 500000` into
`~/.claude/settings.json`, backs the file up to a unique
`settings.json.bak.<id>`, and prints a `rag install --restore <record>`
rollback command. On macOS it also schedules nightly backup; omit this
command for a Codex-only setup.
If you ran the Claude target, hooks reload live; start a new Claude Code
session so it picks up the MCP servers, then review the handlers with `/hooks`
and confirm `/autocompact` reports 500000 tokens from settings. For humans the
same store is available through the CLI:
```bash
rag save --title "Postgres VACUUM tuning" --domain programming \
--dtype lesson --body "autovacuum_vacuum_scale_factor tradeoffs..."
rag search "vacuum tuning" --domain programming
rag get <slug-or-id> # body + incoming/outgoing graph edges
rag status # counts, queue health, last backup/curation
```
For Codex continuity, preview before touching your user configuration, install,
then inspect and trust the changed commands in Codex:
```bash
uv run rag install --codex --check
uv run rag install --codex
# Start Codex, run /hooks, inspect all six agentic-rag commands, then trust them.
uv run rag status
```
The Codex transaction manages only `~/.codex/config.toml`,
`~/.codex/hooks.json`, and `~/.codex/compact_prompt.md`. It prints every
changed path, backup, validation result, and a ready-to-run rollback command:
```bash
uv run rag install --codex --restore /absolute/path/to/codex-rollback-<id>.json
```
Use the exact rollback-record pathname printed by the successful install.
Check mode writes nothing. Repository support does not mean a particular
machine has completed hook trust or live compaction verification; review
`/hooks` and confirm `rag status` after every installation.
The Codex target never installs a scheduler. On a Codex-only macOS setup, use
`uv run rag backup --install-launchd` for scheduled database backups; Linux
uses the handbook's cron/systemd recipes.
---
## What makes it different
Four things that, together, set it apart from both file-based knowledge wikis and hosted RAG stacks:
### 1. Real hybrid search over a curated graph
Documents are chunked and embedded into `halfvec(1024)` columns indexed with **HNSW**, and each chunk is embedded with the multilingual `bge-m3` model — so semantic recall works in any language — and also carries generated `tsvector`s for English/German keyword full-text. A single query runs vector ANN and full-text together and returns **one ranked list**. Documents aren't an undifferentiated pile: they carry a type (`concept`, `lesson`, `signal`, `synthesis`, `reference`, …) and connect through a **typed edge graph** (`references`, `extends`, `depends_on`, `supersedes`, `contradicts`, …), so `rag get` shows you not just a document but its neighbourhood.
### 2. Automatic session-mining — the star feature
This is what makes agentic-rag feel like it grows rather than sits there. When a supported coding session ends, a lifecycle hook enqueues the transcript. A **single-writer worker** drains the queue and calls the configured extraction provider to pull out durable memories, lessons, and signals, each saved through the write gateway behind a **near-duplicate gate**. A fix you discovered today becomes something a future session can recall, with no "remember to write this down" step. It reads only your **local** session transcripts.
### 3. Knowledge domains you grow and curate
Domains are just data — a label for *where to look* (`general` is seeded at init). Add them with `rag domain add`, scope any search with `--domain`, and let the importer derive them from an existing store's topics. Curation is first-class: `rag review` reports near-duplicates, dangling links, and stale pins; refuting a fact archives it with a required reason + evidence; `rag purge` removes only already-refuted documents, and only as `rag_admin`.
### 4. Built-in maintenance, backup, and restore-testing
`rag backup` runs `pg_dump -Fc` locally (plus an optional copy to a synced directory you configure), with rotation. `rag maintenance` is a tiny, single-flight, always-exit-0 job that ticks the worker, rotates logs, and — **weekly** — runs a **report-only restore-test**: it restores your newest dump into an isolated scratch database, compares row counts, and drops it. A backup you've never restored isn't a backup; this one checks itself.
---
## How it works
```
Supported coding sessions rag save · migrate import · MCP write tools
(queued & mined on session end) (you, or an agent)
│ │
└───────────────────┬───────────────────────────┘
│
one audited write gateway
strips secret-shaped tokens · chunks + embeds (local Ollama) · resolves edges · logs
│
┌───────────────┴────────────────┐
PostgreSQL + pgvector typed knowledge graph
documents · chunks halfvec(1024) edges: references · extends · …
HNSW ANN + EN/DE full-text
│
one hybrid ranked search ── vector ⊕ full-text
│
session-start context · prompt-time recall · read-only MCP behind a privilege boundary
```
Claude Code and Codex continuity use a separate operational path. The Claude
flow:
```text
PreCompact ──► bounded deterministic snapshot ──► audited checkpoint
│ └──► priority enrichment job ──► provider CLI
└──► stdout: versioned compact instructions (+ checkpoint id)
Claude compacts (instructions appended)
│
PostCompact ──► mark boundary + store compact_summary as bounded handoff
│
SessionStart(source="compact") ──► checkpoint + handoff, ≤ 10,000 chars ──► next request
```
The Codex flow (`PreCompact` stays silent; `PostCompact` stores no handoff):
```text
PreCompact ──► bounded deterministic snapshot ──► audited checkpoint
│ └──► priority enrichment job ──► provider CLI
▼
Codex compacts
│
PostCompact ──► mark boundary only (cannot inject context)
│
SessionStart(source="compact") ──► bounded checkpoint context ──► next request
```
Codex checkpoint enrichment reads native rollout response items, including records
with null message IDs. It keeps user/assistant prose and safe tool-name hints;
reasoning, system/developer context, event mirrors and tool outputs are excluded.
The code-only repair and remaining live-verification limits are recorded in
[Codex continuity verification](docs/verification/codex-continuity.md).
Native Codex memories are complementary, not the canonical record. With the
installed policy they remain enabled and can be inspected with `/memories`;
agentic-rag is canonical for durable searchable knowledge, audit history, and
explicit continuation checkpoints.
- **Your chosen inference providers.** Every extraction call goes through `agentic_rag.llm`: existing Codex/Claude CLI login or an explicitly configured compatible HTTP endpoint. Transcript digests and checkpoint deltas are character-bounded; mining also sends secret-stripped copies of matching pins. Embeddings default to local Ollama. Optional [managed profiles](docs/provider-indexes.md) support separate local/HTTP vector spaces, protected credentials and explicit bounded corpus rebuilds; selecting an external profile also sends future search queries and semantic duplicate candidate title/body there. Default research skips structured assessment; retrieval still uses the selected embedding profile.
- **One audited write path.** Every change — a manual `save`, a mined memory, an import — funnels through a single gateway that strips secret-shaped tokens, regenerates chunks + embeddings in one transaction, resolves dangling edges, and writes an audit row. Embeddings fail *open* (queued for retry if Ollama is down); nothing else does.
- **Least privilege, by role.** Three login roles enforce a destruction-protection matrix: `rag_reader` (SELECT only, used by search and the read-only MCP), `rag_writer` (INSERT/UPDATE but **no DELETE/TRUNCATE/DROP**), and `rag_admin` (migrate, purge, restore).
- **It steps aside, not in front.** If Ollama is down, search degrades to full-text-only and returns a warning rather than failing; the maintenance job always exits 0.
The full story is in the **[handbook](docs/README.md)** — the mental model, everyday use, configuration, importing an existing wiki, and the architecture and design rationale.
---
## Comparison
agentic-rag sits between two worlds: the file-based **LLM-Wiki** family (human-readable Markdown with a lint/graph layer) and typical **RAG stacks** (hosted or API-driven retrieval you feed documents to). Every cell below is marked honestly — including the rows where each of them beats us.
Legend: ✅ shipped in code and operationally established · 🧪 shipped and installed, live verification pending · ⚠️ partial / caveated · ❌ absent
### vs LLM-Wiki systems (file-based knowledge wikis)
| Capability | **agentic-rag** | File-based LLM-Wiki |
|---|:--:|:--:|
| Hybrid vector + full-text ranked search (ANN at scale) | ✅ | ⚠️ lexical/graph, file-scan |
| Bilingual full-text (EN + DE) + semantic recall | ✅ | ⚠️ |
| Transactional, audited writes through one gateway | ✅ | ⚠️ |
| Scales to a large corpus (HNSW index) | ✅ | ⚠️ file-scan slows |
| Human-readable, git-diffable plain-text store | ⚠️ import/export MD; store is Postgres | ✅ |
| Zero-infrastructure (no DB/service to run) | ❌ needs Postgres + Ollama | ✅ |
| Imports an existing llm-wiki store | ✅ `rag migrate` | ✅ it *is* one |
**Bottom line:** if you want a git-tracked pile of Markdown, a file-wiki wins on its home turf. If you want fast hybrid recall over a growing corpus with transactional safety, agentic-rag wins — and it can **import your existing llm-wiki** to get you there.
### vs typical RAG systems (retrieval frameworks / hosted memory)
| Capability | **agentic-rag** | Typical RAG stack |
|---|:--:|:--:|
| Local-first store, provider CLI under your control, no hosted RAG service ¹ | ✅ | ⚠️ usually a hosted service |
| Auto-populates from Claude Code sessions (mining) | ✅ | ❌ you feed it |
| Codex session mining and continuity | 🧪 | ❌ you feed it |
| Claude Code compaction continuity (checkpoint + handoff) | 🧪 | ❌ |
| Self-curation (dedup, near-dup gate, refute/archive) | ✅ | ⚠️ |
| Typed knowledge graph (edges) alongside vector search | ✅ | ⚠️ |
| One audited write gateway with secret stripping | ✅ | ❌ |
| Read/write privilege boundary for subagents (RO MCP) | ✅ | ⚠️ |
| Turnkey managed hosting / large ecosystem ² | ⚠️ self-host, young | ✅ |
**Bottom line:** a hosted RAG stack wins on turnkey scale and ecosystem. agentic-rag wins on being local-first, using a provider CLI under your control with no hosted RAG service in the loop, self-populating-from-your-own-work, and self-curating — a memory that fills and tidies itself instead of one you have to keep feeding.
<sub>
¹ agentic-rag is <strong>provider-configurable</strong>: LLM-assisted mining, curation,
and bounded checkpoint enrichment use the configured local CLI — Codex with a
ChatGPT login, or Claude with OAuth/API-key authentication — or an opt-in
compatible HTTP extraction endpoint. Account limits and any metering
belong to that chosen provider. Mining can include secret-stripped copies of
all matching pin bodies without mutating stored pin text. <strong>Embeddings default
to local Ollama</strong>; external managed generations receive search queries,
semantic duplicate candidate title/body and explicit corpus rebuild batches.
Default research skips structured assessment; its retrieval uses the selected embedding profile;
explicit research provider mode sends bounded evidence for assessment. There is no
third-party RAG service between you and your data. Most hosted RAG stacks route
your documents through a paid service.<br>
² agentic-rag is <strong>newly public</strong> and self-hosted — the field's clearest edge over us is turnkey managed hosting and a large plugin/integration ecosystem.
</sub>
---
## Configuration
[Reranker settings](docs/reranker-settings.md) adds a server-wide dashboard preset backed by the protected auth policy file. mMARCO DE/EN is the default; the optional model daemon must be prepared and started explicitly. Existing stores require the verified audit-kind migration028.
Config lives in one TOML file at `~/.agentic-rag/config.toml`. Every key is optional — omit a section to keep its defaults.
| Setting | Default | What it does |
|---|---|---|
| `[db] name` | `agentic_rag` | Database name. |
| `[db] host` | `""` (local socket) | Empty = local unix socket; set it for a networked/remote server. |
| `[embed] model` | `bge-m3` | Ollama embedding model tag. |
| `[embed] dim` | `1024` | Fixed to the schema (`halfvec(1024)`); `init-db` refuses a mismatch. |
| `[ollama] url` | `http://localhost:11434` | Local Ollama endpoint. |
| `[backup] local_dir` | `~/.agentic-rag/backups` | Where `pg_dump` archives are written. |
| `[backup] cloud_dir` | — (unset) | **Opt-in** copy to a synced/cloud directory; unset means backups remain local. Provider-bound LLM inputs are disclosed above. |
| `[pg] bin_dir` | auto-resolved | Only needed if `pg_dump`/`pg_restore`/`psql` aren't on `PATH` (e.g. Postgres.app). |
Roles are created **passwordless** by default, relying on local `peer`/`trust` auth (Postgres and agentic-rag on the same machine). For a networked or shared instance, set role passwords with `ALTER ROLE …` and let libpq authenticate via `~/.pgpass` or `PGHOST`/`PGPORT`/`PGPASSWORD` — see the handbook's privacy chapter.
The Claude target separately manages `autoCompactWindow = 500000` in
`~/.claude/settings.json` — a 1M context compacting at 500K with a `[1m]`
model; `model` is reported, never rewritten, and long-context requests above
200K input tokens cost more on API billing. The Codex target separately
manages a 350000 context window and a 250000 total-token compaction threshold,
leaving a 100K reserve and a nominal 22K buffer below the higher-pricing
boundary, plus native memories and the compact prompt. Official GPT-5.6
capacity is 1.05M, but inputs above 272K are subject to higher provider pricing
and may add latency; see
[Configuration](docs/06-configuration-reference.md) and
[Privacy, cost & control](docs/07-privacy-and-cost.md).
---
## 📖 Documentation / Handbook
The full story lives in the **[agentic-rag Handbook](docs/README.md)** — a single, progressively-ordered read from the mental model through everyday use, configuration, importing, and the engine's architecture and design rationale. A few key chapters:
- **Understand** — [What is agentic-rag?](docs/01-what-is-agentic-rag.md) · [The mental model](docs/02-mental-model.md)
- **Use** — [Quick start](docs/03-quick-start.md) · [Working with your memory](docs/04-working-with-memory.md) · [Session mining & curation](docs/05-session-mining-and-curation.md)
- **Configure** — [Privacy, cost & control](docs/07-privacy-and-cost.md)
- **Develop** — [Architecture](docs/10-architecture.md) · [Reference — CLI & MCP](docs/11-reference-cli-and-mcp.md)
Start at the **[handbook index](docs/README.md)** for the one-line "what you'll learn" map of every chapter.
---
## Status
agentic-rag is **young but solid** — a real engine, openly developed. Repository
and rollout state are listed separately below:
- ✅ **Storage & search:** PostgreSQL + pgvector schema, HNSW ANN blended with EN/DE full-text into one ranked list, the typed edge graph, the three-role destruction-protection matrix.
- ✅ **The document write gateway:** secret stripping on document inputs and
generated document writes, one-transaction chunk + embed + edge-resolve +
audit, embeddings that fail open with a retry queue.
- ✅ **Session mining:** hooks → queue → single-writer worker → configured Codex/Claude CLI → gateway, with a near-duplicate gate and provider-outage circuit breaker.
- ✅ **Curation & safety:** `rag review`, refute-as-archive, and admin-only `rag purge` (removes only already-refuted documents, as `rag_admin`).
- ✅ **Maintenance & backups:** `pg_dump` backups with rotation, the tiny always-exit-0 maintenance job, and the **weekly report-only restore-test**. macOS auto-schedules via `launchd`; Linux uses the documented cron/systemd recipes.
- ✅ **Claude integration:** two user-scope MCP servers (read-write + a read-only server behind a privilege boundary for subagents), idempotent install that preserves foreign hooks.
- ✅ **Claude continuity in code:** six Claude hooks, the `PreCompact` stdout
compact prompt, the bounded `compact_summary` handoff, the 10,000-character
SessionStart cap, the managed 1M/500K policy, and `rag install
--check`/`--restore`.
- 🔒 **Claude continuity live rollout:** `rag install`, `/hooks` review,
`/autocompact`, manual/automatic compaction, and SessionEnd tail capture
on the maintainer machine remain open (backlog 0.3).
- ✅ **Codex continuity in code:** audited checkpoints, bounded capture and
restoration, asynchronous enrichment, all six lifecycle handlers, a
versioned compact prompt, recoverable installer/check mode, and checkpoint
health in `rag status`.
- 🔒 **Codex continuity live rollout:** the pre-install whole-diff/security
review and provider-bound pin hardening are complete. The live global
install, `/hooks` trust, manual/automatic compaction, provider-recovery, and
SessionEnd smoke tests remain open. See
[`FEATURES.md`](FEATURES.md) and blocker-first [`BACKLOG.md`](BACKLOG.md).
- ✅ **Features 1–10 adopted locally:** retrieval routing/caches, local reranking,
contextual indexing, filtered vector search, bounded research, thematic summaries,
scoped entity aliases, incremental ingestion and offline failure evaluation.
Schema 001–019 and fresh 10-tool reader / 18-tool main MCP clients pass the
[production acceptance](docs/verification/features9-10-production-adoption.md).
- ✅ **Quality:** a content-free repository with a comprehensive local test
suite; exact verification counts belong in rollout evidence, not a static
badge.
The clearest gap relative to the field is maturity: it's newly public and self-hosted, without the turnkey hosting or large ecosystem of established RAG stacks.
---
## Acknowledgments
agentic-rag builds on other people's ideas and tools:
- **Andrej Karpathy** — the [LLM-Wiki idea](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) that shaped the durable-knowledge model this import path speaks to.
- **The llm-wiki format** — topic-partitioned Markdown with an optional memory store; `rag migrate` imports it wholesale, so an existing wiki carries straight over.
- **[pgvector](https://github.com/pgvector/pgvector)** and **[Ollama](https://ollama.com)** (`bge-m3`) — the local vector search and embeddings underneath everything.
- **Anthropic** — Claude Code, its hooks, and the MCP integration agentic-rag plugs into.
---
Project and global applicability now share [one explicit scope policy](docs/project-scope.md):
`rag search "your question" --project /absolute/repository`.
## Contributing
Every future PR increments the patch version by exactly `0.0.1` and includes its
relevant documentation update. Minor and major bumps require an explicit manual
maintainer instruction and replace the normal patch increment for that PR. Keep package/lock metadata, this version badge and the
changelog consistent; preserve historical versions. See the mandatory
[version and documentation rule](docs/12-contributing.md#version-and-documentation-requirement-for-every-pr).
Tests come first (TDD), and `docs/` is kept in step with the code. A **warn-only doc-reminder hook** ships under `.githooks/`: if a commit touches `agentic_rag/` or `sql/` without touching `docs/`, it prints a reminder — it never blocks. Enable it once per clone:
```bash
git config core.hooksPath .githooks
```
Search now returns diverse sources and query-centered spans with chunk citations. See
[retrieval quality, optional graph expansion and limits](docs/retrieval-quality.md).
Measure retrieval and memory quality with the [synthetic EN/DE benchmark](docs/benchmarks/memory-quality.md): `rag benchmark run --output /tmp/rag-bench-baseline`.
Run the suite with `uv run pytest`. See the handbook's [Contributing](docs/12-contributing.md) chapter for dev setup, the test database, and code layout.
## License
[MIT](LICENSE). The repository is **code-only and content-free** — your documents, embeddings, and config stay in your own PostgreSQL database, on your own machine.
### Bounded project context
Source-backed advisory profiles and selective EN/DE project recall share one local
context service across hooks, CLI and MCP. Exact pins and checkpoint restoration
retain priority. See [usage and limits](docs/project-context.md).
### Confirmed-failure evaluation
Use `rag benchmark optimize --output /new/report` for bounded public offline routing/ranking comparison, `benchmark export-corrections` and `evaluate-corrections` for private confirmed user labels, and explicit `benchmark optimize-mining --mine-model` for public configured-provider prompt measurements. Sealed candidates do not change live policy. [Contracts and limits](docs/failure-evaluation.md), [PB-5.10](docs/playbooks/failure-evaluation.md).
Typed original selection: [Typed metadata filters](docs/metadata-filters.md) (0.6.30/schema039, opt-in registry; existing
knowledge is not automatically relabeled or re-embedded).
This server cannot be deployed
Maintenance
ActivityActive
ResponsivenessOutdated data