Corpus — shared docs + agentic RAG for two coding agents

One Postgres-backed knowledge base with three doors into it:

1. MCP server (:8765) — both coding agents connect here with their own API keys, upload docs, and search/ask over everything.

2. Web app (:8000) — a React SPA (TypeScript, Tailwind, shadcn-style components) over a JSON API: dashboard, document management, hybrid search with visible retrieval ranks, and a streaming cited chat.

3. Postgres + pgvecto.rs — the single source of truth. Agents and humans see the same corpus, filtered by RBAC.

Agent A ──┐                          ┌── chunks: vector(1536) + tsvector
Agent B ──┼── MCP :8765 ──┐          │
          │               ├── core ──┼── documents (owner, visibility, tags)
Humans ───┴── Web :8000 ──┘          └── users / api_keys / chat history
          (React SPA + SSE)

The RAG pipeline

Two agents, one entry point (core/rag/answer.py routes; both are deepagents graphs, so one toolset, one trace format, one streaming driver):

• fast (fast.py) — a single agent with the corpus tools and a low recursion cap (~5 searches max). No subagents, no todo list. Seconds.

• deep (deep_agent.py) — an orchestrator that plans with write_todos, keeps a notes.md scratchpad, and delegates to three sharply differentiated subagents: scout (fast orientation: what does the corpus even contain), researcher (deep dive on one question, several run in parallel), and verifier (adversarial fact-check of the draft: SUPPORTED / UNSUPPORTED / CONTRADICTED / OVERSTATED, PASS or REVISE). Tens of seconds, ~10–40× the cost.

auto picks with a deliberately conservative heuristic (comparative vocabulary, multiple questions, long briefs); users override with Auto / Fast / Deep pills, agents with mode= or the deep_research MCP tool. If a deep run fails it falls back to the fast path rather than erroring. Design notes: docs/DEEP_AGENT_RAG.md.

Role-scoped agent tools. The toolset each agent run receives mirrors the caller's web role (core/rag/agent_tools.py::TOOL_ROLES): viewers get the read-only set (search_corpus, read_document, list_corpus_documents, think); editors add save_note (persist a research synthesis back into the corpus, tagged agent-note); admins add documentation_gaps (what people asked that got no answer). Capabilities are also enforced server-side — the table is UX, not the security boundary. whoami over MCP reports the tools a key's role grants.

Streaming. POST /api/ask is an SSE stream: mode → (step | token)* → done. Tool calls stream as pipeline steps (search: "retry policy", delegate → researcher: …), tokens stream as they're generated, and the done event carries the authoritative answer + resolved citations — so token streaming is progressive enhancement, never the source of truth. The MCP server uses the synchronous ask() on the same engines.

Relevance floor. A bare ORDER BY distance LIMIT k always returns k rows, so uncovered questions would yield confident-looking noise. Dense hits beyond RETRIEVAL_MIN_SIMILARITY are dropped, which makes "we found nothing" possible — and that is what feeds the dashboard's documentation gaps panel (questions the corpus couldn't answer, aggregated by frequency and people). Five people asking about deploy rollback and finding nothing isn't a search problem, it's a missing document. The default floor is provider-aware: the offline dev embedding needs a much higher floor than a real model.

Related MCP server: astra-knowledge-base-mcp

The web app (webapp/ui)

React 18 + TypeScript + Vite + Tailwind v4, shadcn-style primitives over Radix, TanStack Query for data, Recharts for the dashboard, dark/light theme. Django 5 (ASGI) serves the built SPA and owns everything under /api; auth is a signed-cookie session, so the browser never holds a token. Data access is the Django ORM over managed = False models (core/models.py) — schema.sql stays the DDL source of truth because pgvecto.rs columns and generated tsvector columns don't round-trip through Django migrations. Raw SQL survives only where the ORM can't speak: the hybrid-retrieval hot path (<=>, RRF) and two aggregates. All configuration is one typed pydantic-settings model (core/config.py) that both Django and the MCP server consume.

• Dashboard — corpus stats, 30-day charts, composition by tag, per-agent activity, an activity feed, and the documentation-gaps panel.

• Chat — SSE streaming with live pipeline steps, mode pills, a collapsible trace panel, and citation chips with hover previews that deep-link to the exact chunk.

• Search — hybrid results with visible vec # / kw # / rrf ranks, so retrieval quality is debuggable at a glance.

• Documents — upload (file or paste), tag, filter, edit (content edits re-chunk + re-embed), delete.

• Doc viewer — rendered markdown with a sticky TOC, related docs (centroid similarity) + backlinks, "Ask about this doc", and a "chunk seams" toggle that overlays exactly how the doc was cut for retrieval.

• People & access (admin) — team table with role changes, password resets, enable/disable/unlock, dormant-account flags (60 days), agent-key minting and revocation, and the audit log (who did what to whom, with before/after values on role changes).

RBAC & user abstraction

Everything resolves to a Principal (user + role + connection path):

• Web session → Principal via email + password (Argon2id, stored in Postgres). Login is throttled: 8 failed attempts locks the account for 15 minutes, and failures are deliberately indistinguishable so the form can't be used to discover which emails are registered.

• MCP API key → Principal via the key's bound user. Agents are users.

• OAuth 2.1 access token → Principal via the user who approved the client. For MCP clients that only speak OAuth and have nowhere to paste a key: the web app is the authorization server, so the client is sent to the ordinary Corpus login plus a consent screen, and the token it receives is bound to whoever said yes. There is no service account and no default admin — an OAuth client is exactly as privileged as the human behind it. PKCE is mandatory, clients enrol themselves via Dynamic Client Registration (RFC 7591), tokens are opaque and stored only as sha256, and a user can disconnect a client from their account page. See core/oauth.py.

Roles: viewer (search/chat) < editor (+ upload, delete own) < admin (+ manage people, roles, keys, delete anything). Documents are org (visible to all) or private (owner only) — enforced in every SQL query, so MCP tools and web API can't leak across the boundary.

Vector engine: pgvecto.rs

The schema uses pgvecto.rs (CREATE EXTENSION vectors) with an HNSW cosine index. Two things to know:

• The extension lives in the vectors schema; the schema and the connection pool both put it on the search_path so unqualified vector, <=>, and avg(embedding) resolve.

• The app sends embeddings as text ('[1,2,3]'::vector) and never reads vector values back, so there is no client-side type adapter and no reliance on engine-specific implicit casts.

Run it from tensorchord/pgvecto-rs:pg16-v0.3.0 (both compose files do). Migrating an existing pgvector/VectorChord install: migrations/008_pgvecto_rs.sql (rewrites the column type — take a backup).

Quick start

Zero-config (offline, no accounts needed):

bash run_local.sh

Writes a dev .env, builds the UI once (needs node), starts Postgres + web (:8000) + MCP (:8765). Open the site and create the first account — it becomes the admin. Embeddings run on an offline hashing stand-in, so retrieval works but has no semantics; chat returns top passages until you add a GOOGLE_API_KEY. If a local postgres owns 5432: SDR_DB_PORT=5434 bash run_local.sh. Optionally seed sample docs:

.venv/bin/python scripts/dev_seed.py

Full local setup (real retrieval + chat):

cp .env.example .env      # fill in keys (see below)
docker compose up -d      # Postgres + pgvecto.rs, schema auto-applied

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
(cd webapp/ui && npm ci && npm run build)

uvicorn server.asgi:application --reload --port 8000   # terminal 1
python -m mcp_server.server                    # terminal 2

UI development with hot reload: cd webapp/ui && npm run dev (Vite on :5173, proxying /api to :8000).

Required in .env:

• GOOGLE_API_KEY — Gemini embeddings (gemini-embedding-001) and chat answers (gemini-2.5-flash), the defaults. Swap providers per layer: EMBEDDINGS_PROVIDER=openai|voyage|dev, ANSWER_PROVIDER=anthropic (then ANTHROPIC_API_KEY).

Then: create the first account (admin) → People → mint keys for agent-a and agent-b → wire them into each agent per docs/mcp-client-setup.md.

Deploy to production

bash deploy.sh    # on any Ubuntu VM: installs docker, builds, starts, health-checks

Everything ships from one Dockerfile: stage 1 builds the React UI, stage 2 is the Python image that runs as both the web app (default command) and the MCP server (compose overrides the command). Full stack in containers — Caddy (auto-TLS on your domain), web, MCP, Postgres (internal-only, named volume). https://DOMAIN/ is the web app, https://DOMAIN/mcp is what the agents connect to. See docs/DEPLOYMENT.md.

MCP tools

upload_document · search_docs · ask_docs (with mode=) · deep_research · list_documents · read_document · delete_document · whoami

Changing the embedding model

Keep three things in sync: EMBEDDINGS_MODEL, EMBEDDINGS_DIM, and the vector(1536) column in schema.sql. Changing dimension requires re-ingesting (drop the chunks table or ALTER COLUMN ... TYPE vector(N) then re-upload).

Layout

core/              config, db pool, security (Principal/RBAC/API keys),
                   oauth (OAuth 2.1 authorization server for MCP clients),
                   citations (cite_key → deep link), analytics
core/rag/          chunking · embeddings · ingest · retrieve (hybrid+RRF)
                   fast (lean agent) · deep_agent (orchestrator+subagents)
                   agent_tools (per-principal toolset) · answer (routing+SSE)
mcp_server/        FastMCP HTTP server (the agents' door)
server/            Django (ASGI): settings from pydantic, JSON API + SSE,
                   serves the built SPA
core/models.py     ORM models (managed=False — schema.sql owns the DDL)
webapp/ui/         React SPA (Vite + TS + Tailwind + shadcn-style)
schema.sql         applied automatically by docker compose on first boot
migrations/        for existing installs (008 = move to pgvecto.rs)
docs/              deployment, MCP client setup, deep-agent design notes