# Corpus — shared docs, answered by agents

One Postgres-backed knowledge base with four doors into it: a **web app**
(streaming cited chat, hybrid search, analytics), an **MCP server** for coding
agents (API key or OAuth 2.1 sign-in), a **voice agent** on a phone number,
and **Postgres + pgvector** underneath — same corpus, same RBAC, every door.

**▶ [Watch the demo (1:48)](docs/media/corpus-demo.mp4)** — login → search →
streaming chat with citations → chunk seams → MCP + OAuth consent → live
voice-agent call → admin.

![Dashboard](docs/media/dashboard.png)

```
Agent A ──┐                          ┌── chunks: vector(1536) + tsvector
Agent B ──┼── MCP :8765 ──┐          │
Humans ───┼── Web :8000 ──┼── core ──┼── documents (owner, visibility, tags)
Callers ──┘  (SPA + SSE)  │          └── users / oauth / chat history
              voice tool ─┘
```

## The RAG pipeline

| Stage | What | Why |
|---|---|---|
| Chunking | Markdown header-aware with section paths; code/JSON/YAML at line boundaries; sentence windows elsewhere. Per-document strategy switchable (chonkie: recursive/token/sentence/semantic) | Chunks are coherent units, not 512-token slices |
| Contextual embeddings | Embeds `"{title} > {section}\n\n{chunk}"`, stores the raw chunk | Cheap contextual retrieval; real recall lift |
| Hybrid retrieval | pgvector HNSW cosine **+** Postgres full-text, fused with RRF (k=60); a similarity floor makes "found nothing" possible | Keyword leg catches exact identifiers; the floor feeds the **documentation-gaps** panel (unanswered questions = your writing backlog) |
| Answering | Everything is a `deepagents` agent. **fast**: one agent, seconds. **deep**: planner + parallel researcher subagents + adversarial verifier, ~10–40× cost. `auto` routes; deep failures fall back to fast | An agent that sees an empty result can rephrase; a fixed pipeline can't |
| Citations | Every chunk has a stable 8-char `cite_key` derived from (document, position) — re-chunking preserves it. UI renders numbered chips deep-linking to `/docs/{id}#chunk-{ord}` | Citations in old chat history still resolve after re-indexing |

Chat streams over SSE (`mode` → `step`/`token`* → `done`); agent tool-calls
appear as live pipeline steps and a collapsible trace. Design notes:
[docs/DEEP_AGENT_RAG.md](docs/DEEP_AGENT_RAG.md).

## Access model

Everything resolves to a `Principal` (user + role + connection path):

- **Web session** — Argon2id passwords, throttled logins, signed-cookie
  sessions.
- **MCP API key** (`sdr_…`) — bound to a user; agents *are* users.
- **OAuth 2.1** — for MCP clients with nowhere to paste a key. The web app is
  the authorization server: PKCE mandatory, Dynamic Client Registration
  (RFC 7591), single-use 60s codes (replay revokes every derived token),
  rotating refresh tokens, all tokens stored as sha256. The token is bound to
  whoever approved the consent screen — no service account, no default admin.
  Users disconnect clients from their account page. See `core/oauth.py`.
- **Voice** — Sarvam Samvaad calls `POST /api/agent/ask`; answers come back
  TTS-friendly with citation markup stripped.

Roles `viewer < editor < admin`; documents are `org` or `private` — enforced
in every SQL query, so no door can leak across the boundary.

## Web app

React 18 + TS + Vite + Tailwind v4 (shadcn-style primitives, Inter/JetBrains
Mono, dark + light). Django 5 (ASGI) serves the built SPA and the JSON API;
the ORM runs over `managed = False` models — `schema.sql` owns the DDL.

Dashboard (stats, gaps, charts) · Chat (streaming, mode pills, trace,
citation chips) · Search (visible `vec/kw/rrf` ranks, per-doc re-chunking,
admin-frozen retrieval config shared by chat + MCP + ingest) · Documents ·
Doc viewer (TOC, related docs, backlinks, chunk-seams overlay) · Admin
(roles, keys, audit log) · Connect + voice pages.

## Quick start

```bash
bash run_local.sh          # zero-config: offline embeddings, no keys needed
```

Full setup: `cp .env.example .env` (set `GOOGLE_API_KEY`), then

```bash
docker compose up -d                                   # Postgres + pgvector
pip install -r requirements.txt
(cd webapp/ui && npm ci && npm run build)
uvicorn server.asgi:application --port 8000            # web
python -m mcp_server.server                            # MCP
```

First account = admin. Mint agent keys in People, wire them per
[docs/mcp-client-setup.md](docs/mcp-client-setup.md). Providers are
swappable per layer (`EMBEDDINGS_PROVIDER`, `ANSWER_PROVIDER`); keep
`EMBEDDINGS_DIM` in sync with `vector(1536)` in schema.sql if you change
models.

## Deploy

- **Free, no server**: `render.yaml` click-deploys web + MCP to Render;
  database on Neon (pgvector is why this works on managed free tiers).
  [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) Option A, including corpus
  migration via `scripts/migrate_corpus.py`.
- **One VM**: `bash deploy.sh` — Caddy (auto-TLS) → web + MCP + Postgres in
  containers, one Dockerfile for both services.

## MCP tools

`search_docs` · `ask_docs` · `deep_research` · `upload_document` ·
`list_documents` · `read_document` · `delete_document` · `whoami` — the tool
list an agent sees is filtered by its role, and enforced again on call.

## Layout

```
core/         config · security (Principal/RBAC/keys) · oauth (OAuth 2.1 AS)
              citations · analytics
core/rag/     chunking · embeddings · ingest · retrieve (hybrid+RRF)
              fast · deep_agent · agent_tools · answer (routing + SSE)
mcp_server/   FastMCP HTTP server (the agents' door)
server/       Django ASGI: JSON API + SSE, serves the SPA
webapp/ui/    React SPA
schema.sql    DDL source of truth (auto-applied on first boot)
migrations/   for existing installs (013 = move to pgvector)
docs/         deployment · MCP client setup · deep-agent design · media
```