docs-rag

Local, dependency-light retrieval over a repository's current documentation, exposed to coding agents as an MCP server.

The design premise is simple: an agent that greps a repository finds whatever is on disk — including archived plans and superseded designs. A docs-RAG corpus instead indexes a declared scope (which files, which trees), and re-indexes incrementally so that removing or archiving a document removes it from retrieval too. The index is a single SQLite file per corpus; there is no vector database, no framework and no service dependency beyond an embeddings endpoint.

corpus/            markdown (mirrored or git-cloned — see "Corpus config")
data-<corpus>/     index.sqlite  (+ .index.lock)

What it does

• Declared corpus scope — a corpus is data (corpora.json): top files, docs-root files, recursive trees, and directory names to exclude. Nothing tenant-specific lives in code.

• Incremental indexing — sha256 per file; only changed files are re-chunked and re-embedded, deleted files are purged from the index.

• Section-aware chunking — markdown headings start sections; paragraphs are atomic; sections merge up to a character budget and oversized ones are split on line boundaries with a sliding overlap.

• Hybrid retrieval — vector KNN (sqlite-vec, with a numpy brute-force fallback) is fused with FTS5 trigram full-text search via reciprocal rank fusion (k=60). RRF needs no score calibration between the two rankers.

• MCP tools — retrieve(query, k) and corpus_status() over streamable HTTP, so any MCP-capable agent can query the corpus; ragquery.py does the same from a shell.

• Query metering — every tool call appends one line to logs/queries-<corpus>.log (best-effort; never blocks retrieval).

Related MCP server: docs_search

Quickstart (no model server required)

python3 -m venv .venv && .venv/bin/pip install -r requirements.txt -r requirements-dev.txt
bash demo.sh          # builds a synthetic corpus in a temp dir, indexes it, queries it

Intel (x86_64) macOS: the latest cryptography wheel release has no x86_64 build; pin it explicitly — pip install 'cryptography==48.0.1' -r requirements.txt -r requirements-dev.txt.

demo.sh sets RAG_EMBED_BACKEND=stub, a deterministic hashed bag-of-tokens embedding, so the whole pipeline runs offline. It writes nothing outside a mktemp directory.

For real use, point the service at an OpenAI-compatible embeddings endpoint (LM Studio by default) and create a corpus config:

cp corpora.example.json corpora.json          # edit scope/paths; keep it untracked
export RAG_CORPUS=example                     # mandatory: the service refuses to guess
.venv/bin/python indexer.py                   # build/refresh the index
.venv/bin/python ragquery.py -k 4 <<< "how does retrieval fuse the two rankers?"
.venv/bin/python serve.py                     # MCP server (RAG_PORT, default 8765)

Environment: RAG_DIR (default ~/rag-service; override per deployment) positions corpora.json, .env, corpora and indexes; RAG_CORPUS selects the corpus; RAG_CORPUS_DIR, RAG_HOST, RAG_PORT, RAG_SERVER_NAME, RAG_SYSTEM_NOTE, RAG_MCP_TRANSPORT and RAG_EMBED_BACKEND override defaults. .env may hold LMSTUDIO_API_KEY / LMSTUDIO_BASE_URL / EMBED_MODEL.

Corpus config

corpora.example.json documents the schema; the keys that matter:

Only *.md is indexed. A missing entry is skipped rather than failing, so a typo shrinks the corpus silently — check corpus_status() after editing the config.

Refresh and deployment

refresh.sh <corpus> [git-ref] re-fetches a git-backed corpus (when git_origin_repo is set) and re-indexes; for a mirror-style corpus the content is pushed by the owning repository, so nothing is fetched here. Run it per corpus from cron, e.g.

0 * * * * /path/to/docs-rag/refresh.sh example >/dev/null 2>&1

deploy/com.example.docs-rag.plist is a launchd template for one corpus instance; replace the placeholders and bootstrap it per corpus (RAG_CORPUS and RAG_PORT differ per agent).

Clients

Two shell clients ship with the service so a consuming repository does not have to re-implement the protocol:

Both read the corpus scope from the service's config at run time, so they keep no second copy of it — a scope change on the service side cannot silently drift from what is pushed. sync_corpus.sh also guards its target: it refuses any --remote-dir that is not exactly <service-dir>/corpus/<corpus>, because the transfer uses --delete and a typo must not be able to overwrite another corpus. --dry-run prints the payload and changes nothing (it still reads the scope from the service, so the host must be reachable). If another indexer holds the lock — e.g. the service-side cron — sync_corpus.sh retries for up to a minute and then exits non-zero instead of reporting success: the corpus is synced, the index catches up on the next run.

./client/sync_corpus.sh --corpus example --source ~/src/my-repo \
    --host rag-host --service-dir /srv/docs-rag
./client/query.sh --corpus example --query "how does retrieval fuse the two rankers?" \
    --host rag-host --service-dir /srv/docs-rag

Evaluation

eval/ measures retrieval quality against a labelled query set: the real indexer builds the index, the real retrieve path answers, and the harness reports recall@k, MRR and latency. The published run uses rust-lang/book (112 markdown files, commit pinned in the report) with text-embedding-nomic-embed-text-v1.5:

Cross-lingual retrieval (Chinese question, English corpus) is the clear weak spot. The per-query table, the misses and the error analysis live in eval/report-book.md and eval/README.md; reproduce with:

./eval/fetch_corpus.sh
.venv/bin/python eval/run_eval.py --rag-dir eval/.work --corpus book \
    --golden eval/golden/book.jsonl --k 10 --backend lmstudio --fresh --report eval/report-book.md

CI runs the harness offline on a fixture corpus (stub embeddings) with a --min-recall gate, so a metric computation or label-loading regression fails the build without a model server.

What it does not do

• No reranking, no query rewriting, no LLM in the retrieval path — ranking is deterministic.

• Markdown only: no PDF, HTML, code-symbol or image ingestion.

• The MCP endpoint has no authentication: expose it only on a trusted network or overlay.

• Single-host, single-writer: the indexer takes a lock; concurrent indexing of one corpus is skipped rather than queued.

• No deletion of corpora, no index migration between schema versions (delete index.sqlite and re-index when the schema changes).

Development

.venv/bin/ruff check .            # lint
.venv/bin/mypy                    # type check (config in pyproject.toml)
.venv/bin/python -m pytest        # tests (unit + end-to-end, no network)
.venv/bin/shellcheck client/*.sh  # shell check for the clients
.venv/bin/pre-commit run --all-files  # secret scan (baseline: .secrets.baseline)

CI runs the same gates, plus shellcheck on the client scripts and the detect-secrets scan (.github/workflows/ci.yml); install the commit hook once with pre-commit install. Licensed under MIT.