# kb-rag — a local, offline knowledge base for papers and source code

Keep a folder of PDFs, notes and source repositories on your own machine, index
it once, and let a local language model answer questions from it with
citations. Nothing leaves the computer: the model runs in LM Studio (or any
OpenAI-compatible server), the index is one sqlite file, and the two are joined
by a small MCP server that offers a `kb_search` tool. Works on an Apple Silicon
Mac and on Linux; the network is needed only during installation.

Three clients can use it out of the box: LM Studio's own chat window, VS Code
or VSCodium through the Continue extension, and the opencode terminal agent.

## What you get

Large language models are good at explaining things and bad at remembering
yours. A knowledge base fixes that cheaply. You keep the material in ordinary
folders, a script turns it into searchable chunks, and when you ask a question
the model first calls `kb_search`, reads the handful of passages that come
back, and answers from them. For source code the passages carry file paths and
line ranges, so "see `src/main.c` lines 120–180" is a real citation.

The primary goal is not a polished product. Rather, we hope it is small enough
to read in full — four short Python files — so that when retrieval looks wrong
you can see why and tune it.

## How it works

| File | What it does |
|---|---|
| `ingest.py` | Walks `corpus/<category>/`, extracts text (pypdf for PDFs), cuts it into chunks, embeds each chunk with a local model and stores vectors plus a full-text index in `index/kb.db`. Incremental: unchanged files are skipped by sha256. |
| `retrieve.py` | Hybrid search: the 50 nearest vectors plus the 50 best BM25 matches, fused by reciprocal rank fusion, reranked by a cross-encoder, capped at three chunks per file so one long manual cannot crowd out a short paper. |
| `kb.py` | The command line: `kb add`, `kb add-repo`, `kb search`, `kb reindex` and friends. `kb` is a one-line wrapper that points it at this checkout. |
| `kb_mcp_server.py` | Serves `kb_search` over MCP on `127.0.0.1:8765`, so the clients share one warm copy of the models. |

Two details are worth knowing. First, the two halves of the search matter for
different reasons: vectors find concepts phrased differently from the text,
and BM25 finds exact identifiers — a function name, a lemma, a flag — that
embeddings fumble. Source code needs the second half badly. Second, documents
and code are chunked differently. Prose is cut into windows of about 450 tokens
with a 60-token overlap; source files are cut on line boundaries, and every
chunk begins with its path and line range.

A source repository is not copied into the library. `kb add-repo` links it in
place and the indexer follows the link, skipping `.git`, `node_modules`, build
directories, lock files, binaries and anything over 2 MB. After you pull new
commits, `kb reindex --prune` re-embeds only the files whose contents changed.

The retrieval models are `BAAI/bge-base-en-v1.5` (embeddings, 768-d, ~210 MB
quantised) and `BAAI/bge-reranker-base` (~1.1 GB), run on the CPU through
[fastembed](https://github.com/qdrant/fastembed). Together they take about
1.5 GB of memory while the server is up.

## Quick start

```bash
git clone https://github.com/potanin/kb-rag ~/kb
cd ~/kb
./install.sh                 # needs uv; makes .venv, installs pinned packages, puts `kb` in ~/bin
kb warm                      # downloads the retrieval models once (~1.3 GB) — the last step that needs the network

kb add ~/papers/*.pdf        # copy PDFs into the library (category "papers")
kb add-repo ~/src/project    # link a source tree, indexed in place (category "code")
kb search "where is the capability derivation tree updated" --k 5
```

Then start the search server as a service and connect a client:

- macOS: [docs/macos.md](docs/macos.md) — Homebrew packages, LM Studio, launchd.
- Linux: [docs/linux.md](docs/linux.md) — packages, LM Studio or llama.cpp, systemd.
- Clients: [docs/clients.md](docs/clients.md) — LM Studio chat, Continue, opencode.

`install.sh` is safe to re-run. It insists on uv's own Python build because
the Python that Apple ships (and the python.org installer) cannot load sqlite
extensions, which the vector search needs; it prints a one-line check so you
know the venv is good before anything else happens.

## Adding content

A category is just a subfolder of `corpus/`; `papers`, `notes` and `code` are
conventions, not rules, and the search tool can filter by category.

```bash
kb add klein2009.pdf                                 # into papers/ (the default)
kb add ~/Documents/reading --category papers         # a whole folder
kb add ~/notes/*.md --category notes
pandoc thesis.docx -o thesis.md && kb add thesis.md --category notes
ocrmypdf scanned.pdf scanned-ocr.pdf && kb add scanned-ocr.pdf   # scans have no text layer otherwise
```

Source trees are linked rather than copied:

```bash
kb add-repo ~/src/seL4 --name sel4            # corpus/code/sel4 -> ~/src/seL4
kb add-repo ~/src/big/lib --name biglib       # a subtree is fine if the whole tree is too much
kb categories                                 # shows which entries are linked repos
```

The first index of a large repository takes a while — the embedding model runs
on the CPU — so we suggest starting with one project. As a rough guide, a
repository of a few thousand source files produces tens of thousands of chunks,
and we expect that to take somewhere between a few minutes and an hour
depending on the machine. Later runs are incremental.

Which files are indexed is decided by the lists at the top of `ingest.py`:
documents (`.pdf .md .txt .rst .tex .html ...`), source and configuration
files (a long list of extensions plus `Makefile`, `CMakeLists.txt`,
`Dockerfile` and so on). They are meant to be edited.

## Searching from the command line

```bash
kb status
kb search "frobnicate_widgets" --category code --k 3
kb search "ownership types and generics" --category papers --no-rerank
```

Every hit shows the file, the chunk, which half of the search found it
(`dense`, `bm25` or `both`) and the reranker score. Test here before involving
a model: if the right passage does not come back from `kb search`, no prompt
will fix it.

## The search server

One long-running server serves all clients, so the models load once and stay
warm. The platform documents show how to run it as a launchd agent or a
systemd user service; by hand it is:

```bash
kb serve                                          # foreground, Ctrl-C to stop
curl -s http://127.0.0.1:8765/health              # {"ok":true,"service":"kb"}
~/kb/.venv/bin/python ~/kb/check-mcp.py "frobnicate_widgets"   # lists the tool, runs one search
```

The server binds to loopback only, has no authentication, and should stay that
way. It speaks streamable-http at `/mcp`, the current MCP transport; if a
client turns out to want the older SSE transport, run it with
`KB_MCP_TRANSPORT=sse` and use `/sse` in that client's URL. It reads the index
on every query, so adding documents needs no restart; only edits to the Python
files do.

There is also a plain-HTTP `POST /grounding` route for front ends that cannot
call tools: send `{"query": "..."}` and get back a context block to prepend to
the prompt, plus the passages for citations.

## Day to day

```bash
kb add paper.pdf                      # copy in and index
kb add-repo ~/src/thing               # link and index in place
kb reindex --prune                    # after git pull, or after deleting files by hand
kb list | kb categories | kb size | kb status
kb remove paper.pdf                   # drops the copy and its chunks
kb remove sel4                        # unlinks the repo and drops its chunks; the repo itself is untouched
rm index/kb.db && kb reindex          # rebuild from scratch
```

Everything lives in this checkout plus two caches under `~/.cache`, so a
backup is a copy of `~/kb` (linked repositories are not inside it).

## Tuning

Most knobs are environment variables read by the scripts; set them in
`~/bin/kb` and in the service definition so the command line and the server
agree.

| Variable | Default | Effect |
|---|---|---|
| `KB_CHUNK_TOKENS` / `KB_CHUNK_OVERLAP` | 450 / 60 | Chunk size. Smaller chunks give sharper hits on code; larger ones keep more of an argument together in a paper. Change, then rebuild. |
| `KB_MAX_PER_SOURCE` | 3 | Chunks per file in one result set. Raise it when you ask about a single document. |
| `KB_RERANK` | 1 | 0 skips the cross-encoder (saves the 1.1 GB download and some memory; a little less precise on vague questions). |
| `KB_SKIP_DIRS` | `.git,node_modules,build,...` | Directory names to skip inside repositories; replaces the default list. |
| `KB_MAX_FILE_BYTES` | 2000000 | Files larger than this are skipped. |
| `KB_EMBED_MODEL` | `BAAI/bge-base-en-v1.5` | Any 768-dimensional fastembed model; for other sizes change `EMBED_DIM` in `ingest.py` and `retrieve.py`, then rebuild. |
| `KB_MCP_PORT` | 8765 | Change it in the service and in every client config together. |

## Limits

It does not OCR scans or read `.docx` (pandoc and ocrmypdf cover those),
pypdf is mediocre on two-column papers (text order can interleave columns),
and the quality of the final answer is bounded by the chat model you can fit
in memory. We suspect the single best upgrade for a paper-heavy library is a
better PDF extractor in `extract_text()`; for a code-heavy one it is a smaller
chunk size.

## Troubleshooting

- **`AttributeError: enable_load_extension`** — the venv was built on a Python
  without sqlite extension loading. `rm -rf .venv && ./install.sh`.
- **`ModuleNotFoundError: No module named 'mcp.server.fastmcp'`** — mcp 2.x got
  installed; `uv pip install --python .venv/bin/python "mcp>=1.27,<2"`.
- **Models download again** — a cache path fell back to the temporary folder,
  which macOS and many Linux distributions clean out. Check
  `FASTEMBED_CACHE_PATH` and `TIKTOKEN_CACHE_DIR` in `~/bin/kb` and in the
  service; `kb size` shows where it is looking.
- **The client shows no `kb` tool** — the loaded model lacks tool support, the
  tool is switched off for that chat, or the server is down.
  `curl http://127.0.0.1:8765/health` separates the last case.
- **A client refuses the `/mcp` URL** — switch the server to SSE and use `/sse`.
- **Port 8765 in use** — `lsof -i :8765`; change `KB_MCP_PORT` everywhere.
- **A repository indexes far more than expected** — generated or vendored trees
  under names not in the skip list. Add them with `KB_SKIP_DIRS`, or link a
  subtree instead of the root.
- **A file is in the repo but never found** — it is binary, over 2 MB, a lock
  file, or has an extension the indexer does not know; see the lists at the
  top of `ingest.py`.

## Repository layout

| Path | Purpose |
|---|---|
| `install.sh` | venv, packages, folders, `~/bin/kb` |
| `kb`, `kb.py` | wrapper and command line |
| `ingest.py`, `retrieve.py` | indexing and hybrid search |
| `kb_mcp_server.py`, `check-mcp.py` | the MCP server and a client-side check |
| `kb-rules.md` | system-prompt text that makes a model call `kb_search` first |
| `launchd/`, `systemd/` | service definitions for macOS and Linux |
| `config/` | snippets for LM Studio, Continue and opencode |
| `docs/` | platform and client guides |
| `requirements.txt` | pinned package versions |

## Licence

MIT; see [LICENSE](LICENSE).