# codesearch

Semantic search over local source repositories and forum/mailing-list archives,
exposed both as a shell CLI and as an MCP server so any LLM (Claude Code, a
local model, whatever speaks MCP) can look things up while helping you debug.

This is the retrieval half of warpembot, extracted and made project-agnostic.
No email, no GitHub, no auto-replying — just indexing and querying.

## The `csbot` command

One entry point for everything:

```bash
./csbot init warp https://github.com/warpem/warp.git --kb archives/warp-group.mbox
```

Registers a codebase (cloning it if you give a git URL, or just pointing at a
local checkout), ingests any accumulated discussion archive you have for it, and
builds the embeddings. `--kb` is optional — with no archive you get a
code-only knowledge base. Missing archives are reported and skipped, not fatal.

```bash
./csbot update --pull
```

`git pull`s each registered repo and re-embeds only the files whose checksums
changed. Drop `--pull` to index local edits; name repos to do a subset.

```bash
./csbot ask "why does frame series alignment deselect items" --repo warp
```

Retrieves the relevant source and prior discussions, sends them to a local
Ollama chat model, and prints the answer followed by the file:line citations it
drew on. Useful flags: `--no-forum` (code only), `-k N` (more context),
`--model` (override the chat model), and `--context-only`, which prints the
assembled context instead of generating — paste it into Claude or any other
model when you want a stronger answer than the local one.

```bash
./csbot status
./csbot kb warp-group archives/more-threads.mbox
```

`status` shows chunk/document counts per source and flags repos whose path has
gone missing. `kb` adds discussion knowledge to an existing project later.

Two models are involved and they are configured separately in `config.json`:
`embedding_model` (used for indexing and search — changing it invalidates every
index) and `chat_model` (used only by `csbot ask` to write the answer).

## Components

The `csbot` wrapper is thin; the pieces underneath stay usable on their own.

| File | Purpose |
|------|---------|
| `csbot` | Single CLI entry point: `init`, `update`, `ask`, `kb`, `status` |
| `cs_common.py` | Config, Ollama embeddings, on-disk vector index, chunk metadata |
| `cs_index.py` | Build/update indexes: repos (incremental), mbox archives, text/JSON dumps |
| `cs_server.py` | MCP stdio server: `list_sources`, `search_code`, `read_code`, `search_forum` |
| `cs_query.py` | Shell CLI: `code`, `forum`, `context` |
| `config.json` | Which repos and corpora exist, embedding model, chunking params |

Everything is local: embeddings come from Ollama, indexes are `.npz` files of
L2-normalized vectors, search is a dot product. No external services.

## Setup

```bash
pip install -r requirements.txt
```

```bash
ollama pull embeddinggemma:300m
```

`embeddinggemma:300m` is the default embedding model: ~600 MB and 768-dim,
which keeps indexing and query latency sane on Apple silicon.

Vectors from different embedding models are not comparable, so changing
`embedding_model` invalidates every index. `index/manifest.json` records which
model built the indexes: `csbot update` refuses to write into a mismatched index,
and the query paths warn instead of returning nonsense. To switch models,
delete `index/` and rebuild.

Each family also wants its own query/document prefixes (Qwen's `Instruct:`
block, Gemma's `task: search result | query:`, Nomic's `search_query:`). These
live in `EMBED_PROFILES` in `cs_common.py` and are selected automatically from
the model name — add an entry there if you use something else.

Because embeddinggemma's context window is 2048 tokens, `chunk_size` defaults
to 100 lines with 25 lines of overlap. If you move to a long-context embedder
such as `nomic-embed-text` (8192), raising `chunk_size` back to 200 is safe.

Then use `csbot init` to register codebases, or edit `config.json` by hand:

- `repos` — name → path of each checkout you want searchable (relative paths
  resolve against this directory; `~` works).
- `corpora` — name → directory holding forum/mailing-list documents as JSON.

## Direct access to the layers

`csbot` covers the normal workflow; these are the same operations unwrapped, for
scripting or partial re-indexing.

### Indexing

```bash
python3 cs_index.py repos
```

Walks every configured repo, chunks source files into overlapping line
windows (`chunk_size`/`chunk_overlap`), and embeds them. It checksums each file, so re-running only re-embeds
what changed — run it after a `git pull`.

```bash
python3 cs_index.py mbox warp-group archive.mbox
```

Imports a mailing-list archive (Google Groups exports an mbox) into a corpus.
`cs_index.py text <corpus> <dir>` does the same for a directory of `.txt`,
`.md`, or `.json` posts — use it for scraped forum threads; JSON items may
carry `id`, `title`, `author`, `date`, `body`, `url`.

```bash
python3 cs_index.py status
```

Shows chunk/document counts per source and flags repos whose path is missing.

### Querying from the shell

```bash
python3 cs_query.py code "how are CTF parameters fitted" --repo warp --show
```

```bash
python3 cs_query.py forum "tilt series stuck at 0%"
```

```bash
python3 cs_query.py context "why does frame series alignment deselect items" --repo warp --forum
```

`context` prints a paste-ready block — question, matching source with line
numbers, optionally prior discussions — for feeding to a model that has no
tools of its own.

## Querying from an LLM (MCP)

`.mcp.json` registers the server — edit the placeholder path to point at your
clone, then copy the block into any project's `.mcp.json` (or your MCP client's
config) and the model gets four tools:

- `list_sources()` — what's indexed
- `search_code(query, repo="", top_k=5)` — file + line range + score
- `read_code(repo, file, start_line, end_line)` — the actual lines
- `search_forum(query, corpus="", top_k=5)` — prior discussions

The server loads indexes into memory at startup, so restart it after re-indexing.

Guidance worth putting in your `CLAUDE.md` (or system prompt): search code with
a targeted query derived from the question, read the real source before
answering, and treat forum hits as leads rather than truth — archived answers
go stale when the software changes.

## Reusing the warpembot data

The 2188-message Google Group corpus is already in `corpora/warp-group/`, copied
from `../warpembot-public/threads/`. `load_doc` maps the old
`message_id`/`subject`/`sender` keys onto `id`/`title`/`author`, so it needs no
conversion — only re-embedding:

```bash
python3 cs_index.py rebuild warp-group
```

The prebuilt `.npz` indexes that shipped with warpembot were built with
`qwen3-embedding:8b` and are therefore unusable under the current model. They
are parked in `index_qwen8b_legacy/` — delete that directory once the rebuild
is done, or restore it if you ever switch back.