# codebase-rag

A self-updating, symbol-aware vector index **and** reference graph of a codebase, exposed over
MCP, so any MCP-compatible LLM agent can understand a whole repository cheaply instead of burning
tokens on dozens of grep and glob calls.

## Why

Agents working against a shared codebase pay for context in two bad ways: full-context stuffing
(expensive, slow, breaks on large repos) or ad hoc grep/glob exploration (burns tool calls,
produces inconsistent understanding across agents). Most of that grep traffic is not semantic
search; it is structural traversal: "where is this defined", "who calls this", "what does this
depend on", "what breaks if I change it". A vector index alone cannot answer those. codebase-rag
pairs a semantic vector index with a symbol reference graph and serves both over MCP, so an agent
answers those questions in one tool call instead of N greps, and the index stays in sync as the
code changes.

## What it gives an agent

Three layers, all queryable over MCP:

1. Vector layer: semantic search over symbol-level chunks (functions, methods, classes, module
   statements) for Python and JS/TS.
2. Graph layer: a reference/call graph (calls, imports, contains edges) with go-to-definition,
   find-callers, find-callees, dependencies, and neighborhood traversal.
3. Understanding layer: whole-codebase tools built on the graph: a token-budgeted repo map
   (PageRank over the reference graph), hybrid graph-aware retrieval, and change-impact analysis.

## Architecture

```
codebase-rag/
  chunker/     tree-sitter symbol chunking (Python, TS/JS/JSX/TSX) + edge extraction + ignore rules
  store/       LanceDB tables (chunks, edges, annotations, repo_meta) + embedder fingerprinting
  graph/       name resolution + dependency-free PageRank
  sync/        git blob-hash staleness audit + pre-push hook backstop
  mcp/         the MCP tool surface (service core + server wiring)
  cli.py       init / reindex / status / serve / graph
```

Storage, embedding, and reranking are not reimplemented here. codebase-rag depends on
[`rag-timetravel`](https://pypi.org/project/rag-timetravel/) for the `Embedder` protocol, the
reranker, and LanceDB dataset versioning. It uses a direct-write path: `rag-timetravel`'s own
`ingest()` re-chunks with a fixed-size splitter and has a hardcoded schema, so codebase-rag owns
its own symbol-aware LanceDB tables and writes pre-chunked symbol rows directly, reusing only the
embedder and reranker. This package is the code-aware layer on top: chunking, edges, staleness,
sync, and the MCP tools.

### Storage schema

Four LanceDB tables, versioned via `rag-timetravel`'s dataset versioning:

- `chunks`: one row per symbol. Fields include `chunk_id`, `file_path`, `symbol`, `symbol_kind`,
  `content`, `embedding`, `imports`, `git_blob_sha`, `start_line`, `end_line`, `repo_id`,
  `embedder_fingerprint`.
- `edges`: one row per graph edge. Fields: `id`, `src_symbol`, `dst_name`, `edge_kind`
  (`calls` | `imports` | `contains`), `file_path`, `git_blob_sha`, `start_line`, `repo_id`.
- `annotations`: semantic notes attached to a symbol, versioned separately from `chunks` so
  reindexing a symbol never touches its annotations.
- `repo_meta`: per-repo metadata (`repo_id`, `root_path`, `embedder_fingerprint`,
  `last_full_index_sha`, `languages`).

### Embedder fingerprinting

`embedder_fingerprint` is a hash of the embedder's model id and dimensionality, stored in
`repo_meta` at init time. Every write path checks the incoming embedder's fingerprint before
writing; on mismatch the write is rejected rather than silently corrupting search quality for a
shared index. Switching embedders requires an explicit full re-embed.

### Staleness detection

A chunk is stale when its stored `git_blob_sha` no longer matches the current blob hash of its
file (`git hash-object`). `status` walks the index, recomputes current blob hashes, and reports
`ok` / `stale` / `deleted` per file. This is a deterministic audit, independent of any wall-clock
or version-count heuristic.

### Sync

Two triggers keep the index fresh, both re-deriving chunks and edges from the file on disk (the
index is never written directly by an LLM):

- Git pre-push hook (`codebase-rag init --install-hook`): the correctness backstop; catches edits
  from anything that did not go through an MCP client (IDE edits, merges).
- The `reindex_file` MCP tool: the latency optimization; an agent calls it right after editing a
  file so its own session sees fresh results without waiting for a push.

Concurrent `reindex_file` writes are safe: LanceDB uses manifest-based optimistic concurrency, and
the store retries on commit conflict rather than overwriting.

## Install

```bash
pip install codebase-rag
# or, from source:
pip install -e .
```

Requires Python 3.10+ and a git repository as the source of truth for change detection. The
default embedder (`all-MiniLM-L6-v2`, local, 384-dim) needs the optional `local` extra:

```bash
pip install -e ".[local]"
```

## Quickstart

```bash
# 1. Index a repo and install the pre-push hook
codebase-rag init /path/to/repo --install-hook

# 2. Check freshness at any time
codebase-rag status

# 3. Reindex manually (full, or only what changed since a commit)
codebase-rag reindex --full
codebase-rag reindex --changed-since <sha>

# 4. Print the neighborhood of a symbol as text
codebase-rag graph UserService.authenticate --depth 1

# 5. Serve the MCP tools to an agent
codebase-rag serve
```

Point any MCP client at `codebase-rag serve` and the tools below become available.

## Add it to an existing codebase

Five minutes, from the root of a git repository:

```bash
# 1. Install (with the local embedder extra so no API key is needed)
pip install -e ".[local]"        # or: pip install codebase-rag[local]

# 2. (optional) Tell it what to skip, on top of the built-in defaults
#    (node_modules, .venv, dist, build, __pycache__, *.min.js are always skipped)
cat > .codebaseragignore <<'EOF'
# gitignore syntax
migrations/
*.generated.ts
vendor/
EOF

# 3. Build the index and install the git pre-push hook
codebase-rag init . --install-hook

# 4. Confirm it indexed cleanly
codebase-rag status          # expect: ok=<N> stale=0 deleted=0
```

The first `init` embeds every symbol, so it takes a moment on a large repo; subsequent updates are
incremental. From then on the index stays fresh two ways: the pre-push hook reindexes changed files
on every push, and an agent can call `reindex_file` right after editing. Commit `.codebaseragignore`
if you want it shared; the `.codebase-rag/` index directory is local and should stay gitignored
(add `/.codebase-rag/` to your `.gitignore`).

For a team, everyone runs `init` once locally against the same checkout. The index itself is not
committed; it is derived from source, so each clone rebuilds it deterministically.

## Use it with a coding agent

`codebase-rag serve` is a standard stdio MCP server, so any MCP-capable agent can connect with the
usual config. In every case the command is `codebase-rag` and the args are `serve --repo <path>`.

**Claude Code** (from the repo root):

```bash
claude mcp add codebase-rag -- codebase-rag serve --repo .
```

or add it to `.mcp.json` at the project root (shareable with the team):

```json
{
  "mcpServers": {
    "codebase-rag": {
      "command": "codebase-rag",
      "args": ["serve", "--repo", "."]
    }
  }
}
```

**Cursor** (`.cursor/mcp.json` in the project, or the global `~/.cursor/mcp.json`):

```json
{
  "mcpServers": {
    "codebase-rag": {
      "command": "codebase-rag",
      "args": ["serve", "--repo", "/absolute/path/to/repo"]
    }
  }
}
```

**Codex CLI** (`~/.codex/config.toml`):

```toml
[mcp_servers.codebase-rag]
command = "codebase-rag"
args = ["serve", "--repo", "/absolute/path/to/repo"]
```

Any other MCP client (Windsurf, Zed, Continue, a custom Agent SDK host) uses the same
command/args pair. If `codebase-rag` is installed in a virtualenv, point `command` at that env's
executable (for example `/path/to/.venv/bin/codebase-rag`) so the agent launches the right one.

### How an agent should use it

The tools are most effective in this order; a short system-prompt note like the following trains
the agent to prefer them over grep:

> This repo has a `codebase-rag` MCP server. Call `get_repo_map` first to orient. Use
> `search_context` for "how does X work", `find_definition` / `get_callers` / `get_callees` for
> navigation, and `impact_of` before changing a signature. Prefer these over grep. After editing a
> file, call `reindex_file` on it so later searches stay accurate.

A typical loop: `get_repo_map` to learn the important symbols, `search_context("where are requests
authenticated")` to pull a function plus its call context in one shot, `get_callers` /
`impact_of` to scope a change, edit, then `reindex_file` to keep the index in sync.

## MCP tools

### Vector and retrieval

- `search_code(query, k=5, kind=None, include_content=False)`: semantic search over symbols,
  optionally filtered by `symbol_kind` (`function` | `method` | `class` | `module_statement`).
  Returns bounded previews by default; pass `include_content=True` or follow the returned
  `chunk_id` with `get_symbol` when full source is needed.
- `search_context(query, k=1, hops=1)`: hybrid graph-aware retrieval. Finds semantic entry points,
  then graph-expands each `hops` steps into one coherent, token-bounded subgraph. Nodes are tagged
  by role (`match` | `caller` | `callee` | `dependency`). Use this instead of a search followed by
  several follow-up greps.

### Structural navigation (the grep replacements)

- `find_definition(name)`: candidate definitions for a name (go-to-definition), resolved across
  files by name plus import hints.
- `get_callers(symbol)`: every call site of a symbol.
- `get_callees(symbol)`: everything a symbol calls.
- `get_dependencies(file_path)`: the imports a file depends on.
- `neighborhood(symbol, depth=1)`: the connected subgraph around a symbol, token-bounded.

### Whole-codebase understanding

- `get_repo_map(budget_symbols=12, file_path=None)`: a token-budgeted structural map. Ranks
  symbols by PageRank over the reference graph and returns the most central ones as a compact
  skeleton. Call this first to orient in an unfamiliar repo for a few hundred tokens instead of
  reading dozens of files. Pass `file_path` to focus the map on one file and its neighbors.
- `impact_of(symbol, max_depth=3)`: reverse-reachability. The transitive set of symbols that depend
  on the given symbol. Check this before changing a signature.

### Retrieval and bookkeeping

- `get_symbol(symbol_id)`: a single chunk plus its attached annotations.
- `get_file_context(file_path, include_content=False)`: ordered symbol previews for a file.
  Pass `include_content=True` to retrieve every complete chunk.
- `reindex_file(file_path)`: re-derive a single file's chunks and edges from disk. Call after any
  edit; skipping it leaves subsequent searches against that file stale.
- `annotate(symbol_id, note, author="agent")`: attach a semantic note to a symbol (never mutates
  derived chunks or embeddings).
- `status()`: the staleness audit (`ok` / `stale` / `deleted` per file).

## Language support and resolution strategy

- Python (`tree-sitter-python`): top-level functions, classes (whole-class chunk plus one chunk per
  method), and a single module-statement chunk. Decorators attach to their target. Imports are
  collected once per file and attached to every chunk.
- JS/TS (`tree-sitter-typescript`): function declarations, named arrow/function-expression
  bindings, class methods, and exported const/type declarations. `.tsx` / `.jsx` components are
  chunked whole (JSX body included).

Graph edges are extracted from the same parse (no second pass): `contains` (class to method),
`imports` (module to imported name), and `calls` (enclosing symbol to callee identifier).
Resolution of a call/reference name to a concrete definition is **intra-file precise, cross-file
name-based**: exact within a file, matched by name plus import hints across files. Full cross-file
type resolution (the job of an LSP or type checker) is intentionally out of scope; unresolved names
are still stored as useful candidate edges, and ambiguous names return multiple candidates.

Both chunkers skip generated files, `node_modules`, `.venv`, and anything matching a
`.codebaseragignore` file (gitignore syntax).

## Configuration

- Embedder: any model the `rag-timetravel` `Embedder` protocol accepts. Default `all-MiniLM-L6-v2`
  (local). OpenAI-compatible and Ollama endpoints are supported via the `rag-timetravel` factory.
- Reranker: `none` by default; other rerankers from `rag-timetravel` can be selected.
- Default index location: `<repo_root>/.codebase-rag/lancedb`.

## Development

```bash
# Tests (unit + integration are offline and fast; the slow e2e uses a real model)
pip install -e ".[test,local]"
python -m pytest -q                 # full suite
python -m pytest -q -m "not slow"   # skip the model-download e2e
```

## Non-goals

- No non-git staleness backend; git is the source of truth for change detection.
- No languages beyond Python and JS/TS.
- No UI; CLI and MCP tools only.
- No direct LLM writes to chunk text or embeddings; the index is always re-derived from source.
- No hosted multi-tenant service; this ships as a library and CLI.

## License

MIT. See [`LICENSE`](LICENSE).