Codebase Copilot

An MCP server that gives Claude Code, Cursor and Claude Desktop real code search over any repository — syntax-aware chunking, local embeddings, hybrid retrieval, and a measured answer to "is this actually better than naive RAG?"

Live demo: https://codebase-copilot-omega.vercel.app · Remote MCP endpoint: https://codebase-copilot-omega.vercel.app/api/mcp

50 hand-verified questions over honojs/hono (187 files, 26k lines). Full ablation below.

Why

Coding agents find code with grep. That works when you know the identifier and fails when you only know the behaviour ("where do we stop SSG writing outside the output dir?"). Most RAG-over-code demos fix that with fixed-size text windows and one embedding call — and never measure whether it works. This project does the retrieval properly, exposes it over MCP so the host model does the reasoning, and measures every design decision.

Related MCP server: code-search

Architecture

            index time (local CLI)                              query time (stdio or Vercel function)
 repo ─► walker ─► tree-sitter chunker ─► embedder ─► index/ ─► dense cosine ──┐
        .gitignore    symbol-level chunks    bge-small   chunks.json            ├─► RRF ─► (optional rerank) ─► cited chunks
        secret skip   + context header       ONNX, local vectors.bin  BM25 ─────┘
                                                                     code-aware
                                   ┌──────────── one tool registry ────────────┐
                                   │  stdio  ─► Claude Code / Cursor (local)   │
                                   │  Streamable HTTP ─► /api/mcp (hosted)     │
                                   └───────────────────────────────────────────┘

MCP tools

All tools are annotated readOnlyHint, validate input with Zod, return both Markdown and structuredContent, and fail with a message that says what to do next (Unknown repo "x". Available: hono.).

Quick start

git clone https://github.com/yashharkawat/codebase-copilot && cd codebase-copilot && npm install

# 1. Index any repository (first run downloads the 34 MB embedding model)
npm run cli -- index /path/to/repo --id myrepo            # add --include src,lib to narrow it

# 2. Try it from the terminal
npm run cli -- search myrepo "where are sessions invalidated?"

# 3. Connect it to Claude Code (stdio, fully local — your code never leaves the machine)
claude mcp add codebase-copilot -- npx tsx "$PWD/cli/main.ts" serve

# Or use the hosted demo over HTTP (searches honojs/hono)
claude mcp add --transport http codebase-copilot https://codebase-copilot-omega.vercel.app/api/mcp

Cursor / Claude Desktop (mcp.json):

{ "mcpServers": { "codebase-copilot": { "command": "npx", "args": ["tsx", "/abs/path/codebase-copilot/cli/main.ts", "serve"] } } }

Web app: npm run dev. Optional generated answers: copy .env.example to .env.local and add an OpenRouter key.

Evaluation

npm run eval — 50 questions a new contributor might ask, each mapped by hand to the exact function that answers it (eval/hono.questions.json, pinned to commit 098e119). 30 are semantic: phrased without any identifier from the target. A hit requires the right file and an overlapping line range — the right file with the wrong function scores zero.

What the numbers say:

• Hybrid is the biggest single win. BM25 alone beats dense alone on code. Dense retrieval rescues the questions that share no vocabulary with the code; BM25 rescues the ones where a small general-purpose embedding model is vague. Together: 100% of lexical and 80% of semantic questions in the top 5.

• AST chunking mostly improves precision, not recall. Recall@5 moves 84% → 88% under hybrid, but Recall@1 moves 54% → 62%: the first result is the function itself rather than a window that happens to contain part of it. For an agent with a limited context budget, that is the metric that matters.

• The cross-encoder reranker made things worse — Recall@5 88% → 80%, at 658 ms per query. ms-marco-MiniLM was trained on web passages; on code it prefers chunks with English-like comments over the chunk with the right logic. It ships off. I kept it in the table and behind a flag because a negative result you measured is worth more than a best practice you assumed.

• One eval-driven bug fix: the misses exposed that class fields holding arrow functions (#cachedBody = (key) => {…}) were not being treated as methods. That is a chunker bug class, not tuning to the test set; no retrieval parameter was tuned on these 50 questions.

Limitations, stated plainly: one repository, one language, 50 questions, written by an LLM and verified against the source rather than collected from real users. The naive baseline uses overlapping 50-line windows, which makes the overlap criterion slightly easier for it. An agent-level eval in mcp-builder format lives in eval/mcp_evaluation.xml (10 tool-use questions with verified answers); it needs an Anthropic API key to run and has not been run yet.

Security

Designed security-first: the checkpoint in specs/security-checkpoint.md was written before the code.

• No request value ever becomes a file path. Tools answer from the in-memory index. Repo ids must match ^[a-z0-9][a-z0-9._-]{0,63}$ and exist in the on-disk manifest list; chunk ids are map lookups.

• Input validation everywhere (Zod): bounded query length, enum modes, capped limits, 8 KB body cap. Identifier search uses indexOf + boundary checks — there is no user-built RegExp, so no ReDoS.

• The indexer protects you from yourself: honours .gitignore, never follows symlinks, hard-skips .env*, keys, keystores and lockfiles, and skips any file containing a private key, AWS, GitHub, Slack or LLM-provider token.

• Prompt injection: retrieved code is untrusted data. It is fenced in tags carrying a per-request random suffix (code cannot forge the closing tag), the system prompt says so, and the answering model is given no tools — a successful injection can change text, nothing else. Covered by tests/security.test.ts.

• Cost safety: assertFreeModel() throws on any model id that is not a :free slug, before a request is made.

• Web: per-request nonce CSP with strict-dynamic, default-src 'none' on all API routes, HSTS, nosniff, frame-ancestors 'none'. Code is rendered as React text nodes only. Errors are mapped to safe messages; stack traces stay in server logs.

• Rate limits per IP on search, ask and MCP. Known limitation: in-memory, so per warm instance on serverless — a speed bump, not a quota.

• Deliberately no auth: the hosted data is public open source and every tool is read-only. For private code, use stdio — nothing leaves the machine.

Project layout

src/core/     walker · chunker · embedder · bm25 · store · retriever · reranker · indexer
src/mcp/      tool + resource registry (shared by both transports)
src/server/   zod schemas · rate limiter · error mapping · OpenRouter client
cli/          index · search · serve (stdio)
app/          Next.js UI + /api/search · /api/ask · /api/mcp · /api/repos · /api/health
eval/         gold questions · ablation runner · results · MCP agent eval
specs/        design doc · security checkpoint
tests/        28 tests: chunker, tokenizer/BM25, walker, retriever, security
index/hono/   the shipped demo index (2.7 MB)

npm test · npm run typecheck · npm run lint · npm run eval

License

MIT