CodebaseRAG-LanceDB-TreeSitter-MCP-Based-Symbol-Aware-Codebase-Retrieval-System

A self-updating, symbol-aware vector index and relational reference graph of codebases exposed over the Model Context Protocol (MCP), enabling AI coding agents to navigate, understand, and perform structural impact analysis across whole repositories with sub-second precision and strict token bounds.

Features

Related MCP server: OpenCodeHub MCP Server

Prerequisites

• Python: Version 3.10 or higher (CI runs on 3.10, 3.11, 3.12, and 3.13).

• Git: Command-line tool available on the system PATH (used for git hash-object, git diff, and pre-push hook automation).

• C/C++ Build Tools: A standard C compiler environment (GCC, Clang, or MSVC) if installing from source, or use pre-compiled wheels for Tree-sitter binaries.

Check your installed versions:

python --version
git --version

Setup from scratch

1. Clone the repository

git clone https://github.com/ahmadsurti/CodebaseRAG-LanceDB-TreeSitter-MCP-Based-Symbol-Aware-Codebase-Retrieval-System.git
cd CodebaseRAG-LanceDB-TreeSitter-MCP-Based-Symbol-Aware-Codebase-Retrieval-System

2. Create and activate a virtual environment

# Linux / macOS
python3 -m venv .venv
source .venv/bin/activate

# Windows (PowerShell)
python -m venv .venv
.venv\Scripts\Activate.ps1

3. Install dependencies

Install the package in editable mode with the local embedding extra (sentence-transformers for offline vector generation):

# Core package with local offline embedder (all-MiniLM-L6-v2)
pip install -e ".[local]"

For development, testing, and evaluation:

# Full development dependencies including pytest and pytest-asyncio
pip install -e ".[test,local]"

4. Configure exclusion rules (Optional)

codebase-rag includes built-in defaults that always skip node_modules/, .venv/, venv/, .git/, dist/, build/, __pycache__/, .pytest_cache/, *.min.js, *.min.css, *.map, *.pyc, *.lock, and *.generated.*.

To add project-specific ignore patterns, create a .codebaseragignore file at your target repository's root using standard .gitignore syntax:

cat << 'EOF' > .codebaseragignore
# Project-specific exclusions
migrations/
vendor/
docs/_build/
*.gen.ts
EOF

5. Initialize the index and git pre-push hook

Build the initial LanceDB symbol index and install the Git pre-push hook in one command:

codebase-rag init . --install-hook

Note: The initial run downloads the default all-MiniLM-L6-v2 embedding model (~80 MB) to your local Hugging Face cache (~/.cache/huggingface) and computes embeddings for every symbol.

6. Verify index freshness

Audit the index against the working tree to confirm all files are indexed cleanly:

codebase-rag status

Expected output:

ok=<count> stale=0 deleted=0

Configuration / Environment Variables

The default configuration operates 100% offline using the local all-MiniLM-L6-v2 sentence-transformer model and an embedded LanceDB database stored at .codebase-rag/lancedb. In this mode, no environment variables or API keys are required.

If integrating remote embedding providers supported by rag-timetravel (e.g., OpenAI-compatible endpoints), supply the provider's standard environment variables:

# Optional: Only required when using non-default remote embedding endpoints
OPENAI_API_KEY=your_actual_api_key_here

How to use

CLI Workflow

codebase-rag provides five core CLI subcommands for index management, verification, graph inspection, and server hosting:

1. Index a repository

# Index a local directory
codebase-rag init /path/to/project --install-hook

# Or clone and index a remote Git repository in one step
codebase-rag init https://github.com/example/sample-repo.git

2. Verify index freshness

codebase-rag status --repo .

3. Inspect a symbol's graph neighborhood

Inspect connected nodes and in/out edges directly from the terminal without launching an agent:

codebase-rag graph UserService.authenticate --depth 1

4. Reindex changed files

# Reindex all modified and deleted files since a specific commit
codebase-rag reindex --changed-since HEAD~1

# Or perform an exhaustive reindex across all tracked files
codebase-rag reindex --full

5. Launch the MCP server

# Launch stdio transport (default)
codebase-rag serve --repo .

# Or launch streamable HTTP transport for networked clients
codebase-rag serve --repo . --transport http

Agent Integration (MCP Setup)

Connect codebase-rag to any MCP-compliant coding assistant:

Claude Code

Add to your project root from the terminal:

claude mcp add codebase-rag -- codebase-rag serve --repo .

Or create a .mcp.json file in the project root:

{
  "mcpServers": {
    "codebase-rag": {
      "command": "codebase-rag",
      "args": ["serve", "--repo", "."]
    }
  }
}

Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "codebase-rag": {
      "command": "codebase-rag",
      "args": ["serve", "--repo", "/absolute/path/to/repository"]
    }
  }
}

Codex CLI (~/.codex/config.toml)

[mcp_servers.codebase-rag]
command = "codebase-rag"
args = ["serve", "--repo", "/absolute/path/to/repository"]

Recommended Agent Query Pattern

To maximize agent comprehension and minimize token expenditure, prompt agents to follow this operational workflow:

flowchart TD
    A[Orient: get_repo_map] --> B[Investigate: search_context]
    B --> C[Traverse: find_definition / get_callers / get_callees]
    C --> D[Safety Audit: impact_of]
    D --> E[Execute Code Modification]
    E --> F[Sync Index: reindex_file]

1. Orientation: Call get_repo_map(budget_symbols=12) first to obtain a PageRank-weighted structural skeleton of the repository for ~300 tokens instead of reading dozens of files.

2. Contextual Retrieval: Use search_context(query="authentication token validation", hops=1) to pull semantic matches along with immediate caller/callee context in a single call.

3. Symbol Navigation: Use find_definition, get_callers, and get_callees to trace structural relationships across files without burning turns on grep or file-globbing.

4. Pre-Change Impact Scoping: Call impact_of(symbol="UserService.authenticate", max_depth=3) before altering any function signature to identify all transitive dependents.

5. Session Freshness: Call reindex_file(file_path="src/service.py") immediately after modifying a file so subsequent queries in the same agent session reflect the latest AST.

Available MCP Tools (14 Tools)

Vector & Hybrid Retrieval

• search_code(query: str, k: int = 5, kind: str | null = null, include_content: bool = false): Semantic vector search over symbols (function, method, class, module_statement). Returns 500-char truncated previews by default.

• search_context(query: str, k: int = 1, hops: int = 1): Hybrid retrieval uniting vector similarity with 1-hop reference graph expansion (capped at 6 nodes, 5 edges, with role attribution: match, caller, callee, dependency).

Structural Code Graph Navigation

• find_definition(name: str): Cross-file candidate definitions resolved by exact name, suffix matching, and import hints.

• get_callers(symbol: str): Discovers all call sites referencing the specified symbol across the repository.

• get_callees(symbol: str): Lists symbols invoked by the specified function or method with candidate target resolutions.

• get_dependencies(file_path: str): Returns import dependencies declared within the specified source file.

• neighborhood(symbol: str, depth: int = 1): BFS traversal around a symbol returning connected nodes and directed edges (capped at 60 nodes).

Repository-Wide Architecture & Impact

• get_repo_map(budget_symbols: int = 12, file_path: str | null = null, include_flat: bool = false): Token-budgeted structural skeleton ranked by PageRank over the reference call graph.

• impact_of(symbol: str, max_depth: int = 3): Computes reverse transitive dependents up to $N$ hops, automatically following class containment into methods.

Inspection & Freshness Management

• get_symbol(symbol_id: str): Fetches a single indexed chunk row alongside any attached semantic annotations.

• get_file_context(file_path: str, include_content: bool = false): Returns ordered symbol previews for an entire file.

• reindex_file(file_path: str): Re-parses and re-embeds a single file on disk immediately after modification.

• annotate(symbol_id: str, note: str, author: str = "agent"): Attaches a semantic note to a symbol in an independent table without mutating the chunk embedding.

• status(): Runs the deterministic Git blob-hash audit, reporting ok_count, stale files, and deleted files.

Project structure

.
├── .codebaseragignore        # Default ignore overrides (gitignore syntax)
├── .github/
│   └── workflows/
│       └── ci.yml            # Multi-version Python CI matrix (3.10-3.13) & e2e test
├── .gitignore                # Git untracked path specifications
├── LICENSE                   # Apache License 2.0
├── NOTICE                    # Legal attribution and project provenance notice
├── README.md                 # Primary system documentation
├── TUTORIAL.md               # Step-by-step setup and AI client integration tutorial
├── pyproject.toml            # Hatchling build config, dependencies, entry points, tool options
├── scripts/
│   └── eval.py               # Golden retrieval benchmark harness (Hit@k, MRR, latency)
├── serve_sse.py              # SSE HTTP runner for Claude Web and remote connectors
├── src/
│   └── codebase_rag/
│       ├── __init__.py       # Package initialization and exported symbols
│       ├── cli.py            # CLI argument parsing, git URL resolution, and subcommand dispatch
│       ├── chunker/
│       │   ├── __init__.py   # Chunker registry and exports
│       │   ├── base.py       # Chunk and Edge dataclasses, Chunker protocol, shared parse helpers
│       │   ├── ignore.py     # IgnoreMatcher with pathspec support and fnmatch fallback
│       │   ├── markdown.py   # ATX heading-based MarkdownChunker (.md, .mdx)
│       │   ├── python.py     # Tree-sitter AST symbol and call-edge extractor for Python (.py)
│       │   └── typescript.py # Tree-sitter AST extractor for TS/JS (.ts, .tsx, .js, .jsx)
│       ├── graph/
│       │   ├── __init__.py   # Graph module exports
│       │   ├── rank.py       # Pure Python power-iteration PageRank algorithm
│       │   └── resolve.py    # Inverted dotted-suffix index and edge destination resolution
│       ├── mcp/
│       │   ├── __init__.py   # MCP module exports
│       │   ├── server.py     # FastMCP and low-level Server wiring for stdio and HTTP
│       │   └── tools.py      # CodebaseRagService implementation of all 14 MCP tools
│       ├── store/
│       │   ├── __init__.py   # Storage module exports
│       │   ├── fingerprint.py# Cryptographic SHA-1 embedder model fingerprinting
│       │   └── schema.py     # PyArrow schemas and async LanceDB CodebaseStore with retry logic
│       └── sync/
│           ├── __init__.py   # Synchronization module exports
│           ├── hook.py       # Idempotent .git/hooks/pre-push installer and git diff parser
│           └── staleness.py  # Content-addressable git hash-object blob auditor
└── tests/
    ├── conftest.py           # Shared test fixtures (FakeEmbedder, tmp_repo)
    ├── fixtures/
    │   └── golden_qa.json    # Evaluation question set with ground-truth symbols/files
    ├── test_chunker_markdown.py
    ├── test_chunker_python.py
    ├── test_chunker_typescript.py
    ├── test_cli.py
    ├── test_e2e.py
    ├── test_edges_python.py
    ├── test_edges_typescript.py
    ├── test_fingerprint.py
    ├── test_graph_service.py
    ├── test_graph_store.py
    ├── test_hook.py
    ├── test_ignore.py
    ├── test_impact.py
    ├── test_repo_map.py
    ├── test_resolve.py
    ├── test_search_context.py
    ├── test_service.py
    ├── test_staleness.py
    └── test_store.py

Configuration reference

Deployment

codebase-rag is designed as a developer CLI and embedded service that runs locally alongside developer workflows or inside containerized AI agent environments:

• Local stdio Server: The primary deployment mode. Agent hosts (Claude Code, Cursor, Windsurf, Zed, Codex) launch codebase-rag serve --repo . as a child process and communicate through standard input/output streams.

• Networked Streamable HTTP: By supplying --transport http, codebase-rag serves tools over an HTTP endpoint, enabling Docker containers, remote development environments, or cloud sandboxes to query the local index over the network.

• Continuous Integration Pipeline: Configured in .github/workflows/ci.yml with:

  • Automated code formatting and style checks via ruff check .

  • Offline unit and integration test matrix executing across Python 3.10, 3.11, 3.12, and 3.13 with pytest -q -m "not slow"

  • End-to-end integration job running against the real all-MiniLM-L6-v2 model with Hugging Face cache preservation (hf-${{ runner.os }}-all-MiniLM-L6-v2).

• Retrieval Benchmark Gate: The evaluation harness in scripts/eval.py runs against tests/fixtures/golden_qa.json and enforces a strict quality gate:

  python scripts/eval.py --db .codebase-rag/lancedb --repo . --k 5

  The script exits with status 0 if $\text{Hit}@3 \ge 70%$, and 1 otherwise, making it suitable for pull request regression gating.

Troubleshooting

What I learned from building this

Building codebase-rag provided concrete architectural lessons regarding the intersection of language parsing, vector databases, graph theory, and LLM agent UX:

1. Direct-Write Columnar Storage vs. Re-Chunking Pipelines

Standard RAG frameworks rely on monolithic ingest pipelines that split text using fixed character or token windows. When applied to source code, arbitrary splitting fractures abstract syntax trees—severing function signatures from their docstrings, or classes from their member methods. While codebase-rag utilizes rag-timetravel for embedder abstractions and dataset versioning, it deliberately bypasses rag-timetravel's fixed-size ingest() pipeline. By engineering a custom direct-write path to LanceDB using explicit PyArrow schemas (chunks, edges, annotations, repo_meta), the storage layer preserves AST boundaries, line numbers, and symbol identities natively without intermediate re-chunking distortion.

2. Collapsing Graph Traversal Complexity from $O(N \times M)$ to $O(N + M)$

In the initial v1.0.0 implementation, get_repo_map resolved target edge names by executing a full-table chunk query for every reference edge. On a repository with ~1,200 symbols and ~7,000 edges, this produced thousands of database round-trips ($O(\text{edges} \times \text{chunks})$), driving repo-map generation times above 55 seconds and triggering MCP client timeouts. Refactoring resolution in v1.0.1 around an inverted dotted-suffix name index (build_name_index) allowed the entire symbol set to be indexed into memory in a single pass. Resolving edges against this in-memory structure collapsed traversal time to under 300 milliseconds—a ~180x speedup achieved purely through algorithmic reduction rather than caching.

3. Anchoring Freshness in Content-Addressable Git Storage

Maintaining index freshness across multi-branch software development is notoriously fragile. Background file-watcher daemons leak memory, lose sync across rapid branch checkouts, and fail silently across worktrees. Instead of runtime heuristics, codebase-rag anchors staleness detection directly in Git's object database. Computing git hash-object on working-tree files provides a deterministic, zero-overhead hash comparison against the stored git_blob_sha. Coupled with an idempotent Git pre-push hook that reindexes only modified refs (git diff --name-status), the index stays synchronised with source truth without persistent background daemon overhead.

4. Designing Bounded Tool Surfaces for Agent Cognitive Load

Unbounded tool outputs saturate an LLM agent's context window, degrading reasoning quality and inflating operational token costs. Designing MCP tools for coding agents requires defensive bounding:

• search_code returns 500-character truncated source previews with a explicit content_truncated boolean flag, leaving full retrieval to dedicated get_symbol calls.

• search_context restricts expansion to a deterministic envelope of 6 nodes and 5 edges, prioritizing nodes by connectivity to the primary match.

• get_repo_map defaults to 12 PageRank-selected landmark symbols. These bounds allow agents to orient and navigate complex repositories within compact token envelopes (~300 tokens per call) instead of consuming thousands of tokens on blind file dumping.

5. Transactional Integrity via MVCC Retries and Table Decoupling

Because coding agents can invoke reindex_file concurrently while querying vector tables, storage operations must withstand concurrent writes. LanceDB uses manifest-based optimistic concurrency control. Wrapping all delete-and-add operations in an exponential backoff retry loop (_write_with_retry) guarantees that concurrent write conflicts resolve safely without data loss. Furthermore, isolating annotations into an independent table keyed by symbol ID ensures that reindexing or replacing an AST chunk never touches user- or agent-attached semantic commentary.

6. Vector Homogeneity via Cryptographic Fingerprinting

A silent failure mode in vector-search systems occurs when indices are queried or updated using different embedding models or dimensionalities. Cosine similarities become mathematically invalid without triggering overt runtime crashes. Storing an embedder fingerprint (sha1(model_name:dim)) in repo_meta and asserting it on every write transaction (check_fingerprint) guarantees vector space homogeneity at the storage boundary, failing fast with an explicit FingerprintMismatch error rather than silently corrupting search quality.

Biggest Takeaway

Codebases are graphs of semantic symbols, not unstructured streams of text. Effective AI-assisted engineering requires bridging statistical vector retrieval (discovering relevant concepts) with deterministic graph traversal (tracing callers, callees, and change impact). By coupling Tree-sitter AST extraction with LanceDB columnar vectors, deterministic Git staleness audits, and bounded MCP tooling, codebase-rag replaces brute-force grepping with an integrated, sub-second code intelligence layer built specifically for the constraints of autonomous coding agents.

License

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for the full license text and NOTICE for copyright attribution.