# ctxrank

A ranked, token-budgeted map of any codebase, built for AI coding agents.

<p align="center">
  <img src="assets/demo.gif" alt="ctxrank finding dictresize in CPython, listing its callers, then printing a 120-token map for the query dict resize" width="760">
</p>

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="assets/bench-dark.svg">
  <img alt="Time to print a 2048-token map of the Linux kernel, rust-lang/rust and CPython, ctxrank against aider's repo map" src="assets/bench-light.svg">
</picture>

On CPython, ctxrank prints a 2,048-token map of the repo in 1.2 seconds from a cold start and
in 0.16 seconds once its cache exists. aider's own repo map took 88 seconds on the same checkout.
Dumping all of CPython's source as text would cost 20.6 million tokens.

ctxrank reads every source file with tree-sitter, records where each function, type and macro is
defined and where it is used, ranks the definitions with personalised PageRank over that reference
graph, and prints the top signatures that fit in the token budget you give it. An agent gets the
shape of a repo for a couple of thousand tokens instead of reading files one by one.

The ranking idea comes from [aider's repo map](https://aider.chat/docs/repomap.html). I wanted it
outside aider, as one binary any agent can call, fast enough for repos the size of the kernel.

```console
$ ctxrank sym dictresize
Objects/dictobject.c:2206  function  static int dictresize(PyDictObject *mp, uint8_t log2_newsize, int unicode)
Objects/dictobject.c:432  function  static int dictresize(PyDictObject *mp, uint8_t log_newsize, int unicode);

$ ctxrank callers dictresize
Objects/dictobject.c:1899  in insertion_resize
Objects/dictobject.c:3500  in dict_dict_fromkeys
Objects/dictobject.c:3524  in dict_set_fromkeys
...

$ ctxrank map --tokens 120 --query "dict resize"
Lib/multiprocessing/dummy/__init__.py:
⋮
│dict = dict
⋮
Objects/dictobject.c:
⋮
│static int dictresize(PyDictObject *mp, uint8_t log_newsize, int unicode);
⋮
│static int
│dictresize(PyDictObject *mp,
│           uint8_t log2_newsize, int unicode)
⋮
```

## Install

```bash
uvx ctxrank            # run without installing (Python wheel with the binary inside)
pip install ctxrank    # or install it
cargo install ctxrank  # or build from source
```

Prebuilt binaries for Windows, macOS and Linux (x86_64 and arm64) are on the releases page.

## Use it from the command line

| Command | What it does |
|---|---|
| `ctxrank` / `ctxrank map --tokens 2048` | Ranked map of the whole repo |
| `ctxrank map --focus src/a.rs src/b.rs` | Rank what those files depend on; leave the files themselves out |
| `ctxrank map --query "retry logic for uploads"` | Boost identifiers and paths that match the words |
| `ctxrank sym parse_args` | Where a symbol is defined (`--fuzzy` for substrings) |
| `ctxrank callers parse_args` | Every reference, with the function it sits in |
| `ctxrank callees parse_args` | What a function calls, and where those are defined |
| `ctxrank related src/a.rs --query "auth"` | Files ranked by relevance to files and/or words |
| `ctxrank stats --dump-tokens` | Files, symbols, languages, and what a whole-repo dump would cost |

Every command takes `--json`. `-C DIR` picks the repo, `-v` prints timings.

## Use it from an agent (MCP)

`ctxrank mcp` is a stdio MCP server with five tools: `repo_map`, `find_symbol`, `callers`,
`callees` and `related_files`. The index stays in memory and only changed files are reparsed
between calls.

**Claude Code**

```bash
claude mcp add ctxrank -- uvx ctxrank mcp
```

**Cursor** (`.cursor/mcp.json` in the project, or `~/.cursor/mcp.json`)

```json
{
  "mcpServers": {
    "ctxrank": { "command": "uvx", "args": ["ctxrank", "mcp"] }
  }
}
```

**Codex CLI** (`~/.codex/config.toml`)

```toml
[mcp_servers.ctxrank]
command = "uvx"
args = ["ctxrank", "mcp"]
```

If you installed the binary directly, use `"command": "ctxrank", "args": ["mcp"]` instead.

## Use it from Python

```python
import ctxrank

print(ctxrank.repo_map(".", tokens=1024, query="token refresh"))
ctxrank.find_symbol("RepoMap", root=".")  # list of dicts
```

## How it works

1. **Walk.** The `ignore` crate (the walker ripgrep uses) lists files, honouring `.gitignore`,
   `.ignore` and `.ctxrankignore`. Files over 1 MB are skipped as generated.
2. **Tag.** Each file is parsed with its tree-sitter grammar and the grammar's own `tags.scm`
   query, the same one GitHub's code navigation uses, in parallel with rayon. For C and C++ I add
   calls, type uses and macro references, because the upstream C query only has definitions.
3. **Cache.** Tags go into one binary cache file per repo in your OS cache directory. On the next
   run only files whose size or mtime changed are parsed again.
4. **Rank.** Files are nodes. A reference in file A to a name defined in file B is an edge A to B,
   weighted the way aider weights it (long snake/camel names count more, private `_names` and
   names defined in many places count less, sqrt of the reference count). PageRank runs with a
   personalisation vector built from `--focus`, `--mention` and `--query`, and each file's rank is
   shared out over the definitions its edges point at.
5. **Fit.** A binary search picks how many top definitions fit the token budget, then each file is
   printed with its signatures and the enclosing class or impl lines, with `⋮` for skipped code.

Languages: Rust, Python, JavaScript, TypeScript and TSX, Go, C, C++, Java, C#, Ruby, PHP, Swift, Kotlin, Scala, Lua, Elixir, OCaml, Haskell, Zig and Bash.

## Benchmarks

All numbers are from one laptop: AMD Ryzen 9 8940HX (16 cores), 32 GB RAM, NVMe SSD, Windows 11,
measured with hyperfine 2.0 (3 cold runs, 10 warm runs). Each run prints a 2,048-token map.
"Cold" means no cache, so every file is parsed. "One file changed" touches one source file
before each run, so ctxrank reparses that file and reloads the rest from the cache.

| Repo (commit) | Files indexed | ctxrank cold | ctxrank, one file changed | ctxrank warm | aider repo map, cold |
|---|---:|---:|---:|---:|---:|
| CPython (`eeafc73`) | 3,541 | 1.19 s | 0.19 s | 0.16 s | 88.4 s |
| rust-lang/rust (`ae36fe22c`) | 39,986 | 1.82 s | 0.36 s | 0.32 s | crashed after 23 s |
| Linux (`6628e7d4c`) | 67,204 | 12.4 s | 3.94 s | 3.21 s | not done after 30 min |

aider is version 0.86.2, installed from PyPI with `uv pip install aider-chat`, and timed with
`bench/aider_map.py`, which calls its `RepoMap.get_repo_map` the way aider does. On rust-lang/rust
it stopped with `tree_sitter.QueryError: Invalid node type ... module` from one of its bundled tag
queries. On Linux I stopped it at the 30 minute cap, with 81,030 of 97,348 files scanned. So the
only head-to-head time I have is CPython, where ctxrank is 74 times faster cold. aider also caches
its tags, so its warm runs are quicker than its cold one; I did not time those.

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="assets/tokens-dark.svg">
  <img alt="Tokens for a whole-repo text dump against an 8k ctxrank map" src="assets/tokens-light.svg">
</picture>

Token counts use o200k_base (the GPT-4o tokenizer) through `--features exact-tokens`. The "whole
repo" figure counts every indexed file with a path header and code fences, the way repo dump tools
print it.

| Repo | Whole repo as text | 2k map | 8k map |
|---|---:|---:|---:|
| CPython | 20,563,405 | 1,826 | 7,447 |
| rust-lang/rust | 35,323,765 | 1,966 | 8,289 |
| Linux | 331,800,357 | 2,041 | 7,921 |

The 8k map of rust-lang/rust comes out at 8,289 tokens, about 1% over its 8,192 budget, because the
default build estimates tokens from byte counts. Build with `--features exact-tokens` if you need
the budget to be exact.

Reproduce with `bench/run.sh` (needs hyperfine). Raw hyperfine JSON is in `bench/results/`.

## Limits

- Names are matched as text, with no type information. Two unrelated functions called `init` are
  the same identifier. As in aider, names defined in more than 5 files count for less. I also
  ignore names defined in more than 200 files.
- tree-sitter parses C without running the preprocessor, so code hidden behind macros can be
  missed or misread, and kernel-style headers full of `#define`s rank high.
- On the Linux kernel a warm run still takes 3.2 seconds. Most of it is loading an 835 MB cache
  (4 million definitions with their signature text) and rebuilding the symbol table. Smaller
  repos stay well under half a second. Making the cache smaller is the next thing I want to fix.
- `--query` matches words against identifiers and paths (with snake_case and camelCase split and
  crude stemming). There are no embeddings, so a query has to share words with the code.
- I have only benchmarked on Windows so far, so I have no timings from Linux or macOS.
- Map quality is judged by eye. I have not run an evaluation of whether agents finish tasks
  faster with it.

## Credits

The repo-map idea, its weighting heuristics and the output format are from
[aider](https://github.com/Aider-AI/aider) by Paul Gauthier (Apache-2.0). ctxrank is a separate
implementation in Rust and contains no aider code. Tags queries come from the tree-sitter grammar
projects (MIT). See NOTICE.

## Licence

Apache-2.0.