# repopack

Give an LLM the context of a repository — **without sending the whole project over the
network**. Two ways to do it, one tool, zero runtime dependencies (standard library only):

- **`pack`** — squeeze a repository (**any language**) into a single JSON file you can hand
  to a model, under a token ceiling you choose.
- **`serve`** — index your repositories locally and expose them over **MCP**, so a model
  pulls only the snippets a task needs, from all your projects, through one server.

## Install

```bash
pip install -e .           # or just run: python -m repopack ...
```

Python 3.9+. No runtime dependencies — that is the point: it works on a bare interpreter.

## Pack a repository

```bash
# Full pack of the current repository (any path works too)
repopack pack . -o context.json --gzip

# Incremental: only what changed since the last pack
repopack pack . --delta -o delta.json --gzip

# Restore on the other end
repopack unpack context.json -d ./restored
```

`--budget` is a **ceiling on the whole pack**, in estimated tokens — `tree`, `stats` and the
per-file metadata count against it too, not just file content. The command prints how much of
the budget went to content and how much to structure.

### How the context is organized

What is left after the structural floor is spent in **three layers**, in priority order
(README/docs → configs/manifests → entrypoints → the rest, boosted by git churn):

1. **`full`** — complete content, secrets redacted, up to ~72% of what is left
2. **`skeleton`** — imports, signatures and docstrings: the interface survives, the bodies do
   not. Measured on this codebase it keeps ~22% of the original characters (13–29% per file),
   so treat it as "a fifth of the cost", not a tenth
3. **metadata** — path, size, sha256, language (`content: null` plus a reason)

Besides the files, a pack carries `tree` (every path), `stats`, `git.recent_commits`, `todos`
(harvested TODO/FIXME/HACK) and, in `--delta` mode, `deleted_since_last_pack`.

Byte-identical content is packed **once**. The other copies become pointers
(`content: null`, `reason: duplicate_of`, plus the path that carries it) — their paths stay in
the `tree`, and `unpack` resolves the pointer so every file is still restored. The highest
ranked copy is the one that keeps the content. Measured on a repository with a `SKILL.md`
copied into three places: duplicated content went from 0.89% to 0.00%, 8,982 characters freed
for unique context.

In a large repository the structural floor alone can exceed a small budget — 4,212 files cost
about 252k tokens in `tree` and metadata. When that happens the pack is generated **without
file content** and the tool says so on stderr, naming the floor, instead of quietly returning
26× what you asked for. Trimming the `tree` to fit would hide which files exist, which is
exactly what the pack promises to list.

## Serve your repositories over MCP

One server, many repositories — local and remote — instead of one MCP per project.

```bash
repopack repo add .                                  # a local repository
repopack repo add https://github.com/user/project    # or a remote (shallow clone, cached)
repopack index                                       # ingest; the second run is a no-op
repopack serve                                       # MCP at http://127.0.0.1:7777/mcp
```

Running `repopack serve` inside a git repository registers **and indexes** it
automatically (`--no-auto-register` opts out). Only repositories that were never indexed
are ingested at startup, so restarting the server is not a full re-index.

A repository that is registered but never indexed answers nothing, so it is never silent
about it: `repo_stats` counts it under `unindexed`, and a search that comes back empty
names it and tells you to run `repopack index`.

### Tools exposed

| Tool | What it does |
|------|--------------|
| `search_context(query, repo?, budget_tokens?)` | Finds snippets across every repository (or one, via `repo`). Each snippet comes back with its repository, `path:lines` and how old that index is. Respects a token ceiling (default 4000) and reports how many relevant snippets did **not** fit. |
| `get_file(repo, path, start?, end?)` | Reads a file, optionally a line range. |
| `get_tree(repo?)` | Known paths. |
| `list_repos()` | What is being served, when each was indexed, and the last refresh error. |
| `repo_stats()` | Repositories, files, chunks, symbols, how many are failing to refresh, how many are registered but **not indexed**, and the **code-graph health**: edges by kind, average degree, isolated files, and whether the `[ast]` engine is active. |

Protocol: revision **2026-07-28** (stateless), with a compatibility shim for clients from the
`initialize` era (2025-03-26 … 2025-11-25).

### The code graph

`repopack index` builds a graph of the repository alongside the text index, and search
walks it (Personalized PageRank seeded by the lexical hits). Four kinds of edge, each
carrying a different signal:

| edge | where it comes from | note |
|---|---|---|
| `import` | declared dependencies | precise, cheap, and scarce |
| `call` | who calls whom | densest signal, **requires the `[ast]` extra** |
| `sibling` | files in the same directory | weak signal that keeps docs and configs connected |
| `cochange` | files changed in the same commits | the only signal from history; empty on a shallow clone |

Whether the walk helps depends on the graph being **dense**: the ICLR'26 GraphRAG
benchmark finds graph retrieval losing to plain RAG and names sparse, fragmented graphs
as the cause. So `repo_stats` reports the diagnosis, not just a total — measured here:

| repository | files | edges | avg degree | isolated |
|---|---|---|---|---|
| repopack itself | 77 | 913 | 23.7 | 0% |
| a 3-repository, 4,458-file workspace | 4,458 | 29,590 | 13.3 | 8.3% |

(A snapshot: these move as the repositories do. Run `repo_stats` for yours.)

Both are an order of magnitude denser than the sparse profile the paper blames, which is
the reason the walk is switched on by default here. Without the `[ast]` extra the `call`
edges disappear — the densest kind — so check `graph.ast` in `repo_stats` before reading
a thin graph as "this repository has no structure".

### Automatic refresh

Remotes are refreshed in the background (`--refresh-interval`, default 900s, jittered;
`--no-auto-refresh` opts out). When a `fetch` fails the server **keeps serving**, and the
failure shows up in `list_repos` and `repo_stats` — not only in the log. Stale context served
as if it were fresh is the failure mode that visibility exists to prevent.

## Security

- **Binds to `127.0.0.1` by default.** Leaving loopback (`--host 0.0.0.0`) **requires** a
  bearer token: if you do not pass one, the server generates it and prints it once. There is no
  "network without authentication" mode — the index holds the source code of every repository
  in the workspace.
- The `Origin` header is validated on every request (403 when invalid), against DNS rebinding.
- **Redaction happens at ingest**, not on the way out: what is secret never enters the database,
  so there is no path for it to leave. `.env` and its variants stay out by default
  (`--include-env` opts in, and still redacts).
- `get_file` never serves anything outside the repository root — the same guard `unpack` uses.
- Credentials for private repositories come from your **environment** (ssh-agent, git helper);
  repopack stores no token and never prompts for a password (`GIT_TERMINAL_PROMPT=0`).
- Secrets are redacted by pattern (API keys, GitHub/Slack tokens, JWT, private keys, `.env`
  variables, URLs with credentials) **plus a Shannon-entropy sweep**.
- `unpack` audits each restored file against the `sha256` the pack carries, and warns on
  mismatch. The check is conclusive only for content that is intact by definition — complete,
  not truncated, not redacted — because a redacted file diverges from its hash by construction.

> Redaction is a **net, not a guarantee**: it covers variable names containing
> SECRET/TOKEN/KEY/PASSWORD and known key formats, so a line like `MY_PLAIN=value123` gets
> through. That is why the default for `.env` is to not pack the content at all, rather than
> trusting the filter.

## What this is not

- **`unpack` is not a backup tool.** It restores only content that is intact, refuses degraded
  content by default (naming each file and why), and never overwrites an existing file unless
  you pass `--force`. A pack is a context artifact, not a copy of your repository.
- **`skeleton` mode is lossy by design.** It keeps imports, signatures and docstrings and drops
  function bodies. Restoring a skeleton over a real file destroys the original and gives back
  something that will not even parse — which is why it takes `--force` and says so.
- **Skeleton extraction is regex, not a real AST.** It is good enough to keep an interface
  readable and wrong often enough that you should not treat it as a parser. The `[ast]` extra
  does bring real tree-sitter parsing, but it feeds the *code graph*, not `pack --skeleton`:
  installing it makes search better and leaves skeleton output byte-identical.

## Honest limitations

- **Search is lexical (BM25) fused with a code-graph walk, and still has no embeddings.**
  Measured on this repository with `repopack eval` (18 questions, `--no-graph` reproduces the
  first row):

  | ranking | recall@5 | MRR | identifier | conceptual |
  |---|---|---|---|---|
  | BM25 only | 83.3% | 0.596 | 100% / 0.875 | 70% / 0.373 |
  | BM25 + graph (default) | **94.4%** | **0.625** | 100% / 0.823 | **90%** / 0.467 |

  Read those absolute numbers with the ruler's granularity in mind: 18 questions means one
  question is worth 5.6 points, and the corpus being measured is *this repository*, which
  changes every commit. Measured: adding two files whose docstrings talk about budgets and
  ceilings moved recall between 88.9% and 94.4% with no ranking change at all. What the
  gate enforces is therefore a floor one question below the measurement, and the
  *comparison* between the two rows — both measured on the same corpus in the same run,
  which is the part that does not drift.

  The graph costs about **+212 ms per query on a 4,212-file repository** (68 ms → 280 ms; on a
  small one it is 4.5 ms → 7.7 ms), and it costs a little *position* on identifier queries —
  which is why the lexical top hit is pinned. **What it did not fix:** a test file that mentions
  the concept still outranks the implementation for some conceptual questions. *"Where is the
  budget enforced?"* now returns `packer.py` in the top-3, but behind `test_rag_search.py`.
  A dense ranker is still absent.
- **Diversification (MMR) is implemented and deliberately switched off**, because it did not
  win. Measured across λ from 1.0 down to 0.6, the one apparent gain does not survive a change
  of `k` (at k=3 a different λ wins; at k=10 every λ ties at 100% recall while MRR falls
  monotonically as λ drops) — a peak that moves is one question oscillating, not an effect. The
  premise did not hold either: across a 3-repository, 14,606-chunk workspace the average
  Jaccard similarity between returned snippets is 0.04–0.08, and the only identical pairs are
  3-token fixtures. Switching it on costs 4 ms → 97 ms per query. `select.LAMBDA` re-opens the
  decision, and a test refuses the change unless the ruler backs it.
- **Cross-repo search gets noisy** with many repositories and a generic query. What holds today
  is the token ceiling and per-repo attribution.
- **IDF is measured across the whole workspace.** A term that is common in one project and rare
  in the others weighs differently than it would in an isolated search.
- The local gate (`scripts/check.sh`) runs on the Python versions **installed on your machine**,
  which is usually fewer than the matrix. The full 3.9–3.14 matrix is exercised by CI on every
  push; the three lists (classifiers, CI matrix, local gate) are cross-checked by the test
  suite, so they cannot drift apart in silence.

The index — and the `--delta` state — live in `~/.cache/repopack/` (honors `XDG_CACHE_HOME`;
`REPOPACK_HOME` overrides), **never inside the repositories you scan**. A `--delta` state left
in a repository root by an older version is still read, so your delta survives the upgrade; it
is not deleted, because removing a file from someone's repository is more invasive than having
created it.

## Configuration

Create a `.repopackrc` (JSON) at the repository root — see `.repopackrc.example`.
Precedence: defaults < `.repopackrc` < CLI flags.

## Roadmap

- Ranking priors by file kind, so a test file stops taking the first slot from the
  implementation it tests — the one conceptual defect the graph did not fix
- Diversification (MMR) and near-duplicate suppression across repositories
- Optional dense ranker behind a `[rag]` extra, for conceptual queries
- Task-conditioned packing (`--task`)

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). The one rule that is not negotiable: **zero runtime
dependencies**.

## License

MIT — see [LICENSE](LICENSE).