# repolens

**Turn any repository into a deterministic, token-budgeted, AST-aware context pack for LLM agents — with a hash-chained provenance row for every file it reads.**

repolens is a zero-dependency Python tool (standard library only for the core) and an MCP server. It exists for one job: give an autonomous agent *exactly* the context it can afford, in a form that is reproducible, auditable, and structurally aware of the code it describes.

A Cognis Digital flagship tool.

---

## Why these properties matter

Most "AI code context" tooling optimizes for a compression headline and produces output that is nondeterministic, unbudgeted, structurally blind, and unauditable. When that output feeds an agent that *edits your code*, four different properties are what actually matter:

1. **Reproducibility.** The same repository bytes must always yield the same pack — byte for byte. That makes packs diffable in review, cacheable in CI, and safe to reason about. If a pack changes, it is because the repository changed, not because a model or a clock did.
2. **An exact token budget.** An agent has a finite context window and a finite spend. "Roughly smaller" is not a budget. repolens fits a repository into a precise token ceiling and degrades gracefully when it must.
3. **Real structure.** Signatures, imports, and call edges are worth far more per token than raw text. A language-aware map lets the pack say *what the code is* even when there is no room to include *all of the code*.
4. **A tamper-evident record.** When something goes wrong downstream, you need to prove precisely what context was produced from which bytes. repolens emits a hash-chained provenance manifest that makes any alteration detectable.

repolens delivers all four.

---

## Features

- **Deterministic context pack.** Stable ordering plus content addressing: the same repository bytes always produce a byte-identical pack. The pack's *digest* is the head of its provenance chain, so equal digests provably mean equal inputs read in the same order.
- **Exact token budgeting with graceful degradation.** Give `--budget N` and repolens fits the repo into `N` tokens, degrading each file through `full body → signatures → names → omit` in priority order. A pluggable tokenizer abstraction ships with a deterministic heuristic estimator by default and a character-ratio estimator; real model tokenizers can be registered without touching the pipeline.
- **AST-aware maps.** Python is parsed with the standard-library `ast` for exact symbols, signatures, imports, and intra-module call edges. JavaScript/TypeScript, Go, Rust, and Java use lightweight, clearly structured heuristic extractors that report symbols, kinds, signatures, and imports — designed so a full parser can be dropped in behind the same interface later.
- **Keyless relevance ranking.** A dependency-free BM25 implementation ranks files against a free-text query so the budget is spent on what matters. An optional local-embedding ranker is a clearly marked extension point; BM25 is always available and needs no keys, downloads, or network.
- **Hash-chained provenance.** Every file read emits a SHA-256 audit row `(index, path, content_hash, size)`, and each row commits to the previous row's hash. Reordering, inserting, deleting, or altering any row breaks the chain — and `verify` reports the exact first bad row.
- **Multiple output formats.** Compact text, JSON, Markdown, and XML-tagged prompt blocks. `.gitignore`-aware discovery, and a secret-redaction pass strips common key/token formats before anything leaves the repository.
- **MCP server.** A minimal, self-contained JSON-RPC/stdio MCP server exposes a `pack` tool (and a `verify` tool). Agents request a budgeted context pack for a path or query and get the manifest back.
- **Honest benchmarks.** A benchmark harness compares repolens against a naive find + concatenate baseline on bytes, tokens, and wall-clock.

---

## Install

```bash
pip install .
# or, for development:
pip install -e ".[dev]"
```

Requires Python 3.11+. The core has **zero third-party runtime dependencies**; `pytest` is used only for tests.

---

## CLI

```bash
# Build a compact text pack that fits in 8000 tokens
python -m repolens pack . --budget 8000

# Rank against a query and emit XML-tagged prompt blocks
python -m repolens pack . --query "auth middleware" --budget 4000 --format xml

# JSON pack (includes the full provenance manifest) written to a file
python -m repolens pack . --format json -o pack.json

# Re-verify a JSON pack's provenance chain
python -m repolens verify pack.json

# Benchmark against the naive find + cat baseline
python -m repolens bench . --budget 8000

# Run the stdio MCP server
python -m repolens mcp
```

A `repolens` console entry point is installed as well, so `repolens pack .` works once installed.

### Useful flags

| Flag | Effect |
| --- | --- |
| `-b, --budget N` | Exact token budget (omit for unlimited) |
| `-q, --query STR` | BM25 relevance query |
| `-f, --format` | `text` (default), `json`, `markdown`, `xml` |
| `-t, --tokenizer` | `heuristic` (default), `char`, or `char:N` |
| `--no-gitignore` | Do not honor `.gitignore` |
| `--no-redact` | Disable secret redaction |
| `--include-binary` | Include binary files |
| `--no-manifest` | Omit the provenance manifest from output |
| `--max-file-bytes N` | Truncate file bodies larger than `N` bytes (provenance still hashes full bytes) |
| `--follow-symlinks` | Follow symlinks (off by default so a pack cannot escape the repo) |
| `--stats` | Print pack statistics to stderr |

---

## Library API

```python
from repolens import pack_repo, render

pack = pack_repo("path/to/repo", query="database pool", budget=8000)

print(render(pack, "markdown"))
print("digest:", pack.digest)
print("provenance verified:", pack.manifest.verify().ok)
for f in pack.included_files:
    print(f.relpath, f.language, f.level, f.tokens)
```

Everything is a pure data object: `pack_repo(...)` returns a `ContextPack`, and the formatters are pure functions of it.

---

## Determinism, precisely

Two runs over the same bytes produce identical output because every stage is deterministic:

- Files are discovered in sorted, forward-slash path order.
- Provenance rows are recorded in that order and chained by SHA-256.
- Ranking ties break by path; budgeting is a deterministic greedy pass.
- JSON is emitted with sorted keys and fixed separators; all formats use `\n` newlines.

The included test suite proves byte-identical output across repeated runs and across independent copies of the same content.

---

## Provenance model

Each row commits to its own fields *and* the previous row's hash:

```
row_hash = sha256("repolens-provenance-v1\n" + index + "\n" + path + "\n"
                  + content_hash + "\n" + size + "\n" + prev_hash)
```

The first row's `prev_hash` is 64 zero hex characters (the genesis sentinel). The manifest **head** — the last row's hash — is a single value that commits to the whole read. An auditor can recompute the entire chain from the raw repository bytes.

---

## Extending

- **Tokenizers.** Subclass `repolens.tokenizer.Tokenizer`, implement `count`, and call `register_tokenizer(name, factory)`.
- **Language extractors.** Implement the `Extractor` interface in `repolens.languages.base` and register it; the heuristic extractors are structured so a real parser can replace any of them without changing the rest of the system.
- **Ranking.** `embedding_rank` is the hook for an optional local-embedding ranker; BM25 remains the keyless default.

---

## Testing

```bash
python -m pytest -q
```

The suite covers determinism, token budgeting and graceful degradation, each AST/heuristic extractor, `.gitignore` semantics, binary/unicode/empty/huge-file edge cases, secret redaction, the provenance chain and its tamper cases, the four output formats, the benchmark harness, the CLI, and the MCP endpoint.

---

## License

MIT © 2026 Cognis Digital LLC. See [LICENSE](LICENSE).