distil

🚀 Use it now

One command sets you up and tells you what to do next:

pipx install distil-llm
distil onboard      # detects your agent + billing, wires the status line, prints a guided tour

It detects your environment (Claude Code · Codex · Gemini CLI; metered vs subscription) and hands you the exact commands. Or wrap your agent directly — no config, no code change:

# Claude Code on a metered API key — saves real $$:
distil wrap --expand -- claude

# Claude Code on a Pro/Max subscription — flat-rate, ToS-safe (trims context, not $):
distil wrap --lossless-only -- claude

# Codex, Gemini CLI, aider — same pattern; env var auto-selected per agent:
distil wrap --expand -- codex     # → OPENAI_BASE_URL
distil wrap --expand -- gemini    # → GOOGLE_GEMINI_BASE_URL
distil wrap --expand -- aider     # → OPENAI_BASE_URL

Each recognized agent (claude / codex / gemini / aider) auto-selects the right env var and upstream — no --env-var or --upstream flag needed. Prints preset: <agent> detected → <VAR> on start. Explicit flags always win.

Tired of typing distil wrap every time? Make it the default — once:

distil default            # adds a managed shell alias so `claude` always routes through distil
distil default --undo     # remove it anytime (backed up before any change)

It detects your shell (zsh / bash / fish / PowerShell) and billing mode, writes the right line to the rc file your shell actually reads, and tells you what it detected. Want every SDK covered (not just the agent you type)? distil default --always-on runs a persistent proxy service — powerful, but it's a daemon you keep alive.

Then watch genuine savings from your traffic — measured, not estimated:

distil leaderboard          # cumulative tokens + $ saved, from the local ledger
distil dashboard            # live terminal TUI — token-trim + decision-equiv bars, Ctrl-C to exit
distil dissect             # per-session deep-dive: savings, digest inventory, anomalies (--html/--serve)

Validate it on your traffic. --shadow runs a fraction of requests twice (compressed and full) and compares the agent's chosen next action:

distil wrap --shadow 0.1 -- claude   # wrap + shadow 10% of requests
distil shadow-stats                  # live decision-equivalence rate

Honest scope: that's next-action equivalence — a proxy, not task success (E7 shows it doesn't fully transfer under aggressive lossy compression). Distil fails safe to full context.

Will it save money? Only on metered billing (API key) — fewer tokens, fewer dollars. On a flat-rate subscription it trims context + latency, not the bill. Coding agents: short sessions ~7%, big wins on long, many-turn sessions the model never re-reads.

Related MCP server: PlanckBot

💡 Why Distil is different

You don't need byte-equivalence — you need decision-equivalence: your agent taking the same actions with compressed context. That's measurable and certifiable.

• Certified, not estimated — a strategy ships only if a non-inferiority test passes; can't certify → full context.

• Certified end-to-end, too — distil certify-trajectories bounds how many solvable tasks compression can cost (no other compressor certifies either level).

• Reversible, not lossy — digests behind a handle, keeps the original, hands the agent a distil_expand tool. Compress fearlessly.

• Keeps the answer, folds the noise — a per-content-type keep policy pins each kind's load-bearing lines (a log's pass/fail verdict, a traceback's frames, a diff's hunk headers); repeated near-identical error spam is deduped, and on a green run dedup tightens further since that noise didn't fail anything.

• Query-aware — keeps the line you're actually asking about — distil is a proxy, so it sees the agent's intent (its tool_use args + latest ask) in the same request as the output. The line matching what you searched for (a grep hit, a config value, a SHA) is pinned even in arbitrary output — additively, so reversibility and the certificate are untouched. No post-hoc compressor has that query/output pairing. It also goes semantic, and always-on: a zero-dependency bridge — morphology, a curated technical synonym map, and char-trigram fuzz — pins lines that answer the query without sharing a word with it. Ask "the retry limit?" and it keeps max_attempts = 5; ask "the connection timeout?" and it keeps deadline_ms. Two more layers grow from your own traffic, never from a shipped blob: associations distil learns from its content-free expand flywheel (hashed pairs, --expand sessions), and a learned relevance model that is promoted only after its held-out recall beats the lexical baseline on your labels — until promotion, the lexical + bridge layers are exactly what runs. An optional distributional-vector table can be supplied too (pure-Python cosine; none ships). Every layer is additive — it can only widen keeps, so reversibility and the certificate are untouched — and it needs no embeddings or model to work.

• Lossless even on a flat-rate plan — subscription/lossless mode isn't just verbatim: it minifies JSON, collapses duplicate runs, and folds tabular tool output into a compact self-describing table (~70–79% smaller, ToS-safe, no lossy digest). Recent tool outputs stay byte-exact.

• See exactly what happened — distil dissect turns a wrap session into a report: savings by model/mechanism, the digest inventory, billed-usage calibration, latency by path, and a worth-your-attention anomaly list that catches silent failures automatically.

• Compounds on outcomes — expansions and matched failures teach the policy what to protect (signatures only, never content) — always more conservative.

• Streams like it isn't there — SSE relays chunk-by-chunk; TTFT preserved.

Fidelity tiers: lossless (--verbatim) · reversible (byte-recoverable on demand — default) · lossy (every other tool). Only Distil certifies the reversible tier (Headroom ships an uncertified retrieve; Distil's recovery is agent-facing — the model expands mid-task — and gated by the decision-equivalence certificate).

⚡ Prove the numbers yourself — no API key

Don't take the table above on faith. distil bench re-certifies savings and decision-equivalence on a bundled 7-domain corpus, offline, in seconds — the same gate that runs in CI. How we evaluate — and why a compression ratio without a task-success delta is meaningless — is written up in docs/EVALUATION.md, including our own negative result:

uvx --from distil-llm distil bench      # certify savings + quality across 7 domains, in seconds
distil verify                           # byte-fidelity: every compression is exactly reversible
distil validate                         # adversarial real-path gate: invariants on hostile inputs

Three gates, all in CI: bench (non-inferiority on the corpus), verify (byte-fidelity), and validate — which drives the compressor against adversarial inputs (huge/unicode/nested/malformed/marker-injection/secret-looking) and asserts reversibility, reject-if-bigger, recency-exactness, fail-open, and content-free telemetry hold on every one. That last gate exists because a green unit suite kept coexisting with real-traffic bugs; validate is the adversarial layer that catches them.

To be precise about what each layer proves: the per-commit gates grade decision-equivalence with an offline deterministic oracle over the committed corpus (fast, free, runs on every push — but synthetic). A nightly live-cert job re-certifies the same trajectories against a real model (distil certify --runner anthropic), budget-capped with a hard --max-live-calls ceiling so an unattended run can never spend silently. The empirical results above (SWE-bench n=500, live head-to-head n=200) were graded by real models; the per-commit badge alone doesn't claim that.

domain            trajectory                $ saved   distil   aggr  pruned
---------------------------------------------------------------------------
ops/sre           sre-disk-incident           33.1%     PASS   FAIL     615
coding            coding-bugfix               28.7%     PASS   FAIL     736
support           support-refund              32.6%     PASS   FAIL     765
research          research-synthesis          25.7%     PASS   FAIL     809
data-analysis     data-analysis-sql           18.1%     PASS   FAIL     965
devops            devops-rollback             25.0%     PASS   FAIL     857
finance           finance-reconcile           29.1%     PASS   FAIL    1014
---------------------------------------------------------------------------
aggregate: distil cuts $0.14212 -> $0.10402 (26.8% cheaper) reversibly; 5761 tokens causally prunable.
GATE: PASS — every trajectory certified non-inferior; aggressive rejected on all.

Why trust the number? Token-savings numbers are easy to fake — measure quality at low compression, advertise savings at high compression. Distil refuses that: accuracy and compression are measured on the same trajectories, and a strategy that can't pass non-inferiority doesn't ship.

distil certify --strategy distil       # VERDICT: PASS  (100% decision-equivalence)
distil certify --strategy aggressive   # VERDICT: FAIL  (mean diff −1.0, blocked)

distil eval plots the certified compression frontier — a savings-vs-quality curve where every point carries its certification verdict, locating the cliff past which lossy compression drops decisions. The artifact no competitor publishes: benchmark.html.

📊 The proof

Three results, all reproducible, all published with caveats:

• Live head-to-head vs real llmlingua / headroom-ai (graded by claude-opus-4-8): 83.2% savings at 0% decision-change, ~1,000× faster (no ML model loaded vs. competitors' local transformer inference). The live proxy behavior is pinned to the certified strategy by tests/test_live_certified_equivalence.py; the one reviewed delta is a recency carve-out that keeps the last few tool-result turns verbatim (an agent needs its freshest output byte-exact). → benchmark

• E7 (SWE-bench Verified): aggressive lossy compression craters task success (52% → 16%) — a per-step certificate doesn't transfer to multi-turn. The reversible tier survives (56% vs 52%). We publish it because it's true. → E7

• E8–E14 (500-instance agent): the reversible tier is the only compressor non-inferior to full context, generalizes across 5 models / 3 vendors, and the newest digest matches full within noise (42.0% vs 39.2%). → E8–E14

Full methodology, McNemar tests, per-instance data: docs/PAPER.md · PDF.

📡 See it working

Measured on your traffic, never estimated, nothing leaves your machine:

• Per request: x-distil-* response headers (tokens-saved, mode, compressible-tokens, expanded).

• Per machine: distil leaderboard (--html for a page).

• Shadow mode: distil proxy --shadow 0.05 reports the live decision-change rate — streaming-aware.

• Org-wide: distil proxy sidecar + set ANTHROPIC_BASE_URL once; every client routes through it.

Dashboard, status-line plugin, federated leaderboard: Deploy & observability.

🔌 Works with every SDK

One proxy. Point any base_url-honoring client at it — Python, TypeScript, any language — and get cache-aware reversible compression with no code change.

distil proxy --upstream https://api.anthropic.com   # localhost:8788

Prefer in-process? Wrap the client directly — still no call-site change:

from distil.adapters.anthropic import wrap
client = wrap(anthropic.Anthropic())   # compresses the request, keeps the cache warm

(OpenAI — Chat Completions and Responses API — and Gemini route through the proxy: distil wrap -- codex, or point OPENAI_BASE_URL at it. An in-process client wrap exists for the Anthropic SDK only.)

Framework hooks (no proxy, no network hop) — for agent frameworks that own the message list, compress it where it lives:

📦 Install your way

New here? pipx install distil-llm, then distil onboard — it sets you up and guides you (see Use it now). Want to see it prove itself first instead? distil bench runs the certified gate in ~10s, no API key. The matrix below is for picking an install format — everything in it is an alternative, not a requirement.

⚠️ The one gotcha — the name. The PyPI package is distil-llm but the command is distil (the bare name was taken). So pipx install distil-llm → run distil …. pip install distil installs something else.

🔧 Seeing Could not find a version that satisfies the requirement distil-llm (from versions: none)? The package is on PyPI — that error means your pip/pipx is on a Python older than the package's floor, so pip filters every release out. Distil now supports Python 3.9+ (the version macOS ships), so a current install just works; if you still hit this on a very old Python, let uv provision one for you: uvx --python 3.12 --from distil-llm distil bench (or uv tool install --python 3.12 distil-llm). Check yours with python3 --version.

🔧 Got an old version (e.g. 0.25.1) instead of the latest? Public PyPI always serves the newest (pip index versions distil-llm lists them). If you got an older one, your pip/pipx is not resolving against public PyPI — almost always a stale internal mirror (Artifactory / CodeArtifact / Nexus that hasn't synced the latest yet — common right after a release) or a <1.0 version pin in a constraints file / pip.conf. Diagnose and fix:

pip index versions distil-llm     # stops at an old version? → your index/mirror is stale
pip config list ; env | grep -i pip   # look for an index-url or PIP_CONSTRAINT pin
# unblock now — force public PyPI:
pipx install --pip-args="--index-url https://pypi.org/simple/" distil-llm
# (or, if you must use the mirror, ask your platform team to sync distil-llm; it exists upstream)

The import package and CLI are distil; the PyPI distribution is distil-llm (the bare name was taken — so uvx/pip must reference distil-llm, not distil). Distil is a CLI: install it isolated (pipx/uv/brew/Docker), because modern macOS/Linux block system-wide pip install (PEP 668). Node / any language: point your SDK's base_url at distil proxy, or use distil wrap -- <agent> — no Distil-specific package needed.

🧰 Cheat-sheet

Basics are in Use it now and Works with every SDK. Beyond that:

Status line — one pattern in every state: distil · <live> · total ▼<lifetime>.

state	you see	means
saving	distil · ⬢ digest · ▼12.0K · 40% smaller · $0.31 · total ▼27.0M · de 99%	compressing (mode chip: ⬢ digest · ◇ lossless · ▪ verbatim; de = decision-equivalence)
watching	distil · ✓ on · waiting for a large read · total ▼27.0M	on, but no large content yet — savings come from big file/command output
idle	distil · ✓ on · total ▼27.0M	set up and on, no recent traffic
not routed	distil · off — session not routed · total ▼27.0M	this session's requests go straight to the provider — start it with distil wrap (or the always-on env) to compress
bypassing	distil · ⚠ wrapped, agent bypassing proxy · total ▼27.0M	the wrap is up but zero requests reached its proxy in 3+ minutes — the agent pinned its own endpoint. Fix: restart the wrap. Seen mostly with claude.ai-subscription (OAuth) sessions; routing those through a custom base URL is undocumented upstream, and a session occasionally ignores it. scripts/soak-report.sh captures evidence if it persists

The de segment is live decision-equivalence evidence: a ✓/⚠/✗ rate once 50 A/B samples + 30 A/A samples accrue (A/B = compressed-vs-original; A/A = same request replayed against itself — the sampling-noise baseline), de n/50 while collecting. Shadow sampling is on by default at 2% (--shadow 0 disables; --shadow 1.0 samples every request — proves equivalence in minutes at ~3× token cost, then drop back to the default 2%).

Measured: In live validation (signature v3 / 1.13.0), distil preserved the agent's next decision on 100% of 116 sampled production requests (0 changes); temperature-0 A/A self-agreement of 31/31 confirms this is compression fidelity, not sampling noise. Validated result — not a guarantee for all workloads.

▼ = tokens saved · total = lifetime · de = decision-equivalence (verdict once 50 A/B + 30 A/A shadow samples accrue). Sharing the line with git/cwd/model? DISTIL_STATUSLINE=minimal → distil ▼7.8K · 27M total. On a flat-rate subscription, dollars are notional and auto-hidden (DISTIL_SUBSCRIPTION=0/1).

Compression modes — in plain English

You usually don't need to pick. distil onboard detects your billing and sets the right mode for you — it writes it into your setup so every session just works. Pass a flag to override for a specific session.

• digest (the default) — Distil shortens long things (big files, command output, past steps) into short summaries, and can pull back the full original the moment the AI needs it. You save the most, and nothing is truly gone — originals are kept and restored automatically. Most people should just use this.

• expand — Same shortening as digest, but Distil also gives the AI a "show me the full version" button it can press on its own. Best when the AI runs for a long time autonomously (e.g. long coding sessions). Picked automatically if you pay per use (API key).

• lossless-only (a.k.a. --safe) — The cautious setting: Distil only trims things it can rebuild perfectly (like extra blank space), and never summarizes. You save less, but there's zero chance of losing any detail. Picked automatically on a flat monthly subscription.

• verbatim — The lightest touch: just tidies formatting, changes nothing else. Almost no savings. Use it when you want to see or audit exactly what's being sent.

For the technical breakdown:

Subscription users should not force --expand; it crosses the lossless safety boundary. Coding re-reads? Add --session-delta either way.

🧠 How it works

Two techniques carry most of the win — they target where the money actually is in an agent loop, not where it looks like it is.

① Cache-aware compression — the dominant lever

You re-send the growing context every step. With prompt caching a cache read is ~10× cheaper than fresh input, so the real cost is cache misses, not context size. Distil keeps the prefix byte-stable (schema canonicalization + lifting volatile fields like timestamps/UUIDs out of the prefix) and compresses only the volatile tail.

Naive recompression sends fewer tokens yet costs more than not compressing at all, because it rewrites the cached prefix every turn. Distil doesn't — that's the whole game most tools miss.

② Causal / counterfactual pruning — the discovery engine

The eval isn't a ruler bolted on the side; it's a discovery engine. Remove a context block, replay, did any decision change? Blocks that never change a decision are provably free to drop.

distil prune
# doc-0   PRUNE (causally inert)     # speculative retrieval, never cited
# obs-0   keep (changed a decision)  # carries the decision-driving signal

🎓 The certificate (DERC)

The gate answers "is this strategy non-inferior on my corpus?". The Decision-Equivalence Risk Certificate answers the operational one: "for a risk budget I choose (say ≤5% decision-change), how hard can I compress with a guarantee that holds on my real traffic?"

distil conformal --corpus ./mycorpus --alpha 0.05 --delta 0.05
# ✔ CERTIFIED 'lossless' → 57.4% savings; decision-change ≤ 5.0% at 95% confidence (Learn-Then-Test)

It's conformal risk control (Learn-Then-Test / CRC — distribution-free, finite-sample), not a heuristic threshold. The one load-bearing caveat: the guarantee requires exchangeability (calibration traffic ≈ live traffic) and is marginal over that distribution — recalibrate on drift. Full theory + citations: Concepts · docs/PAPER.md.

🏔 The trajectory-level certificate

DERC certifies the step; this certifies the task. Our E7 experiment — and the 2024–26 agent-compression literature — shows per-step fidelity can pass while end-to-end success collapses, so distil also certifies the level users actually feel: run your eval suite twice (full context vs compressed), feed the matched outcomes in, and get a distribution-free bound on how many solvable tasks compression may cost you:

distil certify-trajectories outcomes.jsonl --alpha 0.05 --delta 0.05
# each line: {"task_id": "...", "full_success": true, "compressed_success": true}
# → With confidence 95%, compression degrades at most 5.0% of tasks the full
#   context would have solved (observed 0.5% over 200 matched trajectories).

It refuses to certify on small samples, states its exchangeability assumptions in the certificate itself, and ships an anytime-valid drift monitor (trajectory_risk.drift_monitor) that tells you when live traffic has shifted enough that the certificate is stale. Matched failures also feed the outcome-guided policy (distil.compress.guideline): content classes that break tasks when digested get protected byte-exact, automatically.

🧩 What's inside

40+ shipped capabilities, all real (no stubs): the cache-aware cost engine, causal pruning, the TOST gate + conformal certificate, the proxy + Anthropic/OpenAI/Gemini first-class adapters (Chat Completions, Responses API, and Gemini generateContent), an MCP server, LiteLLM/LangChain/LangGraph hooks, per-agent wrap presets, the Proof Ledger end-of-session printout, the multi-tenant gateway with issued keys and rate limits, encrypt-at-rest for the restore store, learned keep-models, output compression, and an optional Rust hot-path core (build-from-source via maturin; published wheels run the pure-Python engine, same API) — with zero runtime dependencies in the core.

Full module-by-module map: Architecture · Techniques · CLI reference.

🔒 Security & deployment

• Localhost-only by default — the proxy binds 127.0.0.1 and forwards only to the single configured upstream (no SSRF).

• No secret/body logging — request bodies and credentials are never logged.

• Auth-mode gating — a detected subscription/OAuth session auto-selects --lossless-only (Tier-0 verbatim: no Tier-1 digest stubs, no tool injection — provider-ToS-safe); distil wrap -- claude is safe by default, no flag needed. An explicit --expand opts into the recoverable digest even there (you authorized the recovery tool, so nothing is irreversibly lost — issue #28). Without an injected expand tool the agent cannot recover a stub, so --lossless-only folds directly into verbatim.

• Encrypted at rest — digest originals in ~/.distil/restore/ are encrypted with HMAC-SHA256-CTR (encrypt-then-MAC, DSTL1 header, key at chmod 0600), protecting against backup/sync leakage and cross-user reads on shared filesystems. A same-UID attacker who can read both the data files and the key file is explicitly out of scope (see THREAT_MODEL.md). Legacy plaintext files load transparently. DISTIL_NO_ENCRYPT_AT_REST=1 opts out; handles age out after DISTIL_RESTORE_TTL_DAYS (default 14). No data is forwarded upstream.

• Ops-ready — unauthenticated GET /distil/health liveness probe on every entry point (never touches the billed upstream); gateway accounting checkpoints to disk every 30 s (crash-safe, not just on graceful shutdown); DISTIL_DEBUG=1 surfaces everything the fail-open compression path swallows.

• Upgrades apply to live sessions — distil wrap supervises its proxy as a subprocess on a wrap-owned socket; when a new version lands on disk (pipx/pip upgrade) the wrap hot-swaps in a fresh worker — same port, in-flight streams finish on the old one, the agent never restarts. Health-checked with automatic rollback: a broken upgrade keeps the old worker serving. POSIX; kill -USR1 <wrap pid> forces it, DISTIL_HOT_SWAP=0 opts out. On Windows the wrap keeps the historical in-thread proxy (no seamless swap) and warns on version skew instead — upgrades there apply on the next session.

• OpenTelemetry GenAI spans (opt-in) — pip install 'distil-llm[otel]' and every proxied call emits a GenAI semantic-convention span (gen_ai.request.model, gen_ai.usage.input_tokens) plus distil's own story: distil.tokens.original vs distil.tokens.compressed, distil.compression.ratio, distil.shadow.sampled, and distil.session.id for per-session trace correlation — your existing OTel backend sees exactly what compression did to each request. Without the extra installed it's a single boolean check, zero overhead, and an OTel failure can never break the request path. No metrics endpoint exists — the LiteLLM unauthenticated-/metrics leak class of bug is structurally absent.

• Supply-chain hardening — releases carry PEP 740 Sigstore attestations (via PyPI trusted publishing) and a CycloneDX SBOM attached to every GitHub release; OpenSSF Scorecard runs weekly on main. Don't take our word for it — verify a release yourself: uvx pypi-attestations verify pypi --repository https://github.com/dshakes/distil pypi:distil_llm-1.19.0-py3-none-any.whl (or inspect the raw bundle at https://pypi.org/integrity/distil-llm/<version>/<filename>/provenance).

See Deploy & security for topologies (local sidecar, container sidecar, shared gateway) and the threat model.

✅ What we won't pretend

• Self-calibrating token counts — the offline heuristic is directionally accurate; the compression ratio is exact regardless. distil is a proxy, so it sees the provider's real usage.* on every response — it learns the systematic correction from that (content-free, no network) and calibrates the absolute counts to your model + content mix automatically. The leaderboard shows "calibrated to your billed usage (N requests, ±X%)" once enough traffic has flowed; until then it's the raw heuristic (identity, so no skew). For per-string exactness there's still --tokenizer anthropic.

• Default runner is a deterministic stand-in (offline gate with ground truth). Non-circular eval grades real agent traces with a real model — proof harness.

• Credible grading, enforced: majority-vote (single samples let grader noise look like a decision change), a same-family grader, and grading the reversible tier with its distil_expand recovery loop.

• No fabricated weights — the keep-model is a real logistic classifier (96.4% held-out accuracy, 0.98 F1; the committed metrics.json regenerates byte-identically from python -m distil.codec.learned, seed-pinned). The optional transformer codec ships no checkpoint in the package — a demo checkpoint is attached to the v0.1.0 release, and production means retraining on your own traces (distil train-transformer).

Deliberately not a platform

Distil is a compression engine with a correctness gate, not a context suite. We declined what can't go under the certificate:

What we did adopt (it survives the gate): a pluggable salience scorer to protect entities, cache-prefix observability, and framework hooks.

🎯 Both sides of the bill

Distil compresses input/context (comprehensive) and output — generation-side verbosity shaping (PAYG, measured with distil output-savings) plus a reversible output-on-re-entry digest, so verbose past answers stop costing full price as history. Details: Output & I/O.

🔬 Reproducible evaluation & the paper

Every number reproduces from the bundled corpus (distil bench, no key). The non-circular proof harness grades real agent traces with a real model (τ-bench / SWE-bench): benchmarks/PROVE.md. Compiled paper, LaTeX source, and all committed results: docs/PAPER.md · docs/paper/ · paper PDF. Step-by-step: Reproduce the Numbers →

⭐ If distil saved you tokens

A star is how the next engineer finds provable savings instead of a lossy guess — and distil stats --badge gives you a shareable badge of your own measured number to show alongside it. That badge + this repo are the whole marketing department.

🤝 Contributing

PRs welcome — see CONTRIBUTING.md. The one rule that matters: a new compression strategy must pass make gate (non-inferior on every domain, byte-reversible). No green gate, no merge. That's the whole philosophy in one sentence.

Beta program — want early access to v1.20.0 and to help close the GA gate? See docs/BETA.md.

License

Apache-2.0 · “Same potency, less volume.”