tokendiet
This MCP server provides context-compression and model-routing tools for AI coding agents.
Prune noisy CLI, pytest, tsc, or build logs to a token budget while preserving error/traceback anchors.
AST-skeletonize Python source by keeping imports, signatures, docstrings, and focus symbols while eliding unfocused helper bodies.
Compress multi-turn coding-agent traces using cross-turn read deduplication, log pruning, and AST skeletonization.
Route agent turns across Local Flash, Balanced Coding, and Frontier Reasoning tiers with cascade escalation logic.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@tokendiet shrink this pytest log to 200 tokens and keep all errors"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
tokendiet
A context compressor for AI coding agents: order-preserving knapsack pruning of noisy tool logs (C++20 kernel with a pure-Python fallback), Python AST skeletonizing, cross-turn deduplication of repeated file reads, a rule-based model-tier router, and an MCP server.
Coding agents spend a large share of their context window on noisy test-runner output, Docker build logs, unfocused helper code, and the same read_file payload repeated across turns. tokendiet trims that context deterministically before inference. Lines matching error-anchor patterns (tracebacks, error TS…, constraint violations, etc.) are always kept, and skeletonized Python is re-checked with ast.parse. It is a small, focused tool (~2k lines of Python plus a ~220-line C++20 kernel), not a learned compressor.
![]()
Architecture
flowchart LR
Raw["Raw Agent Context<br/>(pytest/tsc/docker logs · Python files · multi-turn reads)"] --> Split{"Tool / Turn Classifier"}
Split -->|CLI / Test Log| CppKnapsack["C++20 Salience Knapsack<br/>(clang++ -O3 · integer halo DP)"]
Split -->|Python Source| ASTSkel["Python AST Skeletonizer<br/>(ast.parse-verified stub elision)"]
Split -->|Repeated Read| CppDedup["C++20 Cross-Turn Block Dedup<br/>([@tokendiet:ref] back-pointers)"]
CppKnapsack --> Diet["Compressed Context<br/>(100% anchor retention)"]
ASTSkel --> Diet
CppDedup --> Diet
Diet --> Router["FinOps Cascade Router<br/>(Local Flash -> Balanced -> Reasoning Tier)"]
Diet --> MCP["MCP stdio Server (.mcp.json)<br/>& FastAPI + React Workbench (:8110)"]Related MCP server: squeeze
Measured Verification & Benchmarks
All checks ran cleanly on local hardware (Python 3.13.9, clang++ -O3 C++20, Ollama qwen3.5:4b):
Verification Suite | Exact Measured Result |
Static Quality Gates |
|
Pytest + Hypothesis Suite |
|
Live |
|
|
|
|
|
|
|
FinOps Cascade Routing | Single-unit bugfix routed to |
Live Ollama (qwen3.5:4b) Task Breakdown
Full artifact: benchmarks/real_qwen_tokendiet.json
Task | Mode | Raw Prompt Tok | Diet Prompt Tok | Token Saved | Latency (Raw → Diet) | Verified |
|
| 1,621 | 312 | 80.75% |
| PASS → PASS |
|
| 1,328 | 283 | 78.69% |
| PASS → PASS |
|
| 823 | 525 | 36.21% |
| PASS → PASS |
|
| 358 | 285 | 20.39% |
| PASS → PASS |
|
| 1,270 | 284 | 77.64% |
| PASS → PASS |
|
| 1,087 | 241 | 77.83% |
| PASS → PASS |
Scope & Limitations
Token counts are estimates. The compressors budget with a
ceil(len(text) / 4)character heuristic, not a real tokenizer; the live-audit token numbers come from Ollama'sprompt_eval_countforqwen3.5:4band will differ for other models' tokenizers.Small, self-constructed benchmark. The live audit is 6 hand-built tasks (synthetic pytest/tsc/Docker/Postgres logs and code) run once at temperature 0 on a local 4B model. "Verified" means the answer contains required keyword groups (e.g. file name and line number), not a human or model-graded judgement. Latency numbers are single runs on one laptop.
Heuristic salience. Line scores come from hand-written regex patterns (tracebacks,
error TS…,Elines, diff hunks, etc.). Log formats the patterns do not recognise can lose useful context; tune the budget conservatively.Python-only skeletonizer. AST skeletons are implemented for Python source only; other languages pass through the log pruner or untouched.
Router is rule-based. The cascade router uses keyword hints, retry count and file count with fixed illustrative per-million-token prices; its "cost reduction" figure is relative to those assumed prices, not a measured bill.
Install
# From PyPI (once published)
pip install tokendiet
# From source (today)
git clone https://github.com/nagarjunak-pixel/tokendiet.git && cd tokendiet
uv sync --all-extras # or: pip install .The C++20 kernel source ships inside the package and is compiled on first import with clang++ or g++ (cached in your temp dir). If no compiler is available, tokendiet falls back to an equivalent pure-Python implementation; tokendiet demo prints which backend is active. The React workbench (tokendiet serve) and the recorded benchmark replay need a source checkout.
Quickstart
1. Install & Run Showcase
uv sync --all-extras
uv run tokendiet demo2. Pipe Bloated Logs or Skeletonize Python Files
# Prune noisy pytest/tsc/docker output down to a 180-token salience budget
pytest -v 2>&1 | uv run tokendiet prune --budget 180
# Skeletonize unfocused functions while keeping focus symbols intact
uv run tokendiet skeleton src/billing.py --focus compute_monthly_discount3. Drop-In Claude Code / Cursor MCP Server (.mcp.json)
{
"mcpServers": {
"tokendiet": {
"command": "uv",
"args": ["run", "--directory", "/path/to/tokendiet", "tokendiet", "mcp-stdio"]
}
}
}Exposed MCP tools:
tokendiet_prune_tool_outputtokendiet_skeletonize_codetokendiet_compress_sessiontokendiet_route_finops
4. Interactive Workbench & Live Ollama Re-Audit
cd ui && npm install && npm run build && cd ..
uv run tokendiet eval-ollama --live-only
uv run tokendiet serve --host 127.0.0.1 --port 8110Why the Native C++20 Kernel Matters
Implemented in src/tokendiet/_csrc/tokendiet_kernels.cpp and automatically compiled via clang++ -O3 (with automatic pure-Python integer parity fallback):
Windowed Non-Anchor Duplicate Suppression: Drops repetitive non-anchor boilerplate whose hash appeared in the preceding 8 lines.
Integer Anchor Halo Propagation: Uses pure
int64_tdivisionraw_score / (1 + distance)to keep traceback frames and nearby failure context intact.Exact Order-Preserving 0/1 Knapsack DP: Guarantees selected lines preserve original chronological ordering and respect token budgets without floating-point drift (see ADR 0001 and ADR 0002).
Author & Links
Built by Nagarjuna Reddy Kamireddy (Nagarjuna Kamireddy / Nagarjuna K) — Lead AI Architect, GenAI & Agentic AI Lead, and Corporate AI Trainer (20+ years across Enterprise Data, SAP HANA & Production AI; AWS Certified Solutions Architect – Professional & NVIDIA-Certified Professional: Agentic AI).
Portfolio: https://nagarjunak-pixel.github.io/
Case Studies & Technical Articles: Case Studies · Architecture Articles · 10-Mandate Role Fit
Master CV & AI Search Index: Master CV ·
llms-full.txt· LinkedIn
Available Tools
4 toolstokendiet_compress_sessionC
Compress a multi-turn coding-agent trace using C++20 cross-turn read deduplication, log pruning, and AST skeletonization.
Inputs: turns (required), focus_symbols, log_budget_tokens.
tokendiet_prune_tool_outputB
Prune noisy CLI, pytest, tsc, or build logs to a token budget using the C++20 Order-Preserving Salience Knapsack while keeping error/traceback anchors.
Inputs: text (required), halo_radius, budget_tokens.
tokendiet_route_finopsC
Route an agent turn across Local Flash, Balanced Coding, and Frontier Reasoning tiers with Cascade Escalation Guard.
Inputs: prompt_summary (required), context_tokens_raw (required), context_tokens_compressed (required), retry_count, +2 more.
tokendiet_skeletonize_codeA
AST-skeletonize Python files by keeping imports, signatures, docstrings, and focus symbols while eliding long unfocused helper bodies.
Inputs: source (required), focus_symbols, min_elide_lines.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
Each tool targets a distinct operation: log pruning, code skeletonization, session compression, and cost-tier routing. However, compress_session explicitly bundles log pruning and AST skeletonization, so its boundary with prune_tool_output and skeletonize_code is fuzzy—an agent may pick the narrower tool when the session-level one was intended (or vice versa).
All four tools share the tokendiet_ prefix and follow a consistent snake_case verb_noun pattern (prune_tool_output, skeletonize_code, compress_session, route_finops). The naming is predictable and readable.
Four tools is a lean, well-scoped surface where each tool maps to a distinct compression/routing concern, with no filler. It is slightly thin—nothing handles token counting or budget configuration—but nothing is redundant.
The set covers the main workflow: prune logs, skeletonize code, compress a full trace, and route to model tiers. It lacks a standalone token-count/budget-estimation utility and any way to inspect or tune pruning/skeletonization settings, which agents would likely want before committing to a compression budget.
Maintenance
Related MCP Connectors
Codebase intelligence for agents: 152 structured artifacts across 21 programs, one call.
Cuts coding agent token burn up to 72% continuously via contradiction-free memory & verification.
Code-map tools for AI agents: see a repo's structure first, then edit only what matters.
Compress coding-agent context ~64%. Hosted Neural Keep MCP for Cursor/Claude/Codex.
Related MCP Servers
- AlicenseAqualityBmaintenanceEnables AI coding agents to reduce token usage by condensing source code into AST skeletons, extracting specific symbols, compressing logs, and estimating token budgets.4MIT
- AlicenseNot gradedqualityBmaintenanceEnables a coding agent to compress noisy tool output such as compiler errors, failing test runs and repeated logs before it enters the context window, and to fetch any collapsed ref back as its exact original text. Two tools, squeeze_fetch and squeeze_compress, keep the reduction reversible by the model itself.1MIT
- AlicenseNot gradedqualityCmaintenanceEnables coding agents to shrink bulky tool output, logs, test runs, and file dumps down to compact briefs, and to record and recall decisions in a small session ledger. This keeps long-running tasks within context limits while preserving errors, findings, and key notes.8 npmMIT
- AlicenseBqualityCmaintenanceEnables coding agents to reduce context tokens by requesting file outlines, individual symbols, compact searches, and condensed test, lint, and type-check failures instead of reading whole files or raw command output.666 npmMIT