Skip to main content
Glama

tokendiet

TokenDiet CI Release v1.0.0 License: MIT

A context compressor for AI coding agents: order-preserving knapsack pruning of noisy tool logs (C++20 kernel with a pure-Python fallback), Python AST skeletonizing, cross-turn deduplication of repeated file reads, a rule-based model-tier router, and an MCP server.

Coding agents spend a large share of their context window on noisy test-runner output, Docker build logs, unfocused helper code, and the same read_file payload repeated across turns. tokendiet trims that context deterministically before inference. Lines matching error-anchor patterns (tracebacks, error TS…, constraint violations, etc.) are always kept, and skeletonized Python is re-checked with ast.parse. It is a small, focused tool (~2k lines of Python plus a ~220-line C++20 kernel), not a learned compressor.


tokendiet Workbench UI

Architecture

flowchart LR
    Raw["Raw Agent Context<br/>(pytest/tsc/docker logs · Python files · multi-turn reads)"] --> Split{"Tool / Turn Classifier"}
    Split -->|CLI / Test Log| CppKnapsack["C++20 Salience Knapsack<br/>(clang++ -O3 · integer halo DP)"]
    Split -->|Python Source| ASTSkel["Python AST Skeletonizer<br/>(ast.parse-verified stub elision)"]
    Split -->|Repeated Read| CppDedup["C++20 Cross-Turn Block Dedup<br/>([@tokendiet:ref] back-pointers)"]
    CppKnapsack --> Diet["Compressed Context<br/>(100% anchor retention)"]
    ASTSkel --> Diet
    CppDedup --> Diet
    Diet --> Router["FinOps Cascade Router<br/>(Local Flash -> Balanced -> Reasoning Tier)"]
    Diet --> MCP["MCP stdio Server (.mcp.json)<br/>& FastAPI + React Workbench (:8110)"]

Related MCP server: squeeze

Measured Verification & Benchmarks

All checks ran cleanly on local hardware (Python 3.13.9, clang++ -O3 C++20, Ollama qwen3.5:4b):

Verification Suite

Exact Measured Result

Static Quality Gates

0 Ruff format issues · 0 Ruff lint issues · 0 Mypy --strict errors (21 files)

Pytest + Hypothesis Suite

14 passed · 97.72% statement coverage (differential C++20 vs Python parity across random budgets & turn IDs)

Live qwen3.5:4b Audit (6 tasks)

100.0% (6/6) verified accuracy preserved · 6,487 -> 1,930 prompt tokens (70.25% reduction) · 53.98% faster end-to-end latency

tokendiet demo (Pytest Log Knapsack)

996 -> 213 tokens (78.61% saved) · 4/4 (100.0%) mandatory error anchors kept

tokendiet demo (AST Skeletonizer)

431 -> 275 tokens (36.19% saved) · 3 helper functions elided · ast.parse=True

tokendiet demo (Multi-Turn Session)

2,289 -> 1,047 tokens (54.26% saved) · 5 repeated blocks deduplicated

FinOps Cascade Routing

Single-unit bugfix routed to local_flash (99.69% cost reduction); retry/formal ledger proof escalated to frontier_reasoning

Live Ollama (qwen3.5:4b) Task Breakdown

Full artifact: benchmarks/real_qwen_tokendiet.json

Task

Mode

Raw Prompt Tok

Diet Prompt Tok

Token Saved

Latency (Raw → Diet)

Verified

TD-01 Pytest ZeroDivisionError

log_prune

1,621

312

80.75%

2970.21ms -> 664.01ms

PASS → PASS

TD-02 TypeScript TS2345 Build

log_prune

1,328

283

78.69%

1002.64ms -> 539.67ms

PASS → PASS

TD-03 Multi-Turn Read + TimeoutError

session_dedup

823

525

36.21%

718.44ms -> 615.84ms

PASS → PASS

TD-04 Python AST Focus Diagnosis

ast_skeleton

358

285

20.39%

624.75ms -> 528.38ms

PASS → PASS

TD-05 Docker ModuleNotFoundError

log_prune

1,270

284

77.64%

1024.04ms -> 552.53ms

PASS → PASS

TD-06 Postgres CheckViolationError

log_prune

1,087

241

77.83%

792.12ms -> 382.16ms

PASS → PASS

Scope & Limitations

  • Token counts are estimates. The compressors budget with a ceil(len(text) / 4) character heuristic, not a real tokenizer; the live-audit token numbers come from Ollama's prompt_eval_count for qwen3.5:4b and will differ for other models' tokenizers.

  • Small, self-constructed benchmark. The live audit is 6 hand-built tasks (synthetic pytest/tsc/Docker/Postgres logs and code) run once at temperature 0 on a local 4B model. "Verified" means the answer contains required keyword groups (e.g. file name and line number), not a human or model-graded judgement. Latency numbers are single runs on one laptop.

  • Heuristic salience. Line scores come from hand-written regex patterns (tracebacks, error TS…, E lines, diff hunks, etc.). Log formats the patterns do not recognise can lose useful context; tune the budget conservatively.

  • Python-only skeletonizer. AST skeletons are implemented for Python source only; other languages pass through the log pruner or untouched.

  • Router is rule-based. The cascade router uses keyword hints, retry count and file count with fixed illustrative per-million-token prices; its "cost reduction" figure is relative to those assumed prices, not a measured bill.

Install

# From PyPI (once published)
pip install tokendiet

# From source (today)
git clone https://github.com/nagarjunak-pixel/tokendiet.git && cd tokendiet
uv sync --all-extras        # or: pip install .

The C++20 kernel source ships inside the package and is compiled on first import with clang++ or g++ (cached in your temp dir). If no compiler is available, tokendiet falls back to an equivalent pure-Python implementation; tokendiet demo prints which backend is active. The React workbench (tokendiet serve) and the recorded benchmark replay need a source checkout.


Quickstart

1. Install & Run Showcase

uv sync --all-extras
uv run tokendiet demo

2. Pipe Bloated Logs or Skeletonize Python Files

# Prune noisy pytest/tsc/docker output down to a 180-token salience budget
pytest -v 2>&1 | uv run tokendiet prune --budget 180

# Skeletonize unfocused functions while keeping focus symbols intact
uv run tokendiet skeleton src/billing.py --focus compute_monthly_discount

3. Drop-In Claude Code / Cursor MCP Server (.mcp.json)

{
  "mcpServers": {
    "tokendiet": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/tokendiet", "tokendiet", "mcp-stdio"]
    }
  }
}

Exposed MCP tools:

  • tokendiet_prune_tool_output

  • tokendiet_skeletonize_code

  • tokendiet_compress_session

  • tokendiet_route_finops

4. Interactive Workbench & Live Ollama Re-Audit

cd ui && npm install && npm run build && cd ..
uv run tokendiet eval-ollama --live-only
uv run tokendiet serve --host 127.0.0.1 --port 8110

Why the Native C++20 Kernel Matters

Implemented in src/tokendiet/_csrc/tokendiet_kernels.cpp and automatically compiled via clang++ -O3 (with automatic pure-Python integer parity fallback):

  1. Windowed Non-Anchor Duplicate Suppression: Drops repetitive non-anchor boilerplate whose hash appeared in the preceding 8 lines.

  2. Integer Anchor Halo Propagation: Uses pure int64_t division raw_score / (1 + distance) to keep traceback frames and nearby failure context intact.

  3. Exact Order-Preserving 0/1 Knapsack DP: Guarantees selected lines preserve original chronological ordering and respect token budgets without floating-point drift (see ADR 0001 and ADR 0002).


Built by Nagarjuna Reddy Kamireddy (Nagarjuna Kamireddy / Nagarjuna K) — Lead AI Architect, GenAI & Agentic AI Lead, and Corporate AI Trainer (20+ years across Enterprise Data, SAP HANA & Production AI; AWS Certified Solutions Architect – Professional & NVIDIA-Certified Professional: Agentic AI).

Available Tools

4 tools
tokendiet_compress_sessionC

Compress a multi-turn coding-agent trace using C++20 cross-turn read deduplication, log pruning, and AST skeletonization.

Inputs: turns (required), focus_symbols, log_budget_tokens.

tokendiet_prune_tool_outputB

Prune noisy CLI, pytest, tsc, or build logs to a token budget using the C++20 Order-Preserving Salience Knapsack while keeping error/traceback anchors.

Inputs: text (required), halo_radius, budget_tokens.

tokendiet_route_finopsC

Route an agent turn across Local Flash, Balanced Coding, and Frontier Reasoning tiers with Cascade Escalation Guard.

Inputs: prompt_summary (required), context_tokens_raw (required), context_tokens_compressed (required), retry_count, +2 more.

tokendiet_skeletonize_codeA

AST-skeletonize Python files by keeping imports, signatures, docstrings, and focus symbols while eliding long unfocused helper bodies.

Inputs: source (required), focus_symbols, min_elide_lines.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.1/5.0

    Scored across 4 tools

    Disambiguation4/5

    Each tool targets a distinct operation: log pruning, code skeletonization, session compression, and cost-tier routing. However, compress_session explicitly bundles log pruning and AST skeletonization, so its boundary with prune_tool_output and skeletonize_code is fuzzy—an agent may pick the narrower tool when the session-level one was intended (or vice versa).

    Naming Consistency5/5

    All four tools share the tokendiet_ prefix and follow a consistent snake_case verb_noun pattern (prune_tool_output, skeletonize_code, compress_session, route_finops). The naming is predictable and readable.

    Tool Count4/5

    Four tools is a lean, well-scoped surface where each tool maps to a distinct compression/routing concern, with no filler. It is slightly thin—nothing handles token counting or budget configuration—but nothing is redundant.

    Completeness4/5

    The set covers the main workflow: prune logs, skeletonize code, compress a full trace, and route to model tiers. It lacks a standalone token-count/budget-estimation utility and any way to inspect or tune pruning/skeletonization settings, which agents would likely want before committing to a compression budget.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      Enables AI coding agents to reduce token usage by condensing source code into AST skeletons, extracting specific symbols, compressing logs, and estimating token budgets.
      4
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables a coding agent to compress noisy tool output such as compiler errors, failing test runs and repeated logs before it enters the context window, and to fetch any collapsed ref back as its exact original text. Two tools, squeeze_fetch and squeeze_compress, keep the reduction reversible by the model itself.
      1
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables coding agents to shrink bulky tool output, logs, test runs, and file dumps down to compact briefs, and to record and recall decisions in a small session ledger. This keeps long-running tasks within context limits while preserving errors, findings, and key notes.
      8 npm
      MIT
    • A
      license
      B
      quality
      C
      maintenance
      Enables coding agents to reduce context tokens by requesting file outlines, individual symbols, compact searches, and condensed test, lint, and type-check failures instead of reading whole files or raw command output.
      6
      66 npm
      MIT