TokenDiet MCP

TokenDiet is a local MCP server that compresses file reads, command output, search hits, and fetched web pages before they enter the agent context. It uses deterministic transforms (outline+, log dedup, snippet caps) and a safety verifier that rolls back when compression would drop protected content or fail to shrink the payload. Token counts use a real BPE encoder (o200k_base by default), not character guesses.

This only helps when the agent gets data through TokenDiet tools. If built-in Read or Bash already loaded the full text into context, calling compress afterward cannot undo that cost.

Two value propositions (honest)

1. Billed cost — savings on profile workloads (logs, web pages, large files, batch reads) when the agent uses TokenDiet tools without extra MCP turns. On small files or drill-down ladders, billed cost can be higher than baseline; the server-side economic guard and batch API exist to prevent that. See benchmarks/2026-07-25-e2e-turn-neutral.md.

2. Context window budget — fewer fresh tokens per turn delays context summarization in long sessions. This benefit is independent of cache pricing and applies even when billed cost is neutral.

Turn-neutral rule (v0.7.0): compression must not multiply agent turns. Implemented via economic guard (T_full ≈ 3K BPE), read(targets[]) batching, outline+ (not outline→expand ladders), and optional shrink proxy for upstream MCP servers.

verified: true means structure verified — check warnings, omitted.bodies, and compression.net_estimate before editing code.

Related MCP server: winnow

Install (from source)

Requires Node.js 20+ and build tools for better-sqlite3 (native addon).

git clone https://github.com/DukeDeSouth/tokendiet-mcp.git
cd tokendiet-mcp
npm install
npm run build

Wire Cursor (example — adjust paths after clone):

node dist/index.js setup --client cursor --project /path/to/your/project

Reload MCP servers in Cursor. Point agents at read, run, search, fetch, expand, and stats instead of raw Read/Grep/Bash for large payloads.

Shrink proxy (wrap upstream MCP servers)

TokenDiet can sit between Cursor and another MCP server and compress results inside the same tool round-trip (no extra agent turns). This does not intercept built-in Cursor/Codex tools — only MCP servers you wrap.

# Example: wrap Playwright MCP
tokendiet-mcp shrink -- npx @playwright/mcp@latest

Cursor mcp.json (stdio) — replace the upstream command with the shrink wrapper:

{
  "mcpServers": {
    "playwright": {
      "command": "node",
      "args": ["/absolute/path/to/tokendiet-mcp/dist/index.js", "shrink", "--", "npx", "@playwright/mcp@latest"]
    }
  }
}

Optional: TOKENDIET_SHRINK_SERVER=playwright selects a profile from shrink.config.json (allowlist/denylist, min_text_tokens, content hints).

What gets compressed:

• tools/list — long tool description strings (schema overhead)

• tools/call — large text content blocks; images/resources pass through byte-safe

Compressed blocks append [compressed by tokendiet — expand(<ref>) for full]. Retrieve the original via the TokenDiet expand tool when the main TokenDiet MCP server is also configured.

See shrink.config.json for per-upstream tuning.

Tools

What to expect (honest ranges)

Measured on our dogfood corpus and E2E harness (benchmarks/), not a universal promise:

• Turn-neutral N-corpus (offline sim): v2 policy turns_ratio 0.38, cost_ratio 0.36 vs baseline

• User A/B v1 (documented failure): turns_ratio 4.42, cost_ratio 2.36 — why v0.7.0 exists

• Test and log output via run: often 68–99% payload saved

• Shrink proxy (Playwright snapshot sim): ≥50% saved, 0 extra turns

• Small files (< ~3K BPE): server returns full content; use built-in Read or batch targets[]

• stats.net_tokens_estimate subtracts turn cost — negative means the call was economically unprofitable

See benchmarks/2026-07-11-dogfood-v3.md and benchmarks/2026-07-25-e2e-turn-neutral.md for methodology.

Limitations

• AST outline modes: TypeScript, JavaScript, Python (via tree-sitter WASM bundled in wasm/)

• search without rg installed uses a slower JS walker (respects .gitignore)

• fetch does not execute JavaScript; private IPs are blocked (SSRF hygiene)

• BPE counts approximate Claude/Gemini tokenizers; relative savings are still meaningful because in/out use the same encoder (docs/TOKENIZER.md)

• Ref cache under ~/.tokendiet/refs (TTL/size capped via env) — local only, no cloud

Privacy

Everything runs on your machine over stdio. No telemetry, no remote compression service.

Development

npm test
npm run e2e:gate
npm run check-disclosure

License

MIT — see LICENSE.