tokencut

The Context Accumulation Problem

AI coding agents (Claude Code, Cursor, Codex, Gemini CLI) run inside multi-turn conversation sessions. Every executed command, test runner output, and inspected file is appended to the conversation history and resent on every subsequent turn:

• Quadratic Token Accumulation: A 2,000-line test run does not consume tokens once—it is transmitted to the model on every subsequent prompt in the session.

• Low Signal-to-Noise Ratio: Routine test passes, compiler progress bars, and ANSI escape sequences dominate terminal dumps. The actual failure or error trace is often under 30 lines.

• Premature Rate Limits: API quotas and usage limits (such as the Claude Code 5-hour window) are exhausted by repetitive output rather than code generation and reasoning.

• Context Window Degradation: Splicing thousands of irrelevant terminal lines into the context window triggers needle-in-a-haystack degradation, increasing the likelihood that the model forgets earlier instructions.

Related MCP server: project-graph-mcp

Comparison

How tokencut compares to standard agent execution, static context packagers (such as Repomix), and generic context buffers:

Empirical Benchmarks

Measured on representative real-world developer workloads across Anthropic Claude, OpenAI, and Google Gemini tokenizers:

Architecture

tokencut is designed around six core mechanisms:

1. Reversible Compress-Cache-Retrieve (CCR)

Context compaction should never cause irreversible data loss. When tokencut truncates repetitive output, it persists the full uncompressed stream to a local SQLite database (~/.tokencut/cache.db) and injects a deterministic reference identifier:

[... 340 lines of routine output omitted by tokencut (-84.1%). Ref: tc_8f2a1b ...]

If an agent or developer needs the omitted output, it can be fetched instantly:

tokencut retrieve tc_8f2a1b --lines 120-160

Or programmatically through the native tokencut_retrieve tool via MCP.

2. Repository Token Profiling (tokencut tree)

Identifies high-consumption files and directories before context is loaded into an agent session:

uvx tokencut tree .

tokencut/  · 170,499 tok (100.0%)
├── src/ · 15,282 tok (9.0%)
├── tests/ · 2,628 tok (1.5%)
└── uv.lock · 148,640 tok (87.2%) [Top Consumer]

3. AST Code Skeletonization (tokencut cat --skeleton)

During multi-file codebase navigation, feeding complete implementation bodies into the prompt exhausts context rapidly. tokencut cat parses Python files via the standard library ast module and other languages (TypeScript, JavaScript, Go, Rust) via structural regex to extract classes, method signatures, type annotations, and docstrings:

# View outline of a module
uvx tokencut cat src/auth.py --skeleton

# Extract a specific class or method
uvx tokencut cat src/auth.py --symbol AuthService.verify_token

# Extract specific line slice with file context
uvx tokencut cat src/auth.py --lines 45-80

4. Git Diff Slimming (tokencut diff)

Package lockfiles (uv.lock, package-lock.json, pnpm-lock.yaml) often generate thousands of lines of machine-generated diffs that crowd out actual application changes. tokencut diff collapses lockfile modifications into summary counts while preserving application code diffs in full fidelity.

5. Credential & Secret Scrubbing

Intercepts API keys (OpenAI sk-*, Anthropic sk-ant-*, Google AIza*, GitHub ghp_*), JWTs, and database URLs containing passwords before they enter agent context or terminal logs.

6. Prompt Cache Optimization (tokencut lint)

Provider prompt caching (Anthropic, Gemini) offers up to 90% cost savings for invariant prompt prefixes. tokencut lint analyzes system instruction files (CLAUDE.md, .cursorrules, system prompts) to identify dynamic timestamps, non-deterministic paths, and volatile headers that invalidate prompt caches.

7. Structured JSON & API Payload Compaction (tokencut json)

When coding agents fetch API responses via curl or inspect JSON data files, hundreds of repetitive array items quickly burn tens of thousands of tokens. tokencut json folds large lists while retaining the first few items and schema annotations, truncates oversized strings (such as base64 images or hashes), and caches the raw JSON in SQLite with a reference ID.

8. System Diagnostics & Auto-Configuration (tokencut doctor)

Inspects your local environment across Python runtime, SQLite cache health, Claude Code CLI, Cursor MCP configurations, and shell aliases. Running tokencut doctor --fix or tokencut install --all automatically writes the required configurations with zero manual editing.

9. Pull Request Token Impact Analyzer (tokencut pr)

Evaluates the net token delta introduced by code changes against main or a target base ref. Categorizes token impact into code, documentation, and lockfiles, and emits a clean Markdown summary for GitHub PR review comments. When run with --max-delta <N>, it acts as an automated CI gatekeeper preventing accidental lockfile or fixture context bloat.

Quickstart

Run directly without installation via uvx:

# Execute a test suite through tokencut
uvx tokencut run -- pytest -v tests/

# Analyze repository token distribution
uvx tokencut tree .

# Inspect code structure without function bodies
uvx tokencut cat src/server.py --skeleton

# Inspect git diff with folded lockfiles
uvx tokencut diff

# Compact large JSON file or API response
uvx tokencut json api_response.json

# Check environment health & auto-configure Cursor / shell
uvx tokencut doctor --fix

# Audit instructions for prompt cache-busting
uvx tokencut lint CLAUDE.md

# Run the terminal demonstration
uvx tokencut demo

Or install globally:

uv pip install tokencut
# or
pip install tokencut

Integrations & Supported Platforms

tokencut operates across desktop applications, AI-enabled IDEs, coding agents, and terminal command-line pipelines.

1-Click Auto-Configuration

Configure all desktop and editor integrations automatically:

uvx tokencut install --all

Or verify setup status across all targets with:

uvx tokencut doctor

Claude Desktop (macOS)

Auto-configure ~/Library/Application Support/Claude/claude_desktop_config.json:

uvx tokencut install --claude-desktop

Or manually add:

{
  "mcpServers": {
    "tokencut": {
      "command": "uvx",
      "args": ["tokencut", "mcp"]
    }
  }
}

ChatGPT Desktop (macOS)

Compatible with ChatGPT Desktop via Developer Mode local MCP tools or CLI piping:

# Start MCP server for ChatGPT Developer Mode
uvx tokencut mcp

Cursor & Windsurf

Auto-configure ~/.cursor/mcp.json:

uvx tokencut install --cursor

Or manually add to mcp.json:

{
  "mcpServers": {
    "tokencut": {
      "command": "uvx",
      "args": ["tokencut", "mcp"]
    }
  }
}

Claude Code CLI

Register tokencut as a native MCP server in one command:

claude mcp add tokencut uvx tokencut mcp

This exposes seven tools directly to Claude:

• tokencut_exec: Runs bash commands with output compaction, CCR caching, and optional --budget.

• tokencut_read: Reads files with support for AST skeletons, symbol extraction, and line ranges.

• tokencut_retrieve: Retrieves omitted slices from cached terminal runs by reference ID.

• tokencut_diff: Generates slim git diffs with lockfile folding.

• tokencut_tree: Profiles directory-level token consumption directly inside conversation.

• tokencut_json: Compresses large JSON payloads and API responses with schema retention.

• tokencut_stats: Reports session and lifetime token savings.

Terminal CLI & POSIX Pipelines (Gemini CLI, Codex, bash, zsh)

tokencut integrates into standard terminal workflows:

# Add 'cc' shortcut to ~/.zshrc or ~/.bashrc
uvx tokencut install --alias

# Run commands with automatic token compaction
cc pytest -v tests/
cc npm test

# Pipe stdout/stderr through tokencut
cargo test 2>&1 | tokencut pipe
curl https://api.github.com/repos/00200200/tokencut/commits | tokencut json

GitHub Actions CI Gatekeeper

Use the official composite action to audit PR token delta or wrap test steps:

- name: Check PR Token Impact
  uses: 00200200/tokencut@main
  with:
    pr-check: 'true'
    max-token-delta: '25000'

Pre-Commit Hook

Add to your .pre-commit-config.yaml to prevent prompt cache-busting before committing:

repos:
  - repo: https://github.com/00200200/tokencut
    rev: main
    hooks:
      - id: tokencut-lint
      - id: tokencut-pr

CLI Reference

Development

# Clone the repository
git clone https://github.com/00200200/tokencut.git
cd tokencut

# Install dependencies in a virtual environment
uv sync

# Run the test suite (37 tests)
uv run pytest -v

# Run the linter
uv run ruff check .

# Run the benchmark suite
uv run python scripts/benchmark_suite.py

License

Released under the MIT License.