# TokenPilot

Automatic token optimization for [Claude Code](https://claude.ai/code). Extends session duration by reducing wasted tokens across every dimension — effort tuning, redundant file reads, tool cost routing, context health tracking, and smart task classification.

Built as a Claude Code hooks + MCP server system. Works alongside [RTK](https://github.com/rtk-ai/rtk) for shell compression and [MCP Compressor](https://github.com/atlassian-labs/mcp-compressor) for schema reduction.

## How It Works

TokenPilot runs as four layers:

1. **Hooks** — intercept Claude Code lifecycle events (session start, prompt submit, pre/post tool use)
2. **MCP Server** — exposes tools for real-time control and monitoring
3. **SQLite Database** — persists session state across hook subprocess calls with WAL mode + serializable isolation
4. **Tool Registry** — maps known tools to estimated costs and cheaper alternatives

```
┌── Claude Code Hooks ──────────────────────────────────────┐
│                                                            │
│  SessionStart        → init session, inject hints          │
│  UserPromptSubmit    → classify task → suggest effort      │
│  PreToolUse (Read)   → dedup file reads + suggest cheaper  │
│  PostToolUse (all)   → track real tool output token costs  │
│                                                            │
└────────────────┬───────────────────────────────────────────┘
                 │
    ┌────────────▼──────────────────┐
    │   TokenPilot MCP Server       │
    │                               │
    │   set_level(1-10)             │  Aggressiveness dial
    │   get_stats()                 │  Live session metrics
    │   get_savings()               │  Token savings report
    │   get_context_health()        │  Context window status
    │   get_tool_report()           │  Most expensive tools
    │   get_file_report(path)       │  File read history
    │   explain_classification(p)   │  Debug classifier
    │   reset_file_tracking()       │  Clear dedup cache
    │                               │
    │   SQLite + Tool Registry      │  Persistent state
    └───────────────────────────────┘
```

## Aggressiveness Scale

Default: **4** (conservative-balanced). Adjustable 1-10 at any time via `/tp level N`.

| Level | Effort Suggestion | File Read Dedup | Thinking Cap | Compact Reminder |
|-------|-------------------|-----------------|-------------|-----------------|
| 1-2   | Never | Notify only | No cap | 90% context |
| 3-4   | Trivial tasks only | Warn on redundant | No cap | 75% context |
| 5-6   | All tasks | Warn + suggest alternatives | Adaptive (10-30K) | 65% context |
| 7-8   | Strong recommendation | Block re-reads | Adaptive (6-18K) | 55% context |
| 9-10  | Enforce | Block + auto-range | Adaptive (4-12K) | 45% context |

**Thinking caps are adaptive** — they scale based on task complexity and classifier confidence. A "trivial" task gets a tighter cap than a "complex" task. If the classifier is uncertain (confidence < 0.5), no cap is applied.

## Task Classifier (v2)

Lightweight regex + keyword classifier with negation detection, adjacency scoring, and quoted-code filtering. No LLM calls, <10ms execution.

| Category | Effort | Model Hint | Example |
|----------|--------|-----------|---------|
| `trivial` | low | haiku | "fix typo in README" |
| `research` | medium | sonnet | "explain how the API routes work" |
| `standard` | medium | sonnet | "add a loading spinner" |
| `complex` | high | opus | "refactor auth across all microservices" |

**v2 improvements:**
- **Negation detection**: "don't refactor" no longer matches the refactor pattern
- **Quoted-code filtering**: backtick-wrapped code is stripped before classification
- **Adjacency scoring**: "add auth to 12 routes" correctly detects complexity from keyword pairs
- **Confidence calibration**: very short prompts get low confidence (0.3) instead of false high confidence

Debug any classification with `/tp explain <prompt>`.

## Tool Cost Registry

TokenPilot knows the estimated token cost of common tools and suggests cheaper alternatives:

| Tool | Avg Tokens | Alternative | Alt Tokens | Savings |
|------|-----------|-------------|-----------|---------|
| Read | ~2000 | jCodeMunch symbol lookup | ~200 | 90% |
| WebSearch | ~2000 | Context7 docs query | ~800 | 60% |
| WebFetch | ~3000 | Context7 docs query | ~800 | 73% |

At level 5+, TokenPilot suggests alternatives when a cheaper tool could do the job.

## Installation

### Prerequisites

- Python 3.10+
- [Claude Code](https://claude.ai/code) CLI
- [FastMCP](https://github.com/jlowin/fastmcp) (`pip3 install fastmcp`)

### Setup

1. Clone to your MCPs directory:

```bash
git clone https://github.com/rish-e/tokenpilot.git ~/MCPs/tokenpilot
```

2. Install dependencies:

```bash
pip3 install -r ~/MCPs/tokenpilot/requirements.txt
```

3. Add hooks and MCP server to `~/.claude/settings.json`:

```json
{
  "hooks": {
    "SessionStart": [
      {
        "hooks": [
          { "type": "command", "command": "~/MCPs/tokenpilot/hooks/session_start.sh", "timeout": 5 }
        ]
      }
    ],
    "UserPromptSubmit": [
      {
        "hooks": [
          { "type": "command", "command": "~/MCPs/tokenpilot/hooks/classify.sh", "timeout": 5 }
        ]
      }
    ],
    "PreToolUse": [
      {
        "matcher": "Read",
        "hooks": [
          { "type": "command", "command": "~/MCPs/tokenpilot/hooks/check_read.sh", "timeout": 5 }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": ".*",
        "hooks": [
          { "type": "command", "command": "~/MCPs/tokenpilot/hooks/post_tool.sh", "timeout": 3 }
        ]
      }
    ]
  },
  "mcpServers": {
    "tokenpilot": {
      "command": "python3",
      "args": ["~/MCPs/tokenpilot/server.py"],
      "env": { "PYTHONPATH": "~/MCPs/tokenpilot" }
    }
  }
}
```

4. Install the `/tp` slash command:

```bash
cp ~/MCPs/tokenpilot/commands/tp.md ~/.claude/commands/tp.md
```

5. Restart Claude Code.

### Optional: RTK for Shell Compression

```bash
brew install rtk-ai/tap/rtk
rtk init -g
```

Adds 60-90% token savings on shell output (build logs, test output, git).

## Usage

TokenPilot runs automatically after installation. You'll see `[TokenPilot]` messages when it detects optimization opportunities.

### Slash Commands

5 commands. That's it.

| Command | What it does |
|---------|-------------|
| `/tp <1-10>` | Set aggressiveness level |
| `/tp on` / `off` | Enable/disable TokenPilot |
| `/tp stats` | Full session dashboard |
| `/tp note <text>` | Add a note to the Project Brain |
| `/tp explain <prompt>` | Debug why a prompt was classified |

### MCP Tools

For power users, all tools are callable directly:

`set_level` `toggle` `get_stats` `get_savings` `get_context_health` `get_tool_report` `get_file_report` `explain_classification` `add_note` `reset_file_tracking`

### CLI (for testing)

```bash
cd ~/MCPs/tokenpilot

python3 server.py init 4                    # Initialize session
python3 server.py classify "fix typo"       # Classify prompt
python3 server.py classify_debug "fix typo" # Debug classification
python3 server.py check_file "/src/app.py"  # Check file dedup
python3 server.py context_health            # Context window status
```

## Project Brain

TokenPilot auto-maintains a `tpcontext.md` file in each project root. This is persistent memory across Claude Code sessions — when you start a new chat, Claude immediately knows where you left off.

**Fully automatic:**
- **First install** — bootstraps from git history (commits, active files, branch)
- **Every session start** — auto-saves previous session, loads brain into context
- **No manual save needed** — it just works

**What it captures:**
- Files modified (from git diff)
- Recent commits
- User notes (via `/tp note "..."`)
- Session stats (duration, prompt count)
- Most active files

**Add context for future sessions:**
```
/tp note "switched to GraphQL — don't touch REST endpoints"
```

Stays under 2K tokens. Keeps last 5 sessions, older ones rotate out.

## Smart Warnings

TokenPilot automatically detects and warns about token-wasting patterns:

- **Rapid-fire prompts** — 3+ short messages in a row triggers a "batch your questions" suggestion
- **Session age** — every 15 prompts, suggests `/compact` or starting fresh
- **Peak hours** — warns once per session during 5-11am PT weekdays (Anthropic burns limits faster during peak)

All warnings appear as `[TokenPilot]` messages and respect the on/off toggle.

## File Structure

```
tokenpilot/
├── server.py            # FastMCP server + CLI entry point
├── classifier.py        # Task classifier (v2: negation, adjacency, debug)
├── config.py            # Aggressiveness scale + adaptive thinking caps
├── db.py                # SQLite persistence (WAL, indexed, serializable)
├── brain.py             # Project Brain — auto-generated tpcontext.md
├── tool_registry.py     # Tool cost estimates + cheaper alternatives
├── tracker.py           # In-memory tracker (used by MCP server process)
├── requirements.txt
├── commands/
│   └── tp.md            # /tp slash command (copy to ~/.claude/commands/)
├── hooks/
│   ├── session_start.sh # SessionStart — init + load brain
│   ├── classify.sh      # UserPromptSubmit — classify + rapid-fire + peak hours + session age
│   ├── check_read.sh    # PreToolUse (Read) — dedup + tool routing
│   └── post_tool.sh     # PostToolUse — real token tracking
└── templates/
    └── claudeignore-default
```

## How Token Savings Stack

| Layer | What | Savings |
|-------|------|---------|
| TokenPilot classifier | Right effort level per task | Thinking token reduction |
| TokenPilot file dedup | Skip redundant file reads | ~2K tokens per blocked read |
| TokenPilot tool routing | Suggest cheaper tool alternatives | 60-90% per substitution |
| TokenPilot PostToolUse | Track actual token costs (visibility) | Measurement enables optimization |
| TokenPilot smart warnings | Batch prompts, session age, peak hours | Prevents context blowup |
| TokenPilot Project Brain | Resume sessions without re-explaining context | 3-5 messages saved per session start |
| RTK | Compress shell output | 60-90% on Bash results |
| MCP Compressor | Compress MCP tool schemas | 70-97% per wrapped server |
| .claudeignore | Exclude build artifacts from search | 30-40% on exploration |

## License

MIT