TokenSafe

1-Click Universal Install

Connect TokenSafe to the AI coding agents on your workstation in one command:

# Option 1: Zero-Install via npx (Recommended for Cursor & Claude Desktop)
npx tokensafe

# Option 2: Python / PyPI
pip install tokensafe && tokensafe

# Option 3: Direct Shell Script (macOS / Linux)
curl -fsSL https://tokensafe.dev/install.sh | bash

On Google Antigravity, the installer additionally symlinks the plugin into ~/.gemini/config/plugins/tokensafe — Antigravity loads it automatically on startup (stale agy-token-optimizer symlinks from older versions are removed, since Antigravity keys plugin enable-state by folder name and would run every hook twice). Once the plugin ships its own mcp_config.json, any leftover user-level tokensafe MCP entry in Antigravity's configs is migrated away (backed up first) so the server is not registered twice.

Optional Dependency: cryptography (Recommended)

The local TokenSafe engine uses Python's standard library only and runs out of the box. One optional package, cryptography, is installed automatically by install.sh and requirements.txt when available:

pip install cryptography   # or: brew install cryptography

The client caches a verified license for up to 1 hour, so even the pure-Python path adds no measurable latency after the first hook invocation.

Related MCP server: Smart Context MCP

Universal Agent Hooks (Claude Code & Cursor)

Beyond MCP, TokenSafe installs native lifecycle-hook adapters inside your agent loop via tokensafe setup:

All adapters fail open: an internal error never stalls the host agent. Cursor's strict valid-JSON response contract is preserved on every error path.

For team workspaces on Antigravity, tokensafe install --project --agents-plugin <path-to-tokensafe> additionally declares the plugin in the project's .agents/plugins.json, so teammates load hooks, skills, and the MCP server straight from the shared repo without running the installer.

Context Budget HUD & Governor

Teams usually notice context exhaustion only when code quality drops mid-task. TokenSafe exposes live window usage as a native MCP tool:

{ "tool": "tokensafe:context_budget", "args": { "model": "claude-x" } }

{
  "window_tokens": 200000,
  "used_estimated": 168000,
  "used_pct": 84.0,
  "breakdown": { "tools": 165000, "history": 2300, "prompts": 700, "other": 0 },
  "threshold": "warning",
  "recommendation": "Context is heavy. Prefer tokensafe:smart_read_file ... and consider tokensafe:compact_conversation."
}

• Window resolution: explicit argument, then TOKENSAFE_CONTEXT_WINDOW, then model-family default (Gemini 1M, Claude 200k, GPT 128k).

• 80% session warning: injected once per session by the pre_invocation hook when context usage approaches the threshold.

• Cascading governor & prefix normalizer: aligns prompt prefixes to 64-token boundaries (for Anthropic cache_control and DeepSeek caching) and progressively folds inactive bodies before overflowing.

• Honest accounting: counts only TokenSafe-instrumented flows and labels them estimated (a floor, never a fabricated total; see METRICS.md).

Per-Seat ROI & Persistent Diff-Aware Read-Cache

• Per-Seat ROI (Team): the Team Seats dashboard table tracks saved tokens, derived USD impact, and the measured vs. estimated split per developer in one grouped query. Inactive seats show —, never guessed figures.

• Diff-Aware Read-Cache (SUBAGENT_DEDUP): prevents subagent swarms from reading identical files repeatedly. Entries are keyed by SHA-256 of raw file bytes, so they survive MCP server restarts and git checkout while invalidating immediately on any real edit (bounded at 256k chars per artifact and 500 recent entries).

Universal MCP Server Configuration

Connect TokenSafe to Claude Desktop (claude_desktop_config.json) or Cursor MCP settings:

{
  "mcpServers": {
    "tokensafe": {
      "command": "npx",
      "args": ["-y", "tokensafe", "mcp"]
    }
  }
}

Three Outcomes for Engineering Teams

Giving coding agents raw access to entire repositories creates three daily problems:

1. Broken edits in large files: reading 2,000-line modules for a 10-line fix causes models to forget imports, invent methods, and regress adjacent code.

2. Secret leaks and unsafe shortcuts: static prompt rules in .cursorrules wear off mid-session. Agents hardcode fallback credentials, disable TLS checks, or stage .env files with git add ..

3. Mid-sprint quota stops and blind spend: bloated diffs, lockfiles, and verbose MCP schemas burn 5-hour limits while engineering leads lack per-seat ROI visibility.

Why Stripping Context Bloat Fixes AI Code Quality

TokenSafe started because of a $400 monthly API bill from running autonomous coding agents across active repositories.

Once engineering teams tested it, lower bills turned out to be the secondary benefit. Agents wrote cleaner patches, stopped inventing non-existent methods, and stopped looping on the same broken fix.

Models rarely write bad code because they lack capability. They write bad code when 15,000 lines of irrelevant context drown out the few contracts that matter.

Polyglot AST Engine (10+ Languages)

TokenSafe supports modern polyglot monorepos out of the box. Its deterministic local AST state machine folds non-target function bodies in < 30ms with zero external runtime dependencies, while preserving target symbols and 1-hop dependencies with full executable bodies.

Supported Language Matrix

Key Engine Guarantees

• Design Token & Palette Preservation: When skeletonizing stylesheets, TokenSafe keeps design tokens (:root { ... }, Tailwind v4 @theme { ... }, CSS variables --*, SCSS $vars) intact so agents never invent colors, radii, or font rules.

• Strict Arrow Function Parser: Arrow functions require => before any { counts as a function body. Destructuring parameters (({ a, b }: Props) =>) and parenthesized ternary expressions (const x = (flag) ? { a: 1 } : { b: 2 }) are left intact.

• Directory Tree Optimization (tokensafe:list_dir): Replaces verbose ls -la output with a compact, depth-bounded tree without permissions, timestamps, or lockfiles.

• Graph-Guided Neighborhood Slicing: Calling tokensafe:smart_read_file(path, symbol="Target") preserves the target body and its 1-hop callees/callers while folding unrelated code.

• Comment & String Bracket Immunity: Multi-line block comments (/* ... */, JSX {/* ... */}), template strings (`${...}`), and escaped quotes are tracked by BraceLexer so brackets never desynchronize.

• Control-Flow Immunity: Statements like if (...), catch (err) {, for (...), and while (...) are protected from folding.

The Hidden Context Tax: MCP Tool Schema Bloat

Coding agent quotas (such as 5-hour session windows or monthly token budgets) often drain after a few prompts before real work starts.

The Hidden System Prompt Tax

Whenever an MCP server is enabled in an editor (Cursor, Claude Desktop, Antigravity, Windsurf), the full JSON schema for every tool is injected into the system prompt on every turn, even when unused:

┌────────────────────────────────────────────────────────────────────────┐
│  SYSTEM PROMPT ANATOMY (EVERY SINGLE TURN)                            │
├────────────────────────────────────────────────────────────────────────┤
│  Base System Instructions                                ~1,500 tokens │
│  Active Workspace Skills & Prompts                       ~8,000 tokens │
│  Regular MCP Tools (context7, prisma, etc.)                ~800 tokens │
│  [!] Heavy MCP Tools (200+ schemas from large servers)   ~6,000 tokens │
├────────────────────────────────────────────────────────────────────────┤
│  TOTAL HIDDEN BASE TAX BEFORE YOUR FIRST LINE OF CODE:                 │
│  ~16,300 tokens PER TURN x 15 turns = 244,500 tokens burned            │
└────────────────────────────────────────────────────────────────────────┘

• 1 server with 50 tools = ~1,500 to 2,500 tokens of static tax per turn.

• 1 server with 200+ tools = ~5,000 to 10,000 tokens per turn.

• Across a 15-turn coding session, 75,000 to 150,000 tokens go solely to tool schemas.

How TokenSafe Cuts MCP Overhead

1. tokensafe doctor (Environment Auditor): Scans configured MCP servers and skills, measures token overhead per turn, and flags servers with $\ge 40$ tools as CRITICAL BLOAT.

2. 1-Click MCP Toggle: Enable or disable heavy servers without manual JSON edits:

   tokensafe mcp disable rea     # Prunes heavy schemas from the system prompt
   tokensafe mcp enable rea      # Re-enables when needed

3. Session Start Bloat Alerts: Surfaces warning badges in chat when heavy MCP bloat is active.

4. Telemetry Retention (tokensafe prune / vacuum): Compacts SQLite metrics with rolling-window rollups, reducing local database footprint by up to 40%.

TokenSafe vs. Graphify: In-Flight Defense vs. Static Reports

How TokenSafe complements or replaces static graph tools like Graphify (graphify-labs/graphify):

Complements & Replaces: If you already maintain Graphify reports for architecture docs, TokenSafe still slices live file reads on the wire. Standalone, TokenSafe needs no pre-indexing step.

TokenSafe Guard: Local Security and Hygiene Shield

Coding agents hold shell execution and file write access across your repositories. Passive prompt rules fade over multi-turn sessions, leaving code vulnerable to leaked keys, TLS bypasses, or unsafe staging.

TokenSafe Guard (src/guard.py) audits tool calls locally before disk or git execution with zero token overhead:

1. Secret & Private Key Leak Prevention (Zero Hardcoded Fallbacks)

• Live Provider Secrets: Blocks hardcoded Stripe live keys, AWS access keys, OpenAI project keys, Anthropic tokens, and GitHub personal access tokens.

• Default Fallback Hardcoding: Rejects non-placeholder defaults such as os.environ.get("DB_PASSWORD", "prod_secret_123") or process.env.JWT_SECRET || "production_secret".

• Raw Private Keys: Blocks 64-character raw hex keys and raw PEM private key blocks in source files.

2. Insecure SSL/TLS Verification Bypass Shield

• Blocks verify=False in Python (requests, httpx, aiohttp, urllib3) and rejectUnauthorized: false or NODE_TLS_REJECT_UNAUTHORIZED='0' in Node.js non-test code.

3. Raw SQL Injection Prevention

• Blocks string interpolation inside query execution calls (cursor.execute(f"SELECT ... {user_id}"), cursor.execute("SELECT ... %s" % val), db.query(\SELECT ... ${id}`)`).

• Enforces parameterized queries (?, %s, $1).

4. Dangerous CORS Wildcard with Credentials Shield

• Intercepts configurations combining Access-Control-Allow-Origin: * with Access-Control-Allow-Credentials: true.

5. Git Staging Protection for Sensitive Files

• Explicit Staging: Blocks git add .env, git add .env.*, git add id_rsa, and git add *.pem.

• Blanket Staging Shield: Intercepts git add ., git add -A, and git commit -a whenever unignored .env files or private keys exist in the working tree.

6. Destructive Shell & Data Exfiltration Prevention

• Destructive Deletion: Blocks commands such as rm -rf /, rm -rf ~, rm -rf *, and rm -rf --no-preserve-root.

• Network Exfiltration: Blocks commands piping sensitive local files to remote URLs (such as curl -d @.env https://... or wget --post-file=id_rsa).

7. Package Typosquatting Early Warning

• Checks pip install, npm install, pnpm add, yarn add, and bun add for known typosquat packages (python-crypto, loadsh, coookie-parser, mongose) and prints a TokenSafe Guard Warning badge.

When an operation is blocked, TokenSafe Guard returns an explicit message: 🛡️ **TokenSafe Guard**: <reason>

Prompt Distillation: Managed Cloud and Three-Tier Cache

TokenSafe distills verbose prompts into compact TOCP S-expressions (-40% to -70% tokens).

1. Zero-Config Managed Cloud vs. Self-Hosted Docker BYOK

Zero External Keys for Team & Solo: Developers never have to create or manage personal Google, OpenAI, or DeepSeek billing accounts to use cloud prompt distillation.

2. Token Arbitrage Guard & Three-Tier Cache

To keep distillation overhead lower than the tokens it saves:

1. Smart Gating (llm_distill_min_tokens: 100): Prompts under 100 tokens stay strictly local via regex heuristics ($0$ LLM tokens, $< 1\text{ms}$).

2. Output Token Ceiling (distill_max_tokens: 60): Caps compiled output at 60 tokens to prevent compiler arbitrage loss.

3. Three-Tier Cache:

   • L1 (In-Memory LRU): Sub-millisecond hits inside active sessions.

   • L2 (Persistent SQLite): Stores normalized SHA-256 prompt hashes in ~/.tokensafe/data/metrics.db across restarts so repeated workflows (git commit, run tests, fix lint) cost 0 tokens. The directory is shared by all agents on the machine (override with TOKENSAFE_DATA_DIR; Docker uses /app/data).

   • L3 (Shared Team Cache): Team Tier. Syncs distilled intents and AST skeletons across teammates and CI runners (/api/team/cache) for warm 0ms team cache hits.

3. Provider & Cache Inspection

Inspect your active license tier, distillation mode, and cache stats anytime:

tokensafe providers

Self-hosted VPC deployment (BYOK provider keys, zero-telemetry / air-gapped isolation) is part of the Enterprise & VPC plan — deployment and support are handled by the TokenSafe engineering team. Contact corp@tokensafe.dev.

Calibrated SWE-bench Benchmark Results

Reference run across 8 production engineering scenarios (131,552 baseline tokens down to 7,263 tokens, -94.5% net token reduction, +28.1% turn latency speedup with 100% accuracy retention):

Token counts vary by ±1% between runs because prompt compaction is measured live on every run while baseline sizes are modeled constants. See METRICS.md for the full simulation disclosure and measured-vs-modeled breakdown.

How to read these numbers: This benchmark is a simulated harness. Compression ratios are real, deterministic tool-output deltas on real files, while the "turn speedup" column models agent behavior. Production events in TokenSafe are explicitly labeled measured or estimated (see METRICS.md).

Metrics Honesty

TokenSafe labels every savings event as one of:

• measured: the model literally received fewer bytes than the tool produced (sliced file reads, filtered command output, AST skeletons).

• estimated: the model received a compact artifact and saved tokens are counted against a reasonable baseline (architecture graph vs. a full repo crawl; folded history digest injected while raw history stays in context).

Totals on the dashboard and landing page always display the split (X metered · Y est.). When a live metric is missing, the UI shows — rather than a placeholder number. USD figures are derived approximations (tokens × $1.5 / 1M), not provider invoices. Full contract: METRICS.md.

Pricing and 7-Day Team Trial

TokenSafe follows an open-core model: verify local context savings and guardrails on your own repo first, then roll out team caching, org policies, and per-seat ROI.

Working solo? Individual Solo plans start at $9 / mo ($89 / yr).

Start a 7-Day Team Trial

CLI Quickstart

# View live savings dashboard in your terminal
tokensafe stats

┌────────────────────────────────────────────────────────────────────────┐
│               ⚡ TOKENSAFE CONTEXT OPTIMIZER DASHBOARD ⚡               │
└────────────────────────────────────────────────────────────────────────┘

Total Tokens Saved:   4,524,156 tokens
Efficiency Gain:      [████████████████████████████░░] 92.5%
Estimated Savings:    $6.7865 USD
Context Traffic:      4,889,255 raw ➔ 370,574 optimized (2132 events)

── Breakdown by Source ──────────────────────────────────────────────────
• AST Skeletonizer (Signatures)       :   3,509 tokens saved (2 events)
• Command Sanitizer (CLI)             : 727,300 tokens saved (263 events)
• Context Governor                    :   5,800 tokens saved (29 events)
• Graph-Guided Slicer (Call-Graph)    :  42,420 tokens saved (14 events)
• Architecture Graph (God Nodes)      : 758,256 tokens saved (13 events)
• Multi-turn History Fold             : 2,897,897 tokens saved (471 events)
• Output Token Guard (write_file)     :  42,466 tokens saved (84 events)
• Prompt Compactor (TOCP + Cache)     :  43,502 tokens saved (1255 events)
• File Line Slicer (AST)              :   3,006 tokens saved (1 events)

Run `tokensafe test "<prompt>"` to test compression on custom input.
Run `tokensafe html` to view interactive browser dashboard.

# Open interactive local dark-mode browser dashboard
tokensafe html

# Export a markdown savings report
tokensafe markdown

# Test prompt compression on custom input in real time
tokensafe test "please fix the validate_token method in src/auth.py so that it checks jwt expiration and returns boolean"

# Audit environment and context token tax (MCP & skills)
tokensafe doctor

# List configured LLM distillation providers and models
tokensafe providers

# Analyze repository God Nodes and dependency topology
tokensafe graph

# Generate compact tokensafe-out/API_SURFACE.md interface snapshot
tokensafe map

# Manage or disable high-overhead MCP tools
tokensafe mcp list
tokensafe mcp disable <server>
tokensafe mcp enable <server>

# Optimize SQLite metrics storage
tokensafe vacuum
tokensafe prune

# Generate or preview Friday team ROI digest email
tokensafe digest

# Run the SWE-bench verification suite
tokensafe benchmark

# Run the unit test suite (compact failure report, full: --traceback / -v)
tokensafe tests

# Run the local transparent AI proxy (0.0.0.0:8765/v1)
tokensafe proxy

# Activate your Team or Solo license key (TS2 Asymmetric Ed25519 Key)
tokensafe activate TS2.eyJlbWFpbCI6...

Self-Hosted Docker Deployment

The repository ships a hardened multi-stage Dockerfile (non-root uid: 1000) and a docker-compose.yml for containerized runs, and the local proxy is available directly via tokensafe proxy (0.0.0.0:8765/v1).

Supported, production-grade self-hosting — private VPC, BYOK provider keys (Gemini, DeepSeek, OpenAI, Claude), zero-telemetry / air-gapped mode, SAML/OIDC SSO — is part of the Enterprise & VPC plan and is deployed together with our engineering team: corp@tokensafe.dev.

Security Architecture & Cloud API Endpoints

TokenSafe Cloud Backend enforces strict zero-trust boundaries across client and server requests:

1. Cryptographic Authentication & Licensing

• Asymmetric RFC 8032 Ed25519 Signatures (TS2.*): Client instances verify digital signatures offline using an embedded 32-byte public key. The private signing key never leaves the cloud environment.

• JWT Session Tokens: Cloud endpoints verify sessions via HS256 JWT bearer tokens or Clerk session tokens.

• Dual Bearer Support: Client endpoints accept either a signed license key (TS2.* / TS1.*) or a JWT in Authorization: Bearer <token>.

• Zero Hardcoded Secrets: all server credentials load strictly from runtime environment variables — nothing is hardcoded in the codebase.

2. Cloud API Endpoint Reference

Seat and license management is available through the dashboard. Privacy controls (e.g. TOKENSAFE_TELEMETRY_DISABLED=1 to disable cloud telemetry sync) are documented in METRICS.md.

Unit Testing

python3 -m unittest discover tests

Zero mandatory runtime dependencies: runs on standard Python 3.10+. The optional cryptography package (requirements.txt) accelerates Ed25519 license verification and enables fail-closed RS256 Clerk verification on the cloud backend.

License

MIT License. Commercial licenses are available for self-hosted B2B and enterprise VPC deployments.

Project Structure

agy-token-optimizer/
├── plugin.json                 # Antigravity plugin manifest
├── hooks.json                  # Lifecycle hook declarations (PreInvocation, PreToolUse, PostToolUse)
├── Dockerfile                  # Hardened multi-stage container deployment
├── docker-compose.yml          # Production Docker compose with multi-provider routing
├── rules/
│   └── AGENTS.md               # TOCP Protocol directives for LLM (prompt-cached)
├── skills/
│   └── token-optimizer/
│       └── SKILL.md            # On-demand Antigravity skill
├── proto/
│   └── tocp.proto              # Protocol Buffers contract
├── src/
│   ├── cli.py                  # Unified CLI dispatcher (npm, pip, standalone)
│   ├── cache_normalizer.py     # Prompt Cache Prefix Normalizer (Anthropic & DeepSeek)
│   ├── governor.py             # Context Window Governor & Token Ceiling (Cascading reduction)
│   ├── agent_firewall.py       # Cross-Agent Deduplication Cache (Subagent Firewall)
│   ├── diff_slicer.py          # AST Diff Slicer & Lockfile Stripper (`git diff -U2`)
│   ├── weekly_digest.py        # Friday Team ROI Digest & Engineer Leaderboard
│   ├── environment_auditor.py  # MCP Server & Skill context token tax auditor
│   ├── command_sanitizer.py    # High-output terminal command sanitizer
│   ├── proxy.py                # Multi-provider transparent AI proxy & SSE broadcaster
│   ├── graph_engine.py         # Sub-30ms AST parser, call graphs & God Node centrality
│   ├── tokenizer.py            # Zero-dependency calibrated BPE token counter
│   ├── compactor.py            # Semantic compactor (Local Regex + Distillation -> TOCP S-Expr)
│   ├── distiller.py            # Multi-provider LLM distillation engine (Gemini, DeepSeek, OpenAI)
│   ├── distiller_cache.py      # Two-tier LRU + SQLite prompt distillation cache
│   ├── deepseek_distiller.py   # DeepSeek-Flash semantic compiler with token arbitrage guard
│   ├── slicer.py               # AST & Line windowing tool optimizer
│   ├── metrics.py              # Persistent SQLite metrics engine
│   ├── dashboard.py            # Visual ANSI terminal, Markdown & reactive SSE web dashboard
│   ├── license_manager.py      # Ed25519 cryptographic license verification
│   ├── mcp_server.py           # Native Model Context Protocol (MCP) server
│   ├── installer.py            # Multi-IDE agent auto-installer & config wizard
├── hooks/
│   ├── pre_invocation.py       # PreInvocation hook entry point
│   ├── pre_tool_use.py         # PreToolUse hook entry point
│   └── post_tool_use.py        # PostToolUse hook entry point
├── bin/
│   ├── tokensafe.js            # Node.js npm launcher (npx tokensafe)
│   └── tokensafe               # Python CLI launcher (pip / standalone)
├── tests/                      # Unit test suite
├── install.sh                  # One-click installer for Antigravity
└── README.md