Skip to main content
Glama
ajdev0

token-shrink

by ajdev0

token-shrink

A local-first, framework-aware token reduction engine — a polyglot AST semantic proxy and MCP server. It prunes implementation bodies out of dependency files while preserving every type signature, interface, and module export, so LLM agents see the full shape of the code at a fraction of the tokens.

The 80–90% reduction target: full type information, no implementation noise. Ring 0 (your active file) stays complete; Ring 1 (its direct imports) is delivered as pruned skeletons.


How it works

       active file                       imports (Ring 1)
  ┌──────────────────┐       ┌──────────────────────┐
  │  src/page.ts     │  ──►  │  src/util.ts         │
  └──────────────────┘       └──────────────────────┘
          ▾                              ▾
  tree-sitter (WASM) ─────────────► prune impl blocks
       parse & query               keep interfaces · types ·
                                   signatures · exports
                                          ▾
                                  pruned skeleton (Ring 0 full source)
                                          ▾
                          Compressed Code Context (Markdown)
                                  │               │
                            via MCP (stdio)   via HTTP (Fastify)
                      get_compressed_code_context  POST /v1/context

Pipeline stages:

  1. Parseweb-tree-sitter loads a .wasm grammar per language (auto-downloaded on first run).

  2. Prune — an S-expression query matches implementation blocks (statement_block, block, compound_statement…), which are replaced with a short token (/* ... */, or pass for Python) using descending-order splicing so offsets stay valid.

  3. Watchchokidar watches the repo, sha1-hashes file contents, and refreshes the cache only on change.

  4. Assemble — the active file's imports are resolved and merged into a Markdown context payload (Ring 0 + Ring 1).


Related MCP server: MCP File Compaction

Install

Requires Node.js 18+.

# run anywhere without installing
#   --root   project root   --port   http port   --host   bind address
npx @ajdev0/token-shrink --root /path/to/project

# or install locally
npm install @ajdev0/token-shrink

Build from source

# install deps
npm install

# compile (tsup -> dist/), typecheck, and run tests
npm run build
npm run typecheck
npm test

The build produces three binaries:

Binary

Entry

Purpose

@ajdev0/token-shrink

dist/cli.cjs

Fastify HTTP server (POST /v1/context)

@ajdev0/token-shrink-mcp

dist/mcp.cjs

MCP stdio server for AI agents

library

dist/index.js

prune(), assemble(), createWatcher()

Publish to npm

npm login
npm run build && npm test
npm pack --dry-run    # preview tarball contents
npm publish

The package name on npm is @ajdev0/token-shrink. After publishing, users can run:

npx @ajdev0/token-shrink-mcp --root /path/to/project

WASM grammars (auto-download)

Grammars are fetched from the official tree-sitter GitHub releases on first use and cached in wasm/:

wasm/
├── tree-sitter-typescript.wasm
├── tree-sitter-javascript.wasm
├── tree-sitter-tsx.wasm
├── tree-sitter-python.wasm
├── tree-sitter-go.wasm
├── ...
  • First run requires network access; afterwards everything is offline and fast.

  • Files are written atomically (*.tmp → rename) with an in-flight lock, so concurrent first-run parses never corrupt the cache.


Usage

1. MCP server (AI agents — Cursor, Claude, Cline, etc.)

Run the stdio MCP server and expose the get_compressed_code_context tool:

# point it at your project
token-shrink-mcp --root /path/to/project

# root also works via env or cwd
ROOT=/path/to/project token-shrink-mcp
cd /path/to/project && token-shrink-mcp

Cursor MCP config (.cursor/mcp.json):

{
  "mcpServers": {
    "token-shrink": {
      "command": "token-shrink-mcp",
      "args": ["--root", "/absolute/path/to/your/project"]
    }
  }
}

Claude Code MCP config — add it to the project's .mcp.json, or register with the Claude CLI:

# register the server for this project
claude mcp add token-shrink -- token-shrink-mcp --root /path/to/project
# persistent flag: -- transport stdio
claude mcp add token-shrink --transport stdio -- token-shrink-mcp --root /path/to/project

or place in .claude/settings.json / project .mcp.json:

{
  "mcpServers": {
    "token-shrink": {
      "command": "token-shrink-mcp",
      "args": ["--root", "/path/to/project"]
    }
  }
}

Cline MCP config — add it to the project's .mcp.json (or mcp.json in the .cline settings directory), or add the server via the Cline UI (MCP Servers → Configure MCP Servers):

{
  "mcpServers": {
    "token-shrink": {
      "command": "token-shrink-mcp",
      "args": ["--root", "/path/to/project"]
    }
  }
}

Auto rule: by default the server writes agent integration rules so the tool is used automatically on every prompt:

  • Cursor: .cursor/rules/token-shrink.mdc

  • Claude Code: .claude/rules/token-shrink.md

  • Cline: .clinerules/token-shrink.md (Cline's .clinerules/ directory — every .md/.txt file there is loaded on every task)

All are sentinel-tagged and never rewrite a user-authored file at the same path. Repeated starts are no-ops. Choose the target(s) with --rule-target=cursor|claude|cline|all (default all, comma-separated values allowed):

# only Claude Code
token-shrink-mcp --root /path/to/project --rule-target=claude

# Cursor + Cline, no Claude rule
token-shrink-mcp --root /path/to/project --rule-target=cursor,cline

# completely disable auto-rules
token-shrink-mcp --root /path/to/project --no-create-rule

Opt out also via --create-rule=false or TOKEN_SHRINK_CREATE_RULE=0.

Tool: get_compressed_code_context

Argument

Type

Required

Description

activeFilePath

string

yes

The file the agent is working on

maxSkeletons

number

no

Cap on Ring-1 files (default 50, max 200)

includeStats

boolean

no

Append approximate token counts

Returns a Markdown payload with the active file fully inlined (Ring 0) and the pruned skeletons of its direct imports (Ring 1).

2. HTTP server (Fastify)

token-shrink --root /path/to/project --port 3000
# env equivalents: ROOT=… PORT=… HOST=…

Route

Method

Body

Returns

/health

GET

status, root, indexed file count

/v1/context

POST

{ activeFilePath, maxSkeletons?, includeStats? }

assembled Markdown + deps

curl -s http://localhost:3000/health
# {"status":"ok","service":"token-shrink","version":"2.0.0","root":".","indexed":182}

curl -s -X POST http://localhost:3000/v1/context \
  -H 'Content-Type: application/json' \
  -d '{"activeFilePath":"./src/page.ts","includeStats":true}'

3. Library API

import { prune, assemble, createWatcher } from 'token-shrink';

// prune a single file -> skeleton (keeps signatures, strips bodies)
const { code, removed } = await prune('src/util.ts', sourceText);

// assemble context for an active file from a warm cache
const { markdown } = assemble('src/page.ts', watcher.cache.entries, {
  includeStats: true,
});

// incremental watcher
const watcher = createWatcher({ root: process.cwd(), ignored: ['node_modules'] });
await watcher.indexAll();

Supported languages

S-expression queries match implementation blocks; interfaces, signatures, and exports are never touched. The Block node column shows the AST node that gets collapsed during pruning.

Language

Extensions

Grammar wasm

Block node

TypeScript

.ts .cts .mts

tree-sitter-typescript.wasm

statement_block

JavaScript

.js .cjs .mjs

tree-sitter-javascript.wasm

statement_block

React / Next.js

.tsx

tree-sitter-tsx.wasm

statement_block¹

React (JSX)

.jsx

tree-sitter-javascript.wasm

statement_block¹

Python

.py .pyi

tree-sitter-python.wasm

blockpass

Dart / Flutter

.dart

tree-sitter-dart.wasm

block

Swift / SwiftUI

.swift

tree-sitter-swift.wasm

statements

Go

.go

tree-sitter-go.wasm

block

Rust

.rs

tree-sitter-rust.wasm

block

Java

.java

tree-sitter-java.wasm

block

Kotlin

.kt .kts

tree-sitter-kotlin.wasm

block

C

.c .h

tree-sitter-c.wasm

compound_statement

C++

.cc .cpp .cxx .hpp .hh .hxx

tree-sitter-cpp.wasm

compound_statement

PHP

.php

tree-sitter-php.wasm

compound_statement

¹ TSX/JSX also preserve 'use client' / 'use server' directive lines inside otherwise-pruned bodies (framework-aware).

Language IDs: typescript · javascript · tsx · jsx · python · dart · swift · go · rust · java · kotlin · c · cpp · php.


Example

Input src/util.ts

export interface User {
  id: number;
  name: string;
}
export function buildGreeting(u: User) {
  const parts = [u.name, u.email];
  return parts.join(' | ');
}
export const formatEmail = (u: User) => {
  return u.email.toLowerCase().trim();
};

Pruned skeleton (Ring 1) — signatures and the interface intact, bodies collapsed:

export interface User {
  id: number;
  name: string;
}
export function buildGreeting(u: User) /* ... */
export const formatEmail = (u: User) => /* ... */;

Design notes

  • Bottom-up splicing — ranges are sorted by start index descending and replaced in place, so earlier offsets never shift and the output stays a valid, parseable file.

  • Regex-based import extraction — resilient across languages; resolves relative imports (./x, ../y), aliases (@/, ~), and skips bare package specifiers.

  • Incremental hashing — files are re-pruned only when their sha1 hash changes; the watcher is debounced (100 ms) and zero-CPU while idle.

  • Ram-safe watchers — sockets / non-regular files are never opened with fs.watch, so stray unix sockets in the tree can't crash the server.


Project layout

token-shrink/
├── package.json / tsconfig.json / tsup.config.ts / vitest.config.ts
├── src/
│   ├── index.ts             # library entry (exports)
│   ├── cli.ts               # Fastify HTTP server
│   ├── mcp.ts               # MCP stdio server
│   ├── parser/
│   │   ├── registry.ts      # extension → language spec + S-queries
│   │   ├── wasm.ts          # auto-download + cache of .wasm files
│   │   └── pruner.ts        # prune(filePath, source) → skeleton
│   ├── watcher/
│   │   └── sync.ts          # chokidar watch + hash cache + import graph
│   └── server/
│       └── assembler.ts     # Ring 0 + Ring 1 Markdown payload
├── tests/                   # pruning integrity + token-reduction tests
└── wasm/                    # auto-downloaded grammars (gitignored)

License

MIT

Maintenance

ActivityMaintained
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    D
    maintenance
    Provides intelligent code context and analysis through semantic compression, AST parsing, and multi-language support. Offers 60-80% token reduction while enabling AI assistants to understand codebases through local analysis, OpenAI-enhanced insights, and GitHub repository integration.
    6
    22
    3
    MIT
  • A
    license
    A
    quality
    D
    maintenance
    Provides code-aware context compression by stripping comments, docstrings, and whitespace while maintaining full logic fidelity for AI agents. It features tools for architectural mapping, symbol searching, and token-budgeted multi-file reading.
    9
    10
    4
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/ajdev0/token-shrink'

If you have feedback or need assistance with the MCP directory API, please join our Discord server