Skip to main content
Glama
token-context-mcp

token-context-mcp

Token Context MCP (v0.3.11)

CI PyPI Python License: MIT

🌐 English: README.md | 🇻🇳 Tiếng Việt: README.vi.md | 🇨🇳 简体中文: README.cn.md | 📝 Prompt Protocol: sample_prompt.md | 📋 Changelog: CHANGELOG.md

NOTE

Architecture Notice: The governance microkernel design is inspired by the Cordis paper (A Programming Paradigm for Spatiotemporal Composability, arXiv:2608.25512). This is an independent open-source implementation not affiliated with or endorsed by DeepSeek.

token-context-mcp is an on-demand, budget-governed Model Context Protocol (MCP) server for autonomous AI coding agents (Claude Code, Cursor, Codex, Google Antigravity). Instead of blindly dumping an entire repository or compressing it into a static snapshot that strips function bodies down to signatures, it provides an interactive AST-level pull pipeline that delivers exact code symbols and downstream impact graphs with verified file:line + SHA-256 evidence.


🖼️ Architecture & Benchmark Infographics (2x Retina Dark Green)

1. 3-Layer Governance Microkernel & 5-Step Progressive Pull Pipeline

2. Context Paradigms: Flat Dump vs. Static Pack vs. Interactive MCP Pull

3. Empirical Benchmarks: 59-Task Head-to-Head & M10 Equal-Budget Suite

4. 7-Dimension Technical Comparison Matrix (vs. ContextCraft v0.3.2)


Related MCP server: Serena

⚡ Core Architectural Pillars

  1. 3-Stage Governance Microkernel (kernel/):

    • Every request passes through a lightweight middleware pipeline:

      • Stage 1 — Identity Resolution (AuthMiddleware): Binds TOKEN_CONTEXT_AGENT_ID when configured and prevents reserved admin impersonation.

      • Stage 2 — Access & Rate Governance (GovernanceMiddleware): Enforces per-agent policies, emergency halt, and sliding-window rate limits.

      • Stage 3 — AST-Boundary Cap & Execution (ExecutionMiddleware + retrieve/): Enforces max_result_tokens at syntactic boundaries without cutting functions in half.

  2. Zero Schema Bloat:

    • The default tools/list payload exposes only 10 core read-only retrieval tools (keeping 11 extension tools—including trajectory_record—and 2 admin tools off the wire until requested). Server-side JSON schemas are slimmed to reduce initial prompt tax.

  3. Incremental Tree-sitter AST & Persistent SQLite Graph (index/):

    • Fast mtime_ns + SHA-256 change detection across Python, JavaScript/TypeScript, Go, Rust, Java, C/C++, and C# with symlink-escape and path traversal guards.

  4. Verified Evidence Ledger ("Lá Phổi" — [ledger].enabled = true by default):

    • Stores verified "query -> answer" citation cards (path:line + AST body SHA-256) in ledger.sqlite outside your repository. Every read re-verifies citations against disk, emits prior_evidence hints (or recheck_from on VERIFY_FAILED), and never prunes unverified code. Zero-hash fast-path adds 0ms overhead on empty ledgers.


📊 Empirical Benchmarks (tiktoken o200k_base Audited)

All numbers are measured using OpenAI's o200k_base tokenizer across two real-world open-source repositories: rich (Python, 213 files, 507k tokens) and fastify (JS/TS, 291 files, 538k tokens):

Benchmark Suite

Metric

token-context-mcp

Baseline / Alternative

Empirical Takeaway

59-Task Head-to-Head (Equal budget: ~7.6k tokens)

Target function in context with Full Body ($\ge 90%$ lines)

33 / 59 (55.9%)

2 / 59 (3.4%) (ContextCraft 8k --focus)

16.5× more full function bodies at equal token budget

59-Task H2H (Sensitivity) (vs. 4× static budget: 32k tokens)

Full Body vs. Signature Only

33 / 59 Full Body (at 7.6k)

6 / 59 Full Body, 40 / 59 (67.8%) Signature Only (ContextCraft 32k)

Static packing prunes bodies to signatures to fit whole-repo maps; interactive MCP pull focuses 100% of budget on the target symbol

M10 Equal-Budget Suite (rich, n=30, ~1.9k tokens)

File Recall & Symbol Recall (95% Bootstrap CI)

0.90 File Recall0.54 Symbol Recall

0.57 File Recall0.10 Symbol Recall (Truncated Grep)

+58% file recall, 5.4× symbol recall over truncated grep at identical token cost

C1 Single-Call Cost (fastify 538k source baseline)

Orientation map get_repo_map@512

682 tokens

538,799 tokens (raw source files)

Compact repo map occupies 0.13% of the repository (~790× smaller)


🚀 Quick Start

Option 1: Run instantly with uvx (Recommended)

# Register, index, and print agent config for the current directory
uvx token-context-mcp quickstart .

Option 2: Install via pip

pip install token-context-mcp

# Register a codebase and launch server
token-context register --repo-id my-project --root /path/to/project
token-context index --repo-id my-project
token-context serve

Desktop GUI Controller (PySide6)

7-tab dark desktop controller (Dashboard, Repositories, Tasks & Graph, Cache & DB with stale snapshot cleanup & ledger.sqlite vacuuming, Ledger & Memory, Agents & Security, and Settings):

uvx --from "token-context-mcp[gui]" token-context-gui

Benchmark & Evaluation Suite ([eval] extra with tiktoken o200k_base)

pip install -e ".[eval]"
python evals/chain_eval.py --output evals/reports/chain_l0_l5.json
python evals/mcp_chain_eval.py --limit-chains 6 --output evals/reports/mcp_chain_eval.json

🤖 Coding Agent Setup

token-context-mcp communicates over local stdio. Add the configuration below to your agent:

1. Claude Code

claude mcp add --transport stdio --scope user token-context -- uvx token-context-mcp serve

2. Cursor

In Cursor Settings > Features > MCP Servers:

  • Name: token-context

  • Type: command

  • Command: uvx token-context-mcp serve

3. Claude Desktop

Add to %APPDATA%\Claude\claude_desktop_config.json (Windows) or ~/.config/Claude/claude_desktop_config.json (macOS/Linux):

{
  "mcpServers": {
    "token-context": {
      "command": "uvx",
      "args": ["token-context-mcp", "serve"]
    }
  }
}

4. Google Antigravity IDE / Gemini CLI

Add to mcp_config.json:

{
  "mcpServers": {
    "token-context": {
      "command": "uvx",
      "args": [
        "token-context-mcp",
        "serve",
        "--output-mode", "text",
        "--schema-profile", "gemini_safe"
      ]
    }
  }
}

🛠 Tool Reference Catalog (10 Default Core Tools)

By default, the server exposes 10 read-only retrieval tools to preserve agent context:

Tool

Primary Purpose

Mode

list_repositories

Lists registered repo_ids (filesystem root paths are never leaked).

Read-only

get_index_status

Returns snapshot metadata, indexed file count, commit SHA, and freshness.

Read-only

get_repo_map

PageRank-based symbol hierarchy within a token budget (@512).

Read-only

find_symbols

Pattern-based symbol matching (*, ? wildcards) with file coordinates.

Read-only

search_source

FTS5 code search returning matched lines and enclosing symbol spans.

Read-only

get_file_skeleton

File outline (imports, class/function signatures with elided bodies).

Read-only

get_symbol_context

Full function/class body, docstrings, and callers/callees (@1024).

Read-only

get_impact_slice

Traverses downstream call graph dependencies before making changes (depth=2).

Read-only

get_module_dependents

Tree-sitter import dependency graph between modules.

Read-only

inspect_symbol

Composite lookup: symbol resolution + full context + 1-hop impact graph.

Read-only

(11 extension tools including SQLite multi-agent memory memory_*, Evidence Ledger trajectory_record, lazy tool discovery *_tools, hardware-adaptive compressor sample_summarize, and 2 admin tools agent_control, audit_logs can be enabled via configuration).


🔒 Security Guarantees & Privacy

  • No Remote Code Execution (RCE): No tool can execute shell commands, eval code, or modify repository source files.

  • 100% Local & Private: All SQLite WAL indexes remain on your machine; zero cloud telemetry or external API calls.

  • Automatic Secret Redaction: Regex-based redaction of API keys, tokens, and private keys ([REDACTED: potential secret]) plus hard denial of sensitive files (.env*, .ssh, *.pem, id_rsa).


📚 Technical Documentation


Maintained by the Token Context MCP community. Licensed under MIT.

Available Tools

10 tools
find_symbolsFind symbolsA
Read-onlyIdempotent

Match declaration names by exact or wildcard (*, ?) pattern (exact first; no bodies/edges; truncated=true if capped). Use search_source for code text or inspect_symbol for body+edges.

Inputs: pattern (required), repo_id (required), kind, limit, +2 more. Returns: data, error, budget, repo_id, +9 more

get_file_skeletonFile skeletonA
Read-onlyIdempotent

Extract ordered imports and public/private declaration headers for one indexed file with bodies elided (content=null if stale; omits tail over budget). Use get_symbol_context for bodies.

Inputs: path (required), repo_id (required), profile, max_tokens, +1 more. Returns: data, error, budget, repo_id, +9 more

get_impact_sliceImpact candidate sliceA
Read-onlyIdempotent

Traverse 0-3 hop AST caller/callee edges from a known symbol_id; candidate slice only (truncated=true if capped by max_nodes/max_tokens). For imports use get_module_dependents.

Inputs: repo_id (required), symbol_id (required), depth, profile, +5 more. Returns: data, error, budget, repo_id, +10 more

get_index_statusIndex statusA
Read-onlyIdempotent

Check snapshot freshness ('fresh', 'stale', 'pending', 'unknown'), counts, and pending_paths after list_repositories. Run before trusting get_repo_map or graph edges; treat stale spans cautiously.

Inputs: repo_id (required), verbose. Returns: data, error, budget, repo_id, +10 more

get_module_dependentsModule dependentsA
Read-onlyIdempotent

File/module lexical import graph (imports & imported_by) for one path or module; flags dynamic imports. Fails if unindexed. For symbol call edges, use get_impact_slice instead.

Inputs: repo_id (required), path, module, profile, +1 more. Returns: data, error, budget, repo_id, +9 more

get_repo_mapRepository mapA
Read-onlyIdempotent

Ranked symbol map within budget (format='compact' tuples, 'grouped' by file, or 'full' objects; omitted_count tracks cuts). query biases graph rank; use find_symbols for exact names.

Inputs: repo_id (required), query, format, profile, +4 more. Returns: data, error, budget, repo_id, +9 more

get_symbol_contextSymbol contextA
Read-onlyIdempotent

Fetch bounded body and 0-3 hop edges for a known symbol_id (use body_offset_line to page long bodies; omits body if over max_tokens). If symbol_id is unknown, call inspect_symbol.

Inputs: repo_id (required), symbol_id (required), depth, profile, +4 more. Returns: data, error, budget, repo_id, +10 more

inspect_symbolInspect symbol (composite)A
Read-onlyIdempotent

1-turn lookup by exact/wildcard name when symbol_id is unknown (use find_symbols for bulk lists, get_symbol_context by ID); returns status='not_found', 'ambiguous', or view='full' packet.

Inputs: query (required), repo_id (required), view, budget_tokens. Returns: data, error, budget, repo_id, +9 more

list_repositoriesRegistered repositoriesA
Read-onlyIdempotent

List registered repo_ids and budget_profiles (empty list if none; roots hidden). Call first before any repo tool; call get_index_status next to verify snapshot freshness.

Inputs: no named parameters. Returns: error, repo_ids, truncated, schema_version, +1 more

search_sourceSearch sourceA
Read-onlyIdempotent

Full-text FTS5 body search returning line snippets (omitted_count on budget cap). For symbol name patterns use find_symbols; for multi-hop call graphs use get_impact_slice.

Inputs: query (required), repo_id (required), limit, expand, +5 more. Returns: data, error, budget, repo_id, +9 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.2/5.0

    Scored across 10 tools

    Disambiguation4/5

    Each tool has a documented niche and the descriptions actively redirect the agent (e.g. get_module_dependents says use get_impact_slice for symbol edges, find_symbols says use search_source for code text), which resolves most overlap. The residual ambiguity is between name-based lookups (find_symbols vs inspect_symbol) and between get_symbol_context vs get_impact_slice, where the boundary is 'bulk vs single' and 'body+edges vs edges only' — learnable but easy to misselect on a first pass.

    Naming Consistency5/5

    All ten tools follow a strict snake_case verb_noun pattern (list_repositories, get_repo_map, find_symbols, inspect_symbol, search_source). The verb choice is also semantically meaningful — get for retrieval, find for pattern matching, search for full-text, inspect for a one-shot packet — rather than arbitrary variation.

    Tool Count5/5

    Ten tools is squarely in the well-scoped range for a code-context/indexing server. Each tool covers a distinct retrieval granularity (repo, file, module graph, symbol, source text) and none feels like filler.

    Completeness4/5

    The read-only surface is broad: repo discovery, freshness checking, ranked maps, name matching, file skeletons, symbol bodies, call edges, import graphs, and full-text search — a full navigation lifecycle for the stated purpose. Gaps are minor (no cross-repo search, no way to retrieve a whole file body, budget profiles are only listable, not configurable), but agents can work around them by composing tools.

    Maintenance

    ActivityActive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      F
      maintenance
      A Model Context Protocol (MCP) server that provides code analysis capabilities using tree-sitter, designed to give Claude intelligent access to codebases with appropriate context management.
      26
      625 PyPI
      312
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      A fully featured coding agent that uses symbolic operations (enabled by language servers) and works well even in large code bases. Essentially a free to use alternative to Cursor and Windsurf Agents, Cline, Roo Code and others.
      29
      31,250 PyPI
      31,560
      GPL 3.0
    • A
      license
      A
      quality
      B
      maintenance
      Enables AI assistants to search and analyze codebases using Abstract Syntax Tree (AST) pattern matching with ast-grep. Supports structural code search, pattern testing, and AST visualization across multiple programming languages.
      4
      500
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Local, read-only MCP server for coding agents. Indexes repeated implementations in your repository (FastAPI routes, pytest fixtures, Pydantic and SQLAlchemy models) and returns source-backed examples with a bounded read plan, or a typed UNKNOWN when evidence is insufficient. No LLM or network calls.
      1
      13 npm
      MIT