token-context-mcp
It’s a read-only, budget-governed MCP server for querying an indexed code repository and understanding symbols, source, imports, and call-graph impact.
List registered repositories and check index freshness/status.
Get a ranked repository symbol map within a token budget, optionally query-biased.
Find symbols by exact or wildcard name, with kind/limit filters.
Full-text search source code and optionally expand 1–2 hop callers/callees.
Get a file skeleton with imports/signatures, or full symbol bodies and neighborhoods by symbol ID.
Traverse caller/callee impact slices and module import/dependent relationships.
Inspect a symbol by name in one turn to resolve ambiguous/not-found cases and retrieve context/relationships.
Use token-budget profiles, truncation diagnostics, SHA-256 evidence, freshness warnings, and secret redaction; it cannot execute commands or modify files.
Provides hardware-aware LLM sampling via Ollama auto-routing, enabling summarization tasks to be delegated to a local Ollama instance with deterministic heuristic fallback.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@token-context-mcp Provide a symbol overview of the video-lecturer repo"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Token Context MCP (v0.3.11)
🌐 English:
README.md| 🇻🇳 Tiếng Việt:README.vi.md| 🇨🇳 简体中文:README.cn.md| 📝 Prompt Protocol:sample_prompt.md| 📋 Changelog:CHANGELOG.md
Architecture Notice: The governance microkernel design is inspired by the Cordis paper (A Programming Paradigm for Spatiotemporal Composability, arXiv:2608.25512). This is an independent open-source implementation not affiliated with or endorsed by DeepSeek.
token-context-mcp is an on-demand, budget-governed Model Context Protocol (MCP) server for autonomous AI coding agents (Claude Code, Cursor, Codex, Google Antigravity). Instead of blindly dumping an entire repository or compressing it into a static snapshot that strips function bodies down to signatures, it provides an interactive AST-level pull pipeline that delivers exact code symbols and downstream impact graphs with verified file:line + SHA-256 evidence.
🖼️ Architecture & Benchmark Infographics (2x Retina Dark Green)
1. 3-Layer Governance Microkernel & 5-Step Progressive Pull Pipeline
2. Context Paradigms: Flat Dump vs. Static Pack vs. Interactive MCP Pull
3. Empirical Benchmarks: 59-Task Head-to-Head & M10 Equal-Budget Suite
4. 7-Dimension Technical Comparison Matrix (vs. ContextCraft v0.3.2)
Related MCP server: Serena
⚡ Core Architectural Pillars
3-Stage Governance Microkernel (
kernel/):Every request passes through a lightweight middleware pipeline:
Stage 1 — Identity Resolution (
AuthMiddleware): BindsTOKEN_CONTEXT_AGENT_IDwhen configured and prevents reservedadminimpersonation.Stage 2 — Access & Rate Governance (
GovernanceMiddleware): Enforces per-agent policies, emergency halt, and sliding-window rate limits.Stage 3 — AST-Boundary Cap & Execution (
ExecutionMiddleware+retrieve/): Enforcesmax_result_tokensat syntactic boundaries without cutting functions in half.
Zero Schema Bloat:
The default
tools/listpayload exposes only 10 core read-only retrieval tools (keeping 11 extension tools—includingtrajectory_record—and 2 admin tools off the wire until requested). Server-side JSON schemas are slimmed to reduce initial prompt tax.
Incremental Tree-sitter AST & Persistent SQLite Graph (
index/):Fast
mtime_ns+ SHA-256 change detection across Python, JavaScript/TypeScript, Go, Rust, Java, C/C++, and C# with symlink-escape and path traversal guards.
Verified Evidence Ledger ("Lá Phổi" —
[ledger].enabled = trueby default):Stores verified
"query -> answer"citation cards (path:line+ AST body SHA-256) inledger.sqliteoutside your repository. Every read re-verifies citations against disk, emitsprior_evidencehints (orrecheck_fromonVERIFY_FAILED), and never prunes unverified code. Zero-hash fast-path adds 0ms overhead on empty ledgers.
📊 Empirical Benchmarks (tiktoken o200k_base Audited)
All numbers are measured using OpenAI's o200k_base tokenizer across two real-world open-source repositories: rich (Python, 213 files, 507k tokens) and fastify (JS/TS, 291 files, 538k tokens):
Benchmark Suite | Metric | token-context-mcp | Baseline / Alternative | Empirical Takeaway |
59-Task Head-to-Head (Equal budget: ~7.6k tokens) | Target function in context with Full Body ($\ge 90%$ lines) | 33 / 59 (55.9%) | 2 / 59 (3.4%) (ContextCraft 8k | 16.5× more full function bodies at equal token budget |
59-Task H2H (Sensitivity) (vs. 4× static budget: 32k tokens) | Full Body vs. Signature Only | 33 / 59 Full Body (at 7.6k) | 6 / 59 Full Body, 40 / 59 (67.8%) Signature Only (ContextCraft 32k) | Static packing prunes bodies to signatures to fit whole-repo maps; interactive MCP pull focuses 100% of budget on the target symbol |
M10 Equal-Budget Suite ( | File Recall & Symbol Recall (95% Bootstrap CI) | 0.90 File Recall0.54 Symbol Recall | 0.57 File Recall0.10 Symbol Recall (Truncated Grep) | +58% file recall, 5.4× symbol recall over truncated grep at identical token cost |
C1 Single-Call Cost ( | Orientation map | 682 tokens | 538,799 tokens (raw source files) | Compact repo map occupies 0.13% of the repository (~790× smaller) |
🚀 Quick Start
Option 1: Run instantly with uvx (Recommended)
# Register, index, and print agent config for the current directory
uvx token-context-mcp quickstart .Option 2: Install via pip
pip install token-context-mcp
# Register a codebase and launch server
token-context register --repo-id my-project --root /path/to/project
token-context index --repo-id my-project
token-context serveDesktop GUI Controller (PySide6)
7-tab dark desktop controller (Dashboard, Repositories, Tasks & Graph, Cache & DB with stale snapshot cleanup & ledger.sqlite vacuuming, Ledger & Memory, Agents & Security, and Settings):
uvx --from "token-context-mcp[gui]" token-context-guiBenchmark & Evaluation Suite ([eval] extra with tiktoken o200k_base)
pip install -e ".[eval]"
python evals/chain_eval.py --output evals/reports/chain_l0_l5.json
python evals/mcp_chain_eval.py --limit-chains 6 --output evals/reports/mcp_chain_eval.json🤖 Coding Agent Setup
token-context-mcp communicates over local stdio. Add the configuration below to your agent:
1. Claude Code
claude mcp add --transport stdio --scope user token-context -- uvx token-context-mcp serve2. Cursor
In Cursor Settings > Features > MCP Servers:
Name:
token-contextType:
commandCommand:
uvx token-context-mcp serve
3. Claude Desktop
Add to %APPDATA%\Claude\claude_desktop_config.json (Windows) or ~/.config/Claude/claude_desktop_config.json (macOS/Linux):
{
"mcpServers": {
"token-context": {
"command": "uvx",
"args": ["token-context-mcp", "serve"]
}
}
}4. Google Antigravity IDE / Gemini CLI
Add to mcp_config.json:
{
"mcpServers": {
"token-context": {
"command": "uvx",
"args": [
"token-context-mcp",
"serve",
"--output-mode", "text",
"--schema-profile", "gemini_safe"
]
}
}
}🛠 Tool Reference Catalog (10 Default Core Tools)
By default, the server exposes 10 read-only retrieval tools to preserve agent context:
Tool | Primary Purpose | Mode |
| Lists registered | Read-only |
| Returns snapshot metadata, indexed file count, commit SHA, and freshness. | Read-only |
| PageRank-based symbol hierarchy within a token budget ( | Read-only |
| Pattern-based symbol matching ( | Read-only |
| FTS5 code search returning matched lines and enclosing symbol spans. | Read-only |
| File outline (imports, class/function signatures with elided bodies). | Read-only |
| Full function/class body, docstrings, and callers/callees ( | Read-only |
| Traverses downstream call graph dependencies before making changes ( | Read-only |
| Tree-sitter import dependency graph between modules. | Read-only |
| Composite lookup: symbol resolution + full context + 1-hop impact graph. | Read-only |
(11 extension tools including SQLite multi-agent memory memory_*, Evidence Ledger trajectory_record, lazy tool discovery *_tools, hardware-adaptive compressor sample_summarize, and 2 admin tools agent_control, audit_logs can be enabled via configuration).
🔒 Security Guarantees & Privacy
No Remote Code Execution (RCE): No tool can execute shell commands, eval code, or modify repository source files.
100% Local & Private: All SQLite WAL indexes remain on your machine; zero cloud telemetry or external API calls.
Automatic Secret Redaction: Regex-based redaction of API keys, tokens, and private keys (
[REDACTED: potential secret]) plus hard denial of sensitive files (.env*,.ssh,*.pem,id_rsa).
📚 Technical Documentation
sample_prompt.md: Standard two-phase prompt templates (EN / VI / CN).CHANGELOG.md: Release history and upgrade notes.launch/00_fact_sheet.md: Verified empirical benchmarks and measurement methodologies.docs/CLIENT_MATRIX.md: Step-by-step MCP client verification matrix.
Maintained by the Token Context MCP community. Licensed under MIT.
Available Tools
10 toolsfind_symbolsFind symbolsARead-onlyIdempotent
Match declaration names by exact or wildcard (*, ?) pattern (exact first; no bodies/edges; truncated=true if capped). Use search_source for code text or inspect_symbol for body+edges.
Inputs: pattern (required), repo_id (required), kind, limit, +2 more. Returns: data, error, budget, repo_id, +9 more
get_file_skeletonFile skeletonARead-onlyIdempotent
Extract ordered imports and public/private declaration headers for one indexed file with bodies elided (content=null if stale; omits tail over budget). Use get_symbol_context for bodies.
Inputs: path (required), repo_id (required), profile, max_tokens, +1 more. Returns: data, error, budget, repo_id, +9 more
get_impact_sliceImpact candidate sliceARead-onlyIdempotent
Traverse 0-3 hop AST caller/callee edges from a known symbol_id; candidate slice only (truncated=true if capped by max_nodes/max_tokens). For imports use get_module_dependents.
Inputs: repo_id (required), symbol_id (required), depth, profile, +5 more. Returns: data, error, budget, repo_id, +10 more
get_index_statusIndex statusARead-onlyIdempotent
Check snapshot freshness ('fresh', 'stale', 'pending', 'unknown'), counts, and pending_paths after list_repositories. Run before trusting get_repo_map or graph edges; treat stale spans cautiously.
Inputs: repo_id (required), verbose. Returns: data, error, budget, repo_id, +10 more
get_module_dependentsModule dependentsARead-onlyIdempotent
File/module lexical import graph (imports & imported_by) for one path or module; flags dynamic imports. Fails if unindexed. For symbol call edges, use get_impact_slice instead.
Inputs: repo_id (required), path, module, profile, +1 more. Returns: data, error, budget, repo_id, +9 more
get_repo_mapRepository mapARead-onlyIdempotent
Ranked symbol map within budget (format='compact' tuples, 'grouped' by file, or 'full' objects; omitted_count tracks cuts). query biases graph rank; use find_symbols for exact names.
Inputs: repo_id (required), query, format, profile, +4 more. Returns: data, error, budget, repo_id, +9 more
get_symbol_contextSymbol contextARead-onlyIdempotent
Fetch bounded body and 0-3 hop edges for a known symbol_id (use body_offset_line to page long bodies; omits body if over max_tokens). If symbol_id is unknown, call inspect_symbol.
Inputs: repo_id (required), symbol_id (required), depth, profile, +4 more. Returns: data, error, budget, repo_id, +10 more
inspect_symbolInspect symbol (composite)ARead-onlyIdempotent
1-turn lookup by exact/wildcard name when symbol_id is unknown (use find_symbols for bulk lists, get_symbol_context by ID); returns status='not_found', 'ambiguous', or view='full' packet.
Inputs: query (required), repo_id (required), view, budget_tokens. Returns: data, error, budget, repo_id, +9 more
list_repositoriesRegistered repositoriesARead-onlyIdempotent
List registered repo_ids and budget_profiles (empty list if none; roots hidden). Call first before any repo tool; call get_index_status next to verify snapshot freshness.
Inputs: no named parameters. Returns: error, repo_ids, truncated, schema_version, +1 more
search_sourceSearch sourceARead-onlyIdempotent
Full-text FTS5 body search returning line snippets (omitted_count on budget cap). For symbol name patterns use find_symbols; for multi-hop call graphs use get_impact_slice.
Inputs: query (required), repo_id (required), limit, expand, +5 more. Returns: data, error, budget, repo_id, +9 more
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 10 tools
Each tool has a documented niche and the descriptions actively redirect the agent (e.g. get_module_dependents says use get_impact_slice for symbol edges, find_symbols says use search_source for code text), which resolves most overlap. The residual ambiguity is between name-based lookups (find_symbols vs inspect_symbol) and between get_symbol_context vs get_impact_slice, where the boundary is 'bulk vs single' and 'body+edges vs edges only' — learnable but easy to misselect on a first pass.
All ten tools follow a strict snake_case verb_noun pattern (list_repositories, get_repo_map, find_symbols, inspect_symbol, search_source). The verb choice is also semantically meaningful — get for retrieval, find for pattern matching, search for full-text, inspect for a one-shot packet — rather than arbitrary variation.
Ten tools is squarely in the well-scoped range for a code-context/indexing server. Each tool covers a distinct retrieval granularity (repo, file, module graph, symbol, source text) and none feels like filler.
The read-only surface is broad: repo discovery, freshness checking, ranked maps, name matching, file skeletons, symbol bodies, call edges, import graphs, and full-text search — a full navigation lifecycle for the stated purpose. Gaps are minor (no cross-repo search, no way to retrieve a whole file body, budget profiles are only listable, not configurable), but agents can work around them by composing tools.
Maintenance
Related MCP Connectors
Cache source-backed repository context so coding agents can reuse knowledge across sessions.
Hosted code graph over MCP: exact callers, dependencies, and cross-repo blast radius for AI agents.
Codebase intelligence for agents: 152 structured artifacts across 21 programs, one call.
Team memory for AI coding agents and code reviews, reviewed by humans and served over MCP.
Related MCP Servers
- AlicenseBqualityFmaintenanceA Model Context Protocol (MCP) server that provides code analysis capabilities using tree-sitter, designed to give Claude intelligent access to codebases with appropriate context management.26625 PyPI312MIT
- AlicenseAqualityBmaintenanceA fully featured coding agent that uses symbolic operations (enabled by language servers) and works well even in large code bases. Essentially a free to use alternative to Cursor and Windsurf Agents, Cline, Roo Code and others.2931,250 PyPI31,560GPL 3.0
- AlicenseAqualityBmaintenanceEnables AI assistants to search and analyze codebases using Abstract Syntax Tree (AST) pattern matching with ast-grep. Supports structural code search, pattern testing, and AST visualization across multiple programming languages.4500MIT
- AlicenseAqualityBmaintenanceLocal, read-only MCP server for coding agents. Indexes repeated implementations in your repository (FastAPI routes, pytest fixtures, Pydantic and SQLAlchemy models) and returns source-backed examples with a bounded read plan, or a typed UNKNOWN when evidence is insufficient. No LLM or network calls.113 npmMIT