tokenizer-mcp
Counts tokens for OpenAI models and raw tiktoken encodings such as o200k_base using tiktoken.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@tokenizer-mcp Count tokens in src/index.js"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
tokenizer-mcp
tokenizer-mcp is a small MCP server that lets an LLM harness like Claude Code count the exact tokens in files (or any string) trivially easily, plus report basic file metrics — lines, characters, and size in KB. Every tool returns a single integer. For token counting the server receives the file/text and a model name and routes to the appropriate backend; the file-metric tools need no tokenization backend at all.
Installation
uv syncRelated MCP server: File Tools MCP Server
Running it directly
uv run server.pyAPI key setup
Counting tokens for Claude requires an Anthropic API key, since the exact count comes from messages.count_tokens. Without a key, the Claude path silently falls back to tiktoken's o200k_base encoding. GPT counts work offline and need no key.
Set the key as an environment variable:
export ANTHROPIC_API_KEY=sk-ant-...On Windows (PowerShell):
$env:ANTHROPIC_API_KEY = "sk-ant-..."Or drop a .env file next to server.py and the server will load it on startup:
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_TOKEN_COUNT_MODEL=claude-opus-4-8ANTHROPIC_TOKEN_COUNT_MODEL sets the default Claude model when the caller omits one; it defaults to claude-opus-4-8. The endpoint tokenizes under the model you pass, so use a current model — the tokenizer changed at Opus 4.7 (~30% more tokens than older models for the same text).
What it exposes
Six tools, each returning an integer.
Token counting (uses the model routing / backends described below):
count_tokens(text, model="")— count tokens in a string.count_tokens_file(file_path, model="")— count tokens in a UTF-8 file at an absolute path.count_tokens_folder(folder_path, model="")— sum of per-file token counts across every text file in a folder, recursively. Dependency/VCS/build directories (.git,node_modules,.venv,__pycache__,dist, ...) are excluded, and binary files are skipped — first by extension, then by a content sniff (NUL bytes / control-character ratio) for unrecognized extensions. Files that aren't UTF-8 are decoded as UTF-16 (when a BOM is present) or cp1252 rather than skipped.
When model is omitted, these use ANTHROPIC_TOKEN_COUNT_MODEL.
File metrics (no tokenization backend; file-only — there are deliberately no raw-string variants):
count_lines_file(file_path)— number of lines, using Pythonstr.splitlines()semantics (a trailing newline terminates the last line rather than adding an empty one).count_chars_file(file_path)— number of characters in the decoded UTF-8 text (Unicode code points).count_kb_file(file_path)— file size in kilobytes,ceil(bytes / 1024)with 1 KB = 1024 bytes (the same convention as the Tokenizer app). The size is read from the filesystem, so it works for any file regardless of encoding.
These are useful when a file's token count is only one of several limits — lines, characters, raw size — that decide whether it fits a given budget (e.g. a harness tool-call payload).
How model routing works
The lowercased model is matched against a short set of rules:
What you pass | Backend used |
A raw tiktoken encoding ( |
|
Anything starting with an OpenAI prefix ( |
|
Everything else (Claude models) | Anthropic |
Wiring it into an MCP client
Add an mcpServers entry pointing at this directory:
{
"mcpServers": {
"tokenizer": {
"command": "uv",
"args": ["--directory", "/absolute/path/to/tokenizer-mcp", "run", "server.py"],
"env": {
"ANTHROPIC_API_KEY": "sk-ant-..."
}
}
}
}The env block is optional; omit it and Claude counts fall back to o200k_base, as above.
Available Tools
6 toolscount_chars_fileA
Count characters (Unicode code points) in a UTF-8 file.
Args:
file_path: Absolute path to the file.
Inputs: file_path (required). Returns: result
count_kb_fileA
File size in KB as ceil(bytes / 1024), matching Windows Explorer.
Args:
file_path: Absolute path to the file.
Inputs: file_path (required). Returns: result
count_lines_fileA
Count lines in a file (str.splitlines semantics: a trailing newline adds no empty line).
Args:
file_path: Absolute path to the file.
Inputs: file_path (required). Returns: result
count_tokensA
Count tokens in text.
Routes by model name to Anthropic/Claude (default) or OpenAI/tiktoken.
Args:
text: The text to tokenize.
model: Model or encoding name (e.g. claude-opus-4-8, gpt-4, o200k_base).
Empty uses ANTHROPIC_TOKEN_COUNT_MODEL.
Inputs: text (required), model. Returns: result
count_tokens_fileA
Count tokens in a UTF-8 file.
Args:
file_path: Absolute path to the file.
model: Model or encoding name. Empty uses ANTHROPIC_TOKEN_COUNT_MODEL.
Inputs: file_path (required), model. Returns: result
count_tokens_folderA
Count tokens across all text files in a folder, recursively.
Skips dependency/VCS/build directories (.git, node_modules, .venv, ...)
and binary files (by extension, then a content sniff). Non-UTF-8 text
falls back to UTF-16 (BOM) or cp1252. Returns the sum of per-file counts.
Args:
folder_path: Absolute path to the folder.
model: Model or encoding name. Empty uses ANTHROPIC_TOKEN_COUNT_MODEL.
Inputs: folder_path (required), model. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 6 tools
Each tool targets a distinct input (text, file, folder) or metric (tokens, lines, chars, KB), so there is no overlap. The count_tokens variants are clearly separated by scope.
All tools follow the predictable count_<metric>_<target> pattern, with count_tokens as the base form. Naming is uniform and easy to infer.
Six tools is well-scoped for a tokenizer server: three for token counting across input types and three for basic file size/line/character metrics. Every tool earns its place.
The surface covers token counting for text, files, and folders, plus file-level character, line, and byte-size metrics. No obvious gaps exist for the stated purpose.
Maintenance
Related MCP Connectors
Free, no-key MCP server: token estimates, JSON Schema conversion, MCP configs, tool search.
MCP server for StyleKit styles, tokens, recipes, lint and repair over stdio or HTTP.
MCP server for OpenAI API (chat completions, image generation, embeddings) via AceDataCloud
Nifty's MCP server — exposes tasks, projects, messages, and files as tools for AI agents.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceAn MCP server that helps AI agents reduce token usage by converting data to TOON format and stripping comments and unnecessary whitespace from code files.89 PyPIMIT
- AlicenseNot gradedqualityDmaintenanceAn MCP server that provides file and directory tools for LLMs, including project structure analysis, file reading, and project context resources, with support for .gitignore patterns.MIT
- AlicenseNot gradedqualityCmaintenanceToken-efficient MCP server for multi-language project analysis (Java, TypeScript, JavaScript, Markdown, Python) with plugins, semantic search, and static analysis.MIT
- AlicenseNot gradedqualityDmaintenanceMCP server for deterministic, zero-dependency context-window math, enabling token estimation, text truncation, and budget reporting without a tokenizer.MIT