tokenizer-mcp
Counts tokens for OpenAI models and raw tiktoken encodings such as o200k_base using tiktoken.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@tokenizer-mcpCount tokens in src/index.js"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
tokenizer-mcp
tokenizer-mcp is a small MCP server that lets an LLM harness like Claude Code count the exact tokens in files (or any string) trivially easily, plus report basic file metrics — lines, characters, and size in KB. Every tool returns a single integer. For token counting the server receives the file/text and a model name and routes to the appropriate backend; the file-metric tools need no tokenization backend at all.
Installation
uv syncRelated MCP server: toon-parse-mcp
Running it directly
uv run server.pyAPI key setup
Counting tokens for Claude requires an Anthropic API key, since the exact count comes from messages.count_tokens. Without a key, the Claude path silently falls back to tiktoken's o200k_base encoding. GPT counts work offline and need no key.
Set the key as an environment variable:
export ANTHROPIC_API_KEY=sk-ant-...On Windows (PowerShell):
$env:ANTHROPIC_API_KEY = "sk-ant-..."Or drop a .env file next to server.py and the server will load it on startup:
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_TOKEN_COUNT_MODEL=claude-opus-4-8ANTHROPIC_TOKEN_COUNT_MODEL sets the default Claude model when the caller omits one; it defaults to claude-opus-4-8. The endpoint tokenizes under the model you pass, so use a current model — the tokenizer changed at Opus 4.7 (~30% more tokens than older models for the same text).
What it exposes
Six tools, each returning an integer.
Token counting (uses the model routing / backends described below):
count_tokens(text, model="")— count tokens in a string.count_tokens_file(file_path, model="")— count tokens in a UTF-8 file at an absolute path.count_tokens_folder(folder_path, model="")— sum of per-file token counts across every text file in a folder, recursively. Dependency/VCS/build directories (.git,node_modules,.venv,__pycache__,dist, ...) are excluded, and binary files are skipped — first by extension, then by a content sniff (NUL bytes / control-character ratio) for unrecognized extensions. Files that aren't UTF-8 are decoded as UTF-16 (when a BOM is present) or cp1252 rather than skipped.
When model is omitted, these use ANTHROPIC_TOKEN_COUNT_MODEL.
File metrics (no tokenization backend; file-only — there are deliberately no raw-string variants):
count_lines_file(file_path)— number of lines, using Pythonstr.splitlines()semantics (a trailing newline terminates the last line rather than adding an empty one).count_chars_file(file_path)— number of characters in the decoded UTF-8 text (Unicode code points).count_kb_file(file_path)— file size in kilobytes,ceil(bytes / 1024)with 1 KB = 1024 bytes (the same convention as the Tokenizer app). The size is read from the filesystem, so it works for any file regardless of encoding.
These are useful when a file's token count is only one of several limits — lines, characters, raw size — that decide whether it fits a given budget (e.g. a harness tool-call payload).
How model routing works
The lowercased model is matched against a short set of rules:
What you pass | Backend used |
A raw tiktoken encoding ( |
|
Anything starting with an OpenAI prefix ( |
|
Everything else (Claude models) | Anthropic |
Wiring it into an MCP client
Add an mcpServers entry pointing at this directory:
{
"mcpServers": {
"tokenizer": {
"command": "uv",
"args": ["--directory", "/absolute/path/to/tokenizer-mcp", "run", "server.py"],
"env": {
"ANTHROPIC_API_KEY": "sk-ant-..."
}
}
}
}The env block is optional; omit it and Claude counts fall back to o200k_base, as above.
Available Tools
6 toolscount_chars_fileA
Count characters (Unicode code points) in a UTF-8 file.
Args:
file_path: Absolute path to the file.
| Name | Required | Description | Default |
|---|---|---|---|
| file_path | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavioral disclosure. It provides useful behavioral detail by specifying UTF-8 encoding and the Unicode code point counting semantics, but it does not mention error behavior for invalid UTF-8, missing files, or whether newlines are included. This is adequate for a simple read operation but not exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two lines long, front-loads the core behavior, and contains no filler. The 'Args' clarification about absolute paths earns its place by guiding correct invocation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter counting tool with an output schema available, the description covers the necessary input semantics and the key counting behavior. It does not describe edge cases or error handling, but those are relatively minor for this tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It does by documenting file_path as an absolute path, which adds meaningful constraint beyond the schema's bare 'File Path' string type. This is sufficient for the tool's single parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Count') and resource ('UTF-8 file'), and clarifies that it counts Unicode code points rather than bytes or lines. This clearly differentiates it from sibling tools like count_tokens_file, count_lines_file, and count_kb_file.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The purpose implies when to use this tool — when a Unicode code point count of a file is needed — but it does not explicitly mention alternatives or state conditions for choosing another sibling tool. Usage guidance is implied rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
count_kb_fileA
File size in KB as ceil(bytes / 1024), matching Windows Explorer.
Args:
file_path: Absolute path to the file.
| Name | Required | Description | Default |
|---|---|---|---|
| file_path | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It adds meaningful behavioral detail by specifying the rounding method and the Windows Explorer compatibility, which goes beyond simply saying 'returns file size'. It does not disclose error behavior or what happens for non-file paths, but the core behavior is transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: one sentence defining behavior and formula, followed by the single argument's purpose. Every word adds value, and the key behavioral detail is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple single-parameter tool with an output schema, the description provides the essential formula, the matching behavior, and the path requirement. It is nearly complete, but omits edge-case behavior such as nonexistent files, directories, or permission failures.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. It does so by specifying that file_path is an 'Absolute path to the file', adding a meaningful constraint beyond the schema's bare string type. Since there is only one parameter and its role is clear, this is sufficient.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool computes file size in KB using a specific formula, ceil(bytes / 1024), and explicitly matches Windows Explorer's display convention. This distinguishes it from sibling tools that count tokens, lines, or characters.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The usage context is implied by the name and description: use this when you need a file's size in KB. However, there is no explicit guidance about when to prefer this tool over siblings, no exclusions, and no mention of invalid inputs such as nonexistent files or directories.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
count_lines_fileA
Count lines in a file (str.splitlines semantics: a trailing newline adds no empty line).
Args:
file_path: Absolute path to the file.
| Name | Required | Description | Default |
|---|---|---|---|
| file_path | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description carries the behavioral disclosure burden. It usefully discloses splitlines semantics, including the trailing-newline edge case, and requires an absolute path. It does not mention error or encoding behavior, but for a simple read-only counting tool the stated behavior is meaningful.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise, front-loads the critical semantic rule, and includes only a single necessary parameter explanation. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity, an existing output schema, and the disclosure of the important splitlines behavior, an agent has enough information to call it correctly. Minor gaps remain around when to choose it over sibling tools and how errors are reported, but these are not blocking for this tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides no description coverage, but the description's args section defines file_path as "Absolute path to the file," adding a real constraint beyond the schema's bare string type. This adequately compensates for the schema gap for a single-parameter tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
"Count lines in a file" states a specific verb, resource, and metric. The added splitlines semantics further disambiguates exactly what is counted, and the tool name and siblings make the distinction from count_tokens_file and count_chars_file clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no explicit guidance on when to select this tool over sibling tools such as count_tokens_file or count_chars_file. Usage is only implied by the name and purpose, with no mention of alternatives or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
count_tokensA
Count tokens in text.
Routes by model name to Anthropic/Claude (default) or OpenAI/tiktoken.
Args:
text: The text to tokenize.
model: Model or encoding name (e.g. claude-opus-4-8, gpt-4, o200k_base).
Empty uses ANTHROPIC_TOKEN_COUNT_MODEL.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| model | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description carries the full burden. It discloses two meaningful behaviors not visible in the schema: tokenizer selection is routed by model name with an Anthropic default, and an empty model falls back to ANTHROPIC_TOKEN_COUNT_MODEL. This is useful context beyond a bare 'count tokens' statement, though it does not mention failure modes for unsupported model names.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences plus a compact Args list; every line adds information (purpose, routing, parameter behavior). The core purpose is front-loaded and no filler is present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple token-counting tool with an output schema present, the description covers purpose, parameter semantics, and routing. It does not explicitly address edge cases (e.g., unknown model, empty text), but the complexity is low enough that these omissions are not harmful.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must clarify the parameters, and it does. 'text' is explained only minimally, but 'model' is well specified with concrete examples (claude-opus-4-8, gpt-4, o200k_base) and the empty-string fallback behavior, adding real semantic value over the schema's bare type/default.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Count tokens in text'), which directly differentiates it from the file/folder siblings by scoping to raw text. The model-routing clause adds useful specialization without making the purpose ambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies its use case through 'in text' and the sibling names suggest file/folder alternatives, but it never explicitly states when to choose this tool over count_tokens_file/count_tokens_folder or mentions any exclusion criteria. The routing and default model info is model-selection guidance, not tool-selection guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
count_tokens_fileA
Count tokens in a UTF-8 file.
Args:
file_path: Absolute path to the file.
model: Model or encoding name. Empty uses ANTHROPIC_TOKEN_COUNT_MODEL.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | ||
| file_path | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations available, the description carries the burden of behavioral disclosure. It does add useful details: the file must be UTF-8, and an empty model defaults to ANTHROPIC_TOKEN_COUNT_MODEL. It does not describe what happens with invalid paths, encoding errors, or file-size limits, but for a straightforward read-only counting operation this is acceptable. The description adds some behavioral context without being exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is tight and front-loaded: a one-line purpose followed by an Args section. There is no filler, and each sentence adds value. It covers both parameters and the default behavior without unnecessary elaboration, making it easy for an agent to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool, the description covers the essential inputs and default behavior. The output schema exists, so return-value documentation is not required from the description. However, it lacks guidance on error conditions, what constitutes a valid UTF-8 file, and how to choose between this and sibling counting tools, leaving moderate gaps for an agent that lacks prior context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does meaningfully. It clarifies file_path must be an absolute path, and it explains model as 'Model or encoding name' with an explicit default behavior. Both parameters are given semantic meaning beyond their names, types, and defaults, making the tool usable without inspecting external documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action ('Count tokens') on a specific resource ('a UTF-8 file'), which is immediately distinguishable from sibling tools like count_tokens_folder, count_lines_file, count_chars_file, and count_kb_file. The file path parameter and encoding qualifier make the scope explicit. Unlike a vague 'Process' definition, this gives an agent a clear understanding of what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool should be used when a token count for a file is needed, and the sibling list provides alternative counting tools. However, there is no explicit guidance on when to prefer this over count_tokens or count_tokens_folder, nor any exclusions for non-UTF-8 files or other edge cases. Usage context is present but left largely to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
count_tokens_folderA
Count tokens across all text files in a folder, recursively.
Skips dependency/VCS/build directories (.git, node_modules, .venv, ...)
and binary files (by extension, then a content sniff). Non-UTF-8 text
falls back to UTF-16 (BOM) or cp1252. Returns the sum of per-file counts.
Args:
folder_path: Absolute path to the folder.
model: Model or encoding name. Empty uses ANTHROPIC_TOKEN_COUNT_MODEL.
| Name | Required | Description | Default |
|---|---|---|---|
| model | No | ||
| folder_path | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations to lean on, the description discloses key behaviors: skipped dependency/build directories, binary file detection strategy, encoding fallbacks, and the aggregated return value. This is unusually thorough and goes well beyond a simple statement of the operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-organized: a one-sentence summary, followed by behavioral details, then an Args section. Every sentence carries relevant information with no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter tool, the description fully covers inputs, exclusions, encoding handling, and return semantics. Even with an output schema present, the explicit statement that it returns the sum of per-file counts removes any ambiguity about the result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must do all the work. It explains folder_path must be an absolute path and clarifies the model parameter, including its default env-var behavior. This adds meaningful semantic value beyond the raw schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening sentence names the specific operation: counting tokens across all text files in a folder, recursively. This clearly distinguishes it from sibling tools like count_tokens_file or count_lines_file.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The recursive folder-wide scope strongly implies when to use this tool versus single-file alternatives, but it never explicitly names the alternatives or states when not to use it. The usage context is clear from the description, though exclusions are absent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
TDQS
Each tool targets a distinct input (text, file, folder) or metric (tokens, lines, chars, KB), so there is no overlap. The count_tokens variants are clearly separated by scope.
All tools follow the predictable count_<metric>_<target> pattern, with count_tokens as the base form. Naming is uniform and easy to infer.
Six tools is well-scoped for a tokenizer server: three for token counting across input types and three for basic file size/line/character metrics. Every tool earns its place.
The surface covers token counting for text, files, and folders, plus file-level character, line, and byte-size metrics. No obvious gaps exist for the stated purpose.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Cloudflare Workers MCP server: ai-token-counter
Nifty's MCP server — exposes tasks, projects, messages, and files as tools for AI agents.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
Related MCP Servers
- AlicenseAqualityDmaintenanceAn MCP (Model Context Protocol) server that provides real-time LLM token pricing data for 60+ AI models across 15 providers.6152MIT
- AlicenseNot gradedqualityDmaintenanceAn MCP server that helps AI agents reduce token usage by converting data to TOON format and stripping comments and unnecessary whitespace from code files.MIT
- AlicenseNot gradedqualityDmaintenanceAn MCP server that provides file and directory tools for LLMs, including project structure analysis, file reading, and project context resources, with support for .gitignore patterns.MIT
- AlicenseNot gradedqualityCmaintenanceToken-efficient MCP server for multi-language project analysis (Java, TypeScript, JavaScript, Markdown, Python) with plugins, semantic search, and static analysis.MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/the-phase-space/tokenizer-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server