Skip to main content
Glama

tokenizer-mcp

tokenizer-mcp is a small MCP server that lets an LLM harness like Claude Code count the exact tokens in files (or any string) trivially easily, plus report basic file metrics — lines, characters, and size in KB. Every tool returns a single integer. For token counting the server receives the file/text and a model name and routes to the appropriate backend; the file-metric tools need no tokenization backend at all.

Installation

uv sync

Related MCP server: File Tools MCP Server

Running it directly

uv run server.py

API key setup

Counting tokens for Claude requires an Anthropic API key, since the exact count comes from messages.count_tokens. Without a key, the Claude path silently falls back to tiktoken's o200k_base encoding. GPT counts work offline and need no key.

Set the key as an environment variable:

export ANTHROPIC_API_KEY=sk-ant-...

On Windows (PowerShell):

$env:ANTHROPIC_API_KEY = "sk-ant-..."

Or drop a .env file next to server.py and the server will load it on startup:

ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_TOKEN_COUNT_MODEL=claude-opus-4-8

ANTHROPIC_TOKEN_COUNT_MODEL sets the default Claude model when the caller omits one; it defaults to claude-opus-4-8. The endpoint tokenizes under the model you pass, so use a current model — the tokenizer changed at Opus 4.7 (~30% more tokens than older models for the same text).

What it exposes

Six tools, each returning an integer.

Token counting (uses the model routing / backends described below):

  • count_tokens(text, model="") — count tokens in a string.

  • count_tokens_file(file_path, model="") — count tokens in a UTF-8 file at an absolute path.

  • count_tokens_folder(folder_path, model="") — sum of per-file token counts across every text file in a folder, recursively. Dependency/VCS/build directories (.git, node_modules, .venv, __pycache__, dist, ...) are excluded, and binary files are skipped — first by extension, then by a content sniff (NUL bytes / control-character ratio) for unrecognized extensions. Files that aren't UTF-8 are decoded as UTF-16 (when a BOM is present) or cp1252 rather than skipped.

When model is omitted, these use ANTHROPIC_TOKEN_COUNT_MODEL.

File metrics (no tokenization backend; file-only — there are deliberately no raw-string variants):

  • count_lines_file(file_path) — number of lines, using Python str.splitlines() semantics (a trailing newline terminates the last line rather than adding an empty one).

  • count_chars_file(file_path) — number of characters in the decoded UTF-8 text (Unicode code points).

  • count_kb_file(file_path) — file size in kilobytes, ceil(bytes / 1024) with 1 KB = 1024 bytes (the same convention as the Tokenizer app). The size is read from the filesystem, so it works for any file regardless of encoding.

These are useful when a file's token count is only one of several limits — lines, characters, raw size — that decide whether it fits a given budget (e.g. a harness tool-call payload).

How model routing works

The lowercased model is matched against a short set of rules:

What you pass

Backend used

A raw tiktoken encoding (o200k_base, cl100k_base, p50k_base, p50k_edit, r50k_base, gpt2)

tiktoken.get_encoding

Anything starting with an OpenAI prefix (gpt-5, gpt-4, gpt-3.5, gpt-3, chatgpt, o1, o3, o4, text-davinci, text-embedding, code-davinci, text-curie, text-babbage, text-ada)

tiktoken.encoding_for_model

Everything else (Claude models)

Anthropic messages.count_tokens

Wiring it into an MCP client

Add an mcpServers entry pointing at this directory:

{
  "mcpServers": {
    "tokenizer": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/tokenizer-mcp", "run", "server.py"],
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}

The env block is optional; omit it and Claude counts fall back to o200k_base, as above.

Available Tools

6 tools
count_chars_fileA

Count characters (Unicode code points) in a UTF-8 file.

Args:
    file_path: Absolute path to the file.

Inputs: file_path (required). Returns: result

count_kb_fileA

File size in KB as ceil(bytes / 1024), matching Windows Explorer.

Args:
    file_path: Absolute path to the file.

Inputs: file_path (required). Returns: result

count_lines_fileA

Count lines in a file (str.splitlines semantics: a trailing newline adds no empty line).

Args:
    file_path: Absolute path to the file.

Inputs: file_path (required). Returns: result

count_tokensA

Count tokens in text.

Routes by model name to Anthropic/Claude (default) or OpenAI/tiktoken.

Args:
    text: The text to tokenize.
    model: Model or encoding name (e.g. claude-opus-4-8, gpt-4, o200k_base).
        Empty uses ANTHROPIC_TOKEN_COUNT_MODEL.

Inputs: text (required), model. Returns: result

count_tokens_fileA

Count tokens in a UTF-8 file.

Args:
    file_path: Absolute path to the file.
    model: Model or encoding name. Empty uses ANTHROPIC_TOKEN_COUNT_MODEL.

Inputs: file_path (required), model. Returns: result

count_tokens_folderA

Count tokens across all text files in a folder, recursively.

Skips dependency/VCS/build directories (.git, node_modules, .venv, ...)
and binary files (by extension, then a content sniff). Non-UTF-8 text
falls back to UTF-16 (BOM) or cp1252. Returns the sum of per-file counts.

Args:
    folder_path: Absolute path to the folder.
    model: Model or encoding name. Empty uses ANTHROPIC_TOKEN_COUNT_MODEL.

Inputs: folder_path (required), model. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.4/5.0

    Scored across 6 tools

    Disambiguation5/5

    Each tool targets a distinct input (text, file, folder) or metric (tokens, lines, chars, KB), so there is no overlap. The count_tokens variants are clearly separated by scope.

    Naming Consistency5/5

    All tools follow the predictable count_<metric>_<target> pattern, with count_tokens as the base form. Naming is uniform and easy to infer.

    Tool Count5/5

    Six tools is well-scoped for a tokenizer server: three for token counting across input types and three for basic file size/line/character metrics. Every tool earns its place.

    Completeness5/5

    The surface covers token counting for text, files, and folders, plus file-level character, line, and byte-size metrics. No obvious gaps exist for the stated purpose.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that helps AI agents reduce token usage by converting data to TOON format and stripping comments and unnecessary whitespace from code files.
      89 PyPI
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that provides file and directory tools for LLMs, including project structure analysis, file reading, and project context resources, with support for .gitignore patterns.
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Token-efficient MCP server for multi-language project analysis (Java, TypeScript, JavaScript, Markdown, Python) with plugins, semantic search, and static analysis.
      MIT