# tokenizer-mcp

`tokenizer-mcp` is a small MCP server that lets an LLM harness like Claude Code count the exact tokens in files (or any string) trivially easily, plus report basic file metrics — lines, characters, and size in KB. Every tool returns a single integer. For token counting the server receives the file/text and a model name and routes to the appropriate backend; the file-metric tools need no tokenization backend at all.

## Installation

```bash
uv sync
```

## Running it directly

```bash
uv run server.py
```

## API key setup

Counting tokens for Claude requires an Anthropic API key, since the exact count comes from `messages.count_tokens`. Without a key, the Claude path silently falls back to tiktoken's `o200k_base` encoding. GPT counts work offline and need no key.

Set the key as an environment variable:

```bash
export ANTHROPIC_API_KEY=sk-ant-...
```

On Windows (PowerShell):

```powershell
$env:ANTHROPIC_API_KEY = "sk-ant-..."
```

Or drop a `.env` file next to `server.py` and the server will load it on startup:

```
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_TOKEN_COUNT_MODEL=claude-opus-4-8
```

`ANTHROPIC_TOKEN_COUNT_MODEL` sets the default Claude model when the caller omits one; it defaults to `claude-opus-4-8`. The endpoint tokenizes under the model you pass, so use a current model — the tokenizer changed at Opus 4.7 (~30% more tokens than older models for the same text).

## What it exposes

Six tools, each returning an integer.

**Token counting** (uses the model routing / backends described below):

- `count_tokens(text, model="")` — count tokens in a string.
- `count_tokens_file(file_path, model="")` — count tokens in a UTF-8 file at an absolute path.
- `count_tokens_folder(folder_path, model="")` — sum of per-file token counts across every text file in a folder, recursively. Dependency/VCS/build directories (`.git`, `node_modules`, `.venv`, `__pycache__`, `dist`, ...) are excluded, and binary files are skipped — first by extension, then by a content sniff (NUL bytes / control-character ratio) for unrecognized extensions. Files that aren't UTF-8 are decoded as UTF-16 (when a BOM is present) or cp1252 rather than skipped.

When `model` is omitted, these use `ANTHROPIC_TOKEN_COUNT_MODEL`.

**File metrics** (no tokenization backend; file-only — there are deliberately no raw-string variants):

- `count_lines_file(file_path)` — number of lines, using Python `str.splitlines()` semantics (a trailing newline terminates the last line rather than adding an empty one).
- `count_chars_file(file_path)` — number of characters in the decoded UTF-8 text (Unicode code points).
- `count_kb_file(file_path)` — file size in kilobytes, `ceil(bytes / 1024)` with 1 KB = 1024 bytes (the same convention as the Tokenizer app). The size is read from the filesystem, so it works for any file regardless of encoding.

These are useful when a file's token count is only one of several limits — lines, characters, raw size — that decide whether it fits a given budget (e.g. a harness tool-call payload).

## How model routing works

The lowercased `model` is matched against a short set of rules:

| What you pass                                           | Backend used                                       |
| ------------------------------------------------------- | -------------------------------------------------- |
| A raw tiktoken encoding (`o200k_base`, `cl100k_base`, `p50k_base`, `p50k_edit`, `r50k_base`, `gpt2`) | `tiktoken.get_encoding`                            |
| Anything starting with an OpenAI prefix (`gpt-5`, `gpt-4`, `gpt-3.5`, `gpt-3`, `chatgpt`, `o1`, `o3`, `o4`, `text-davinci`, `text-embedding`, `code-davinci`, `text-curie`, `text-babbage`, `text-ada`) | `tiktoken.encoding_for_model`                      |
| Everything else (Claude models)                         | Anthropic `messages.count_tokens`                  |

## Wiring it into an MCP client

Add an `mcpServers` entry pointing at this directory:

```json
{
  "mcpServers": {
    "tokenizer": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/tokenizer-mcp", "run", "server.py"],
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}
```

The `env` block is optional; omit it and Claude counts fall back to `o200k_base`, as above.