Count tokens and estimate cost
tokenizeCounts tokens in text using bundled byte-level BPE, returns token IDs, and estimates token counts and USD cost for configured model families.
Instructions
Tokenizes text with the bundled byte-level BPE (exact counts and token ids for that vocabulary), and estimates token counts and United States dollar cost for common model families from the configured price table. Per-model counts are character-based approximations, not the providers' tokenizers.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"text": "<text>"
},
"name": "tokenize"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | Text to tokenize | |
| models | No | Model names from the price table to estimate (default: all) | |
| maxTokenIds | No | Maximum number of token ids to return (default 2000) | |
| includeTokenIds | No | Return token ids and strings (default true) | |
| expectedOutputTokens | No | Expected completion length in tokens, used for output cost (default 0) |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| bytes | Yes | ||
| notes | Yes | ||
| estimates | Yes | ||
| builtInBpe | Yes | ||
| characters | Yes |