Skip to main content
Glama
mcp-ml-tools
by adeoluo

Count tokens and estimate cost

tokenize
Read-onlyIdempotent

Counts tokens in text using bundled byte-level BPE, returns token IDs, and estimates token counts and USD cost for configured model families.

Instructions

Tokenizes text with the bundled byte-level BPE (exact counts and token ids for that vocabulary), and estimates token counts and United States dollar cost for common model families from the configured price table. Per-model counts are character-based approximations, not the providers' tokenizers.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "text": "<text>"
    },
    "name": "tokenize"
  }
}

Input Schema

NameRequiredDescriptionDefault
textYesText to tokenize
modelsNoModel names from the price table to estimate (default: all)
maxTokenIdsNoMaximum number of token ids to return (default 2000)
includeTokenIdsNoReturn token ids and strings (default true)
expectedOutputTokensNoExpected completion length in tokens, used for output cost (default 0)

Output Schema

NameRequiredDescriptionDefault
bytesYes
notesYes
estimatesYes
builtInBpeYes
charactersYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.2.2

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnlyHint, idempotentHint, closed-world), so the description's added value is the accuracy caveat: per-model counts are character-based approximations, not the providers' tokenizers. That is a genuinely important behavioral disclosure an agent would otherwise get wrong when reporting costs as exact. It stops short of describing limits on returned ids or truncation behavior, which live only in the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the exact-tokenization capability, then the estimation capability and its caveat. The parenthetical 'exact counts and token ids for that vocabulary' is dense but earns its place by pre-empting a wrong assumption. Minor cost: the approximation caveat arrives after the estimate claim rather than with it.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists and all five parameters are documented in the schema, so the description does not need to explain return values. It covers both modes and the accuracy caveat, which is what an agent needs to call this correctly. The only missing piece is behavior under the size/id caps (2,000,000 chars, maxTokenIds default 2000), which the agent must infer from the schema alone.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 100% schema description coverage the baseline is 3, and the description earns above that by clarifying intent behind parameters the schema only labels: `models` is scoped to names from the configured price table, token ids are tied to the bundled vocabulary, and per-model estimation is character-based. It does not add syntax examples or explain how `expectedOutputTokens` interacts with `models`, leaving a small gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names two specific verbs and their resources: exact byte-level BPE tokenization of text (with counts and token ids) and USD cost/token estimation across model families from a configured price table. It also draws the crucial distinction between the exact bundled vocabulary and the approximate per-model numbers, so an agent knows precisely what it gets. None of the listed siblings (bm25_search, eval_compare, json_schema_validate, prompt_injection_scan) overlap in purpose, so no sibling disambiguation is needed.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the selection rule clearly: use the bundled BPE when you need exact counts/ids for that vocabulary, use the price-table path when you need cost or per-model estimates. It does not state explicit exclusions or name an alternative tool, but there is no competing sibling to route to, so the implied guidance is sufficient in practice.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.