Count tokens
ai_token_countCount LLM tokens in text or files using OpenAI tiktoken tokenizers, and check whether the text fits common context windows (8k–1M).
Instructions
Count LLM tokens in a text or text file with the OpenAI tiktoken tokenizers ("o200k_base", default, used by GPT-4o and the o-series; or "cl100k_base", used by GPT-4/3.5 and text-embedding-3). Also returns characters and words, and whether the text fits in common context windows (8k, 32k, 128k, 200k, 1M: e.g. "fits in 200k: yes"). Claude's tokenizer is not public, so for Claude the count is an estimate (typically within ~10-20%; Claude usually needs a little more), returned as a rough range. Runs locally, no API calls. Give "text" or "path" (file path or base64). Large texts (many MB) take a few seconds.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Or: a UTF-8 text/markdown file, as a local path or base64 (raw or a data: URI). | |
| text | No | The text (inline). | |
| encoding | No | Tokenizer: "o200k_base" (GPT-4o / o-series, default) or "cl100k_base" (GPT-4 / GPT-3.5 / text-embedding-3). | o200k_base |