tokenize
Tokenize UTF-8 text into individual tokens with a supported deterministic Algenta tokenizer model (default text.tokenizer; call list_models for every supported model id). Use this when you need the token strings themselves; call count_tokens when you only need the number. Read-only and deterministic: the same input and model always return the same tokens, and nothing is stored. Returns the resolved model id, its tokenizer_kind, the tokens array, and token_count. An unsupported model id fails with model_not_supported.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"input": "<input>"
},
"name": "tokenize"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| input | Yes | UTF-8 text to tokenize. | |
| model | No | Tokenizer model id from list_models. | text.tokenizer |