llama_tokenize
Encode text into token IDs for LLM processing, with options to add special tokens and include token strings.
Instructions
Convert text to token IDs
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| content | Yes | Text to tokenize | |
| add_special | No | Add BOS/EOS tokens | |
| with_pieces | No | Include token strings |