Chunk text for RAG
ai_chunk_textGenerate token-sized chunks from text, Markdown, or files for retrieval-augmented generation (RAG) and embeddings, preserving headings, paragraphs, sentence boundaries, overlap, and offsets.
Instructions
Split text, markdown or a text file into token-sized chunks for retrieval (RAG) and embeddings. Splits at markdown headings (ATX "#" headings, never inside code fences) and paragraphs first, then sentences (lines inside code blocks), and only cuts mid-sentence when a single sentence is longer than maxTokens. Consecutive chunks of the same section share about overlapTokens of text. Each chunk returns index, text, tokens, startChar/endChar (JavaScript string offsets, end exclusive, so text === source.slice(startChar, endChar) once CRLF line endings are normalised to \n) and the heading path it sits under. Tokens are counted with tiktoken (o200k_base by default; for Claude treat counts as estimates). Give "text" or "path" (file path or base64); give "output" to also write the chunks as JSONL (one JSON object per line).
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "ai_chunk_text"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Or: a UTF-8 text/markdown file, as a local path or base64 (raw or a data: URI). | |
| text | No | The text (inline). | |
| output | No | Also write the chunks as JSONL to this file (or folder ending in "/"). | |
| encoding | No | Tokenizer: "o200k_base" (GPT-4o / o-series, default) or "cl100k_base" (GPT-4 / GPT-3.5 / text-embedding-3). | o200k_base |
| markdown | No | Treat "#" lines as section headings (default true). Set false for plain text. | |
| maxChunks | No | Most chunks to return inline (the JSONL file always has all). Default 2000. | |
| maxTokens | No | Largest chunk, in tokens. Default 500. | |
| includeText | No | Return each chunk's text inline. Default: true, or false when "output" is given. | |
| overlapTokens | No | Tokens repeated from the end of the previous chunk (same section only). Default 50. Must be below maxTokens. |