Provides a reusable GitHub Action that runs md-chunk-lite on Markdown files during CI, generates JSONL chunks, reports oversized split sections, can fail the build when sections are split, and can upload the output as an artifact.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@io.github.tidybytetools/md-chunk-lite Split guide.md into heading-based chunks and save them to guide.chunks.jsonl"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Free tool that splits Markdown into chunks by heading. Every chunk knows its heading path and the line numbers it came from. Need PDF, Word or web page files turned into chunks (with page numbers for PDFs), or whole folders? The paid doc2md does that: doc2md on Gumroad
Good for AI search (RAG), embeddings, and docs bots. Plain code, not AI, so the same file always gives the same chunks. Runs on your own computer. MIT license. Python 3.10 or newer, on Windows, Mac and Linux.
pip install md-chunk-liteOr use pipx, which keeps it in its own space:
pipx install md-chunk-liteWant the newest code from GitHub instead?
pip install git+https://github.com/tidybytetools/md-chunk-liteRelated MCP server: mcp-server-markdown
md-chunk-lite guide.md # writes guide.chunks.jsonl
md-chunk-lite docs/*.md -o all.jsonl # many files into one file
md-chunk-lite guide.md -o - # print the chunks on screen
md-chunk-lite guide.md --max-chars 800 # smaller chunks
md-chunk-lite guide.md --outline # just show the headings
cat guide.md | md-chunk-lite - # read from stdinExample run:
Saved: guide.chunks.jsonl (7 chunks, biggest 151 chars)Each line of the .jsonl file is one chunk:
{
"id": "guide.md#3",
"source": "guide.md",
"index": 3,
"heading_path": ["Garden Guide", "Planting", "Tomatoes"],
"heading_level": 3,
"anchor": "tomatoes",
"start_line": 16,
"end_line": 24,
"part": 1,
"parts": 1,
"chars": 151,
"text": "### Tomatoes\n\nPlant tomatoes after the last frost. Give them sun.\n\n```python\n..."
}heading_path: all the headings above this chunk, from the top.
anchor: the #link GitHub makes for that heading, so you can link straight to it.
start_line and end_line: where the chunk is in the file (the first line is 1).
part and parts: a big section is split into parts. part 2, parts 3 means the 2nd of 3.
chars: how long text is.
--outline prints the headings:
Garden Guide [line 8, #garden-guide, 14 chars]
Tools [line 10, #tools, 47 chars]
Planting [line 14, #planting, 11 chars]
Tomatoes [line 16, #tomatoes, 151 chars]When you run it in a terminal it ends with one short line about the paid doc2md. --quiet turns it off, and it never shows when output is piped or redirected.
Each heading starts a new section. A chunk never mixes two sections, so every chunk has one clear heading path.
A small section becomes one chunk. Its text is copied straight from your file, so the line numbers match.
A big section is split at blank lines. Lists, tables and code blocks stay whole when they fit.
A table that is too big is split by rows, and each part gets the header row again.
A code block that is too big is split by lines, and each part gets its own ``` fence again, with the same language.
A very long line is cut at the end of a sentence, then at a space if it has to be.
A heading with no text of its own (just more headings under it) does not get its own chunk, but it still shows in the heading_path of the chunks under it.
It knows # headings and underlined headings (=== and ---). It skips headings inside code blocks and YAML front matter (the --- block at the very top).
Options:
--max-chars N: biggest chunk, in characters (default 1500, smallest 200). About 4 characters make 1 token in English.
--overlap-chars N: when a section is split, start each part with up to N characters from the end of the part before it (default 0). It is never added inside a code block or table, and never between two different sections.
--prefix-path: start each chunk's text with its heading path, like Garden Guide > Planting > Tomatoes. This often helps search. It counts toward --max-chars.
--json: print one JSON object for scripts and AI agents instead of saving a file.
Sizes are characters, not tokens. It does not load a tokenizer.
Markdown and plain text only. PDF, Word and HTML are not read. The paid doc2md turns those into chunks (with page numbers for PDFs).
No folders. Give it files (your shell can expand docs/*.md). The paid doc2md walks whole folders.
Not a full Markdown parser. A heading inside an HTML block, or an underlined heading that spans more than one line, may be handled wrong.
Overlap uses whole words from the part before, so it can be a little shorter than you asked for.
md-chunk-lite (free) | doc2md (paid) | |
Reads | Markdown and text files | PDF, Word (.docx) and HTML |
Chunks with the heading path | yes | yes |
Line numbers and anchors | yes | no (page numbers for PDFs instead) |
Rough token count on each chunk | no | yes |
Whole folders, sub-folders too | no | yes |
MCP server | yes | no |
Pick what fits you. If you already use LangChain, its MarkdownHeaderTextSplitter also splits by heading. Chonkie and semchunk split any text by token count. md-chunk-lite is small, MIT, has no AI or tokenizer parts, gives you line numbers and anchors, and has an MCP server built in.
There is a small free MCP server built in. It needs uv.
Run it by hand to check it starts (it waits quietly for an app to talk to it):
uvx md-chunk-lite mcpClaude Desktop (claude_desktop_config.json) or Cursor (~/.cursor/mcp.json):
{
"mcpServers": {
"md-chunk-lite": {
"command": "uvx",
"args": ["md-chunk-lite", "mcp"]
}
}
}uvx --from md-chunk-lite md-chunk-lite-mcp starts the same server.
Tools (each one takes a file path or Markdown text):
Tool | What it does | Changes files? |
| Splits into chunks. Returns 100 at a time; use | No |
| Lists the headings with line numbers and anchors. | No |
| Gets one section by heading or anchor. | No |
| Saves the chunks to a | Writes one |
None of them use the internet.
Chunk your docs on every push, so your AI search or docs bot always gets fresh chunks. It also warns you when a section is too big for one chunk, right on the line in the pull request.
name: Docs chunks
on: [push, pull_request]
permissions:
contents: read
jobs:
chunks:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: tidybytetools/md-chunk-lite@v1
with:
files: docs/**/*.md
max-chars: "1500"
artifact-name: docs-chunks # download the JSONL from the run pageWant a big section to fail the check instead of just warn? Add fail-on-split: "true".
It runs the code in the same tag you pick, with only Python's standard library. No API key, no account, and nothing is sent outside GitHub. It works on Linux, Windows and Mac runners.
Input | Default | What it does |
|
| Which files to read. Glob patterns, one per line or split by commas. |
|
| Patterns to skip. |
|
| Biggest chunk, in characters (200 or more). |
|
| Characters from the part before to repeat at the start of the next part. |
|
| Start each chunk's text with its heading path. |
|
| Where to save the chunks. |
|
| Fail when a section is too big for one chunk. |
|
| Fail when no files match. |
| (empty) | If set, upload the JSONL as an artifact with this name. |
| (empty) | Python to set up. Empty uses the runner's Python (3.10 or newer). |
Output | What it is |
| How many files were read. |
| How many chunks were made. |
| Size of the biggest chunk, in characters. |
| How many sections were too big and got split. |
| Path of the JSONL file. |
The run page also gets a short report: chunks per file, and a list of every section that got split, with its line number.
Use when: you have a Markdown or text file (docs, notes, a README) and need chunks with heading paths and line numbers for search, embeddings or citing sources. Use outline_markdown and get_section to read just one part of a long file.
Don't use when: the file is PDF, Word or HTML (convert it first), or you need token-exact sizes.
Input: a file path or Markdown text, plus max_chars (default 1500), overlap_chars (default 0) and prefix_path (default false).
Output: chunks with id, source, index, heading_path, heading_level, anchor, start_line, end_line, part, parts, chars, text.
CLI exit code is 0 on success and 1 on error (message on stderr, or {"ok": false, "error": ...} with --json). See also llms.txt.
from md_chunk_lite import chunk_file, chunk_markdown, outline, get_section
chunks = chunk_file("guide.md", max_chars=1000)
for c in chunks:
print(c["heading_path"], c["start_line"], c["end_line"])
install = get_section(open("README.md").read(), heading="Install")The chunker file uses only Python's standard library. Installing with pip also adds the mcp package for the MCP server.
pip install "mcp>=1.10,<2"
python3 -m unittest discover -s testsThe MCP tests are skipped if mcp is not installed.
MIT. See LICENSE.
Split a Markdown file or Markdown text into chunks by heading, for search, embeddings or RAG.
Use this when you need a long Markdown document (docs, notes, a README) broken into pieces that each know where they came from. A chunk never mixes two sections. Each chunk has heading_path (like ["Guide", "Install"]), anchor (the GitHub-style #link), start_line and end_line (1-based, in the file), part and parts (when a big section had to be split), chars and text. Code blocks and tables are kept whole when they fit; when they don't, each piece gets the fence or table header again. Plain code, not AI: same input, same chunks.
Do not use it for PDF or Word files (convert them to Markdown first), to just see the headings (use outline_markdown), or to read one section (use get_section). Sizes are characters, not tokens.
Returns {"source", "total", "start", "chunks": [...], "next_start"}. next_start is null when there are no more chunks.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Full path to a local Markdown or text file (UTF-8, up to 10 MB). Leave empty if you pass text. | |
| text | No | Markdown text to use instead of a file. Leave empty if you pass path. | |
| limit | No | Most chunks to return in one call. Default 100. | |
| start | No | First chunk number to return (1 = the first). Use next_start from the last call to get more. | |
| max_chars | No | Biggest chunk size in characters (not tokens). Default 1500. | |
| prefix_path | No | If true, each chunk text starts with its heading path, like "Guide > Install". | |
| overlap_chars | No | Characters repeated from the end of one part at the start of the next, only when a section is split. Must be at most half of max_chars. Default 0. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, but the description adds substantial behavior beyond them: chunks never mix sections, code blocks and tables stay whole, split pieces get the fence or table header repeated, and output is deterministic ("same input, same chunks"). It also notes units are characters, not tokens, and describes pagination via next_start.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
It is front-loaded with the core purpose and the disambiguation rules, and most sentences carry information an agent needs. It is on the long side for a tool whose schema is fully documented, with some density that could be trimmed, but nothing is filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having seven parameters and no output schema, the description enumerates the chunk fields (heading_path, anchor, start_line, end_line, part/parts, chars, text) and the envelope keys including next_start. An agent has everything needed to call it and to interpret the response.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all seven parameters are already well documented upstream; this sets the baseline at 3. The description reinforces that sizes are characters rather than tokens, but adds no new syntax, format, or interaction details for path/text, limit, start, prefix_path, or overlap_chars.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource ("Split a Markdown file or Markdown text into chunks by heading") and immediately names the intended application (search, embeddings, RAG). It is clearly distinguishable from siblings: outline_markdown is for headings only and get_section for reading a single section.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit when-to-use sentence (long docs, notes, READMEs needing provenance-aware pieces) and explicit when-not conditions: no PDF/Word, use outline_markdown to list headings, use get_section to read one section. Alternatives are named with the condition that selects them.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the text of one section of a Markdown file or text, found by heading title or anchor.
Use this when you only need one part of a long document, like the "Install" section of a README, so you don't have to read the whole file. If two headings have the same title, the first one is returned and "matches" tells you how many there are; use the anchor (from outline_markdown) to pick another.
Do not use it to split the whole document (use chunk_markdown).
Returns {"title", "heading_path", "level", "anchor", "start_line", "end_line", "chars", "matches", "text"}. If nothing matches, the error lists the headings.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Full path to a local Markdown or text file (UTF-8, up to 10 MB). Leave empty if you pass text. | |
| text | No | Markdown text to use instead of a file. Leave empty if you pass path. | |
| anchor | No | Anchor to find, like "install" or "install-1" for the second "Install". Wins over heading. | |
| heading | No | Heading title to find, like "Install". Not case sensitive. Leave empty if you pass anchor. | |
| include_subsections | No | If true (default), include the smaller headings under it. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover the safety profile (readOnly, idempotent, non-destructive, closed-world), but the description adds genuine behavior the agent could not infer: duplicate headings resolve to the first match with a "matches" count, and a failed lookup returns an error that lists available headings. It also enumerates the return keys, which matters because there is no output schema. It stops short of describing subsection expansion behavior in prose, leaving that to the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight paragraphs ordered by importance: what it does, when to use and when not to, then return/error shape. No filler sentences and no restatement of the title.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description compensates by listing the exact return fields and describing the no-match error, plus the duplicate-heading edge case. An agent has everything needed to call it correctly and interpret the result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline would be 3, but the description adds meaning beyond the schema by explaining the interplay between heading and anchor for duplicate titles and what "matches" means in the result. The anchor/heading precedence is already documented in the schema, so that portion is repetition rather than added value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence gives a specific verb (return the text), a scoped resource (one section of a Markdown file or text), and the selection mechanism (heading title or anchor). This clearly separates it from chunk_markdown, which is named explicitly as the whole-document alternative.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states when to use it with a concrete example (the "Install" section of a README) and states when not to use it, naming the sibling to use instead (chunk_markdown). It also gives a fallback path for the ambiguous case: use the anchor from outline_markdown when duplicate headings exist.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List every heading in a Markdown file or text, with its level, path, anchor, line number and size.
Use this when you want a quick map of a long document before you read or chunk it, or to find the exact heading or anchor to pass to get_section. Headings inside code blocks and YAML front matter are ignored. Both "# Title" and underlined (=== / ---) headings are found.
Do not use it to get the text itself (use get_section or chunk_markdown).
Returns {"source", "count", "headings": [{"level", "title", "path", "anchor", "line", "end_line", "chars"}]}. chars is the size of that heading's own text, not counting the sections under it.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Full path to a local Markdown or text file (UTF-8, up to 10 MB). Leave empty if you pass text. | |
| text | No | Markdown text to use instead of a file. Leave empty if you pass path. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, so the safety profile is covered; the description earns credit by adding real edge-case behavior that annotations cannot express — headings inside code blocks and YAML front matter are ignored, and both ATX and underlined headings are supported. It also documents the return shape because no output schema exists, including the non-obvious semantics of "chars" (own text only, excluding subsections). It does not say what happens when neither path nor text is supplied, which is the one behavioral gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose and use case come first, exclusions second, return shape last, each in its own paragraph. The inline return object is verbose in form but justified because there is no output schema, and every sentence conveys distinct information with no repetition of the name or title.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only two-parameter tool with full schema coverage and no output schema, the description covers purpose, routing, parsing edge cases, and the full return contract. The only missing piece is default behavior when both optional parameters are empty and how errors (missing file, oversized file) surface, which is minor but not nothing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the schema already explains the path/text mutual exclusion and the 10 MB UTF-8 limit. The description adds no further meaning to either parameter, so the baseline 3 applies — the schema is doing all the parameter work.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ("List every heading in a Markdown file or text") and immediately enumerates the fields returned (level, path, anchor, line, size), so an agent knows exactly what it gets. It is clearly distinguished from siblings get_section and chunk_markdown, which are named in the same paragraph.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit positive trigger ("quick map of a long document before you read or chunk it, or to find the exact heading or anchor to pass to get_section") and an explicit exclusion ("Do not use it to get the text itself") with the correct alternatives named. There is no ambiguity about routing between outline_markdown and its siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Split a Markdown file into chunks and save them as a JSONL file (one chunk per line).
Use this when the chunks are going into a search index, vector database or another program, and you want a file instead of a long chat reply. The chunks are the same as chunk_markdown gives. It writes only the one .jsonl file and replaces it if it is already there. It never changes the Markdown file.
Do not use it to look at chunks in the chat (use chunk_markdown).
Returns {"output_path", "chunks", "bytes", "biggest_chunk_chars"}.
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | Full path to the local Markdown file to split (UTF-8, up to 10 MB). | |
| max_chars | No | Biggest chunk size in characters (not tokens). Default 1500. | |
| output_path | No | Where to save. Must end in .jsonl. Default: next to the input, as NAME.chunks.jsonl. | |
| prefix_path | No | If true, each chunk text starts with its heading path, like "Guide > Install". | |
| overlap_chars | No | Characters repeated from the end of one part at the start of the next, only when a section is split. Must be at most half of max_chars. Default 0. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=false, destructiveHint=true and idempotentHint=true, so the description's job is to scope the damage, which it does: it writes only the one .jsonl file and replaces it if present, and it never modifies the input Markdown. That clarifies exactly what the destructive hint applies to. It stops short of mentioning any failure modes (e.g., invalid output_path extension, unreadable input), which keeps it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four short blocks, front-loaded with the core action, then destination guidance, then the write-scope caveats, then the return shape. No sentence repeats the name or the schema, and each one carries distinct information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, and the description compensates by naming the returned keys (output_path, chunks, bytes, biggest_chunk_chars). Combined with full parameter documentation, explicit overwrite behavior, and sibling routing, an agent has everything needed to call this correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with defaults, bounds, and semantics (max_chars in characters not tokens, overlap limited to half of max_chars) all documented in the schema. The description adds only the relationship "the chunks are the same as chunk_markdown gives" and does not elaborate on output_path or prefix_path beyond what the schema states, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: "Split a Markdown file into chunks and save them as a JSONL file (one chunk per line)." It explicitly positions itself against the sibling chunk_markdown, noting the chunks are the same but the destination differs, so an agent can distinguish the two without reading either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a concrete when-to-use condition (chunks feeding a search index, vector database, or another program where a file is preferable to a long chat reply) and an explicit exclusion with the alternative named: "Do not use it to look at chunks in the chat (use chunk_markdown)." Both the positive and negative routing cases are covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v1.1.0chunk_markdownget_sectionoutline_markdownsave_chunks_jsonlScored across 4 tools
Each tool targets a distinct operation: chunking the whole document, listing headings, extracting one section, and persisting chunks to a file. Descriptions include explicit 'do not use it for X, use Y' guidance that makes the boundaries unambiguous.
All four names follow a verb_noun shape, but the noun varies: two end in _markdown (chunk_markdown, outline_markdown) while get_section and save_chunks_jsonl drop the markdown qualifier and name a sub-resource instead. Still predictable and readable, just not a single uniform pattern.
Four tools is exactly right for a focused Markdown chunking utility, with no redundant or filler entries. Each tool covers a clear step in the read-outline, inspect-section, chunk, persist workflow.
The core lifecycle (map, extract, chunk, export) is fully covered with no dead ends for the stated purpose. Minor gaps exist, such as no token-based chunk sizing (explicitly noted as character-based) and no batch/multi-file operation, but agents can work around both.
Convert files, URLs, and documents to clean, AI-ready Markdown via MCP.
Document-to-Markdown MCP server — convert PDF, Office and HTML into LLM-ready Markdown.
Converts PDF, scans, DOCX, XLSX and PPTX to Markdown with tables and page-aware RAG chunks.
Parse PDF/Word/PPT/HTML to Markdown; tables as JSON, image extraction, RAG chunking, page ranges.
