ducsplit

Conceptual document splitter for verified, study-sized chunks — CLI and MCP.

ducsplit splits large PDF and DOCX documents into study-sized conceptual chunks, not blind page ranges. It is built for developers using AI coding assistants (Claude Code, Codex, Cursor, Grok, OpenCode, or any MCP-capable host) who need to break textbooks, lecture notes, and manuals into sessions an agent can actually work with.

Repository: github.com/alifazelidehkordi/ducsplit

Table of Contents

• Why This Exists

• Key Features

• Architecture

• Requirements

• Installation

• Quick Start: CLI

• Quick Start: MCP

• CLI Reference

• MCP Tools

• Configuration

• Output Structure

• Host-Agent Workflow

• Output Format Trade-Offs

• Edge Cases and Limitations

• Troubleshooting

• Development

• Design Principles

• License

Related MCP server: BigContext MCP

Why This Exists

Long study documents overwhelm AI assistants: they exceed context windows, trigger lost-in-the-middle failures, and waste tokens. Simple page slicing makes it worse — it cuts through arguments, divorces tables from their explanations, and ignores how the author actually structured the material.

ducsplit combines deterministic document parsing with host-agent reasoning:

• The tool extracts structured elements from PDF or DOCX into a shared intermediate representation (IR).

• It proposes structurally safe boundary candidates — only after complete paragraphs, headings, lists, or tables.

• Your host AI agent reads the surrounding content and chooses the best conceptual cut (or extends the window if the topic continues).

• Output is verified for missing, duplicated, or mismatched content.

• Each chunk can be analyzed into bilingual Persian/English study metadata.

• Final study indexes link to every session and include a practical Study Focus column.

ducsplit does not bundle its own LLM. The host agent supplies judgment; ducsplit supplies parsing, constraints, persistence, writing, and verification.

Key Features

• Conceptual splitting — targets 5–10 page study sessions, but lets concept completeness override page count.

• Safe-cut constraints — the LLM can only choose from deterministic candidates, preventing mid-paragraph, mid-list, and mid-table cuts.

• Dual PDF parsing — pymupdf4llm for Markdown-like extraction plus OpenDataLoader for layout reconciliation when Java is available.

• DOCX support — headings, paragraphs, lists, tables, and embedded images via python-docx.

• CLI and MCP — use doc-splitter directly or expose it as an MCP server to AI coding assistants.

• Verified output — checks element coverage, duplicates, word-count drift, table rows, image references, and PDF page coverage.

• Markdown and PDF chunks — write semantic Markdown chunks, extracted PDF chunks, or both.

• Boundary overlap for PDFs — includes adjacent boundary pages to reduce loss when conceptual cuts occur mid-page.

• Agent-authored bilingual study indexes — provides verified context for the host agent to write study-index-fa.md and study-index-en.md with session links, study focus, page ranges, and estimated reading time.

• Agent-decision enforcement — rejects auto-generated reasons (auto-cut, auto from section_headings), template study-focus, and index commits without reading every chunk file.

• Realistic study-time estimates — defaults to 80 words per minute (study speed) rather than casual reading speed (200 wpm) for technical/medical content.

Architecture

flowchart TD
    A[PDF or DOCX input] --> B[Format Detector]

    B -->|PDF| C[pymupdf4llm Parser]
    B -->|PDF| D[OpenDataLoader Layout Parser]
    C --> E[Document IR]
    D --> F[PDF Reconciler]
    F --> E

    B -->|DOCX| G[python-docx Parser]
    G --> E

    E --> H[Structure Analyzer]
    H --> I[Safe Boundary Candidate Generator]
    I --> J{Host AI Agent}
    J -->|cut| K[Boundary Session]
    J -->|extend| I

    K --> L[Writer]
    L -->|markdown| M[.md Chunks]
    L -->|pdf| N[.pdf Chunks]
    L --> O[manifest.json]

    O --> P[Verifier]
    P --> Q[verification-report.json]

    M --> R{Host AI Agent}
    N --> R
    R --> S[Chunk Analysis Store]
    S --> T[Index Generator]
    T --> U[study-index-fa.md]
    T --> V[study-index-en.md]

Requirements

Java is recommended for better PDF reconciliation. Parsing falls back to pymupdf4llm alone if OpenDataLoader cannot run.

Installation

git clone https://github.com/alifazelidehkordi/ducsplit.git
cd ducsplit

python3 -m venv .venv
source .venv/bin/activate

pip install -e ".[dev]"
npm install

java -version   # must succeed for full PDF reconciliation

The Python package exposes the CLI:

doc-splitter --help

If the command is not on your shell path:

python -m doc_splitter.cli --help

Quick Start: CLI

Start a split session:

doc-splitter run \
  --input ./book.pdf \
  --out ./output/book \
  --min-pages 5 \
  --max-pages 10

This parses the document, writes ir.json, creates .split-session.json, and prints the first boundary decision context.

Repeat the boundary loop until the context reports status: "complete":

doc-splitter boundary-context --out ./output/book

Ask your host agent to read the JSON, choose one safe candidate, and return either:

{
  "action": "cut",
  "element_id": "el-042",
  "reason": "The current explanation ends here; the next section starts a different topic."
}

Commit the decision:

doc-splitter commit-boundary \
  --out ./output/book \
  --action cut \
  --element-id el-042 \
  --reason "The current explanation ends here; the next section starts a different topic."

If the concept continues beyond the current window:

doc-splitter commit-boundary \
  --out ./output/book \
  --action extend \
  --reason "The same mechanism continues into the next example."

After boundaries are complete, write and verify chunks:

doc-splitter write --out ./output/book

Analyze each chunk:

doc-splitter analysis-context --out ./output/book --chunk-id 1

Commit the host agent's bilingual analysis:

doc-splitter commit-analysis \
  --out ./output/book \
  --chunk-id 1 \
  --topic-fa "تنظیم گلوکز و پاسخ انسولین" \
  --topic-en "Glucose Regulation and Insulin Response" \
  --study-focus-fa "مسیرهای اصلی تنظیم قند خون، نقش انسولین و تفاوت پاسخ طبیعی و پاتولوژیک را مرور کنید." \
  --study-focus-en "Review the main glucose-control pathways, insulin's role, and how normal and pathological responses differ." \
  --coherence confident \
  --reason "The chunk covers one continuous regulatory mechanism."

Ask the host agent to write the final indexes after every chunk has analysis:

doc-splitter index --out ./output/book
doc-splitter commit-index \
  --out ./output/book \
  --fa-file ./agent-written-study-index-fa.md \
  --en-file ./agent-written-study-index-en.md

PDF-native chunks

doc-splitter run \
  --input ./book.pdf \
  --out ./output/book-pdf \
  --output-format pdf \
  --overlap-pages 1

Quick Start: MCP

After npm install, register the MCP server with your host CLI (replace the path if you cloned elsewhere):

REPO=/absolute/path/to/ducsplit

claude mcp add doc-splitter -s user -- node "$REPO/server.js"
codex mcp add doc-splitter -- node "$REPO/server.js"
grok mcp add doc-splitter -s user -- node "$REPO/server.js"
opencode mcp add doc-splitter -- node "$REPO/server.js"

Or use the helper script:

./scripts/install-mcp.sh

For project-level MCP configuration, use .mcp.json:

{
  "mcpServers": {
    "doc-splitter": {
      "command": "node",
      "args": ["/absolute/path/to/ducsplit/server.js"],
      "env": {
        "DOC_SPLITTER_PYTHON": "/absolute/path/to/ducsplit/.venv/bin/python3"
      }
    }
  }
}

Then ask your agent:

Use the doc-splitter MCP tools to split book.pdf into 5–10 page conceptual Markdown chunks under output/book. Choose only safe boundary candidates, write the chunks, verify integrity, analyze every chunk in Persian and English, then author and commit the study indexes.

CLI Reference

Commands

Common Options

Available on run, parse, boundary-context, commit-boundary, write, verify, and index unless noted.

run

doc-splitter run --input ./book.pdf --out ./output/book

Required: --input PATH (source .pdf or .docx).

parse

doc-splitter parse --input ./book.docx --out ./output/debug

Writes ir.json without starting a boundary workflow.

boundary-context

Returns JSON with status, chunk_number, content_window, safe_candidates, instructions, min_pages, and max_pages. The host agent must choose only from safe_candidates.

commit-boundary

write / verify

write writes chunk files, manifest.json, and verification-report.json. Exits non-zero if verification fails.

verify checks:

• every expected IR element appears exactly once

• no unknown elements are present

• chunk word counts match the source within tolerance

• Markdown tables preserve expected rows

• Markdown image references are present

• PDF outputs cover all non-skipped source pages

commit-analysis

coherence must be confident or needs_review. When all chunks have analysis, semantic-review-report.json is written.

index

Requires every chunk to have committed content analysis, including both study-focus fields. Returns verified chunk metadata and instructions for the host agent. It does not write the index prose.

commit-index

Stores the final study-index-fa.md and study-index-en.md bodies authored by the host agent. The command validates that every chunk file is referenced.

MCP Tools

The MCP server is a thin Node.js wrapper around the Python CLI. It uses DOC_SPLITTER_PYTHON when set; otherwise it runs python3.

Configuration

CLI / MCP Options

Internal Defaults

Defined in src/doc_splitter/config.py. Not all have CLI flags yet.

Output Structure

A typical Markdown run:

output/book/
├── 01_intro-to-glucose-regulation.md
├── 02-insulin-signaling-and-feedback.md
├── images/
├── ir.json
├── manifest.json
├── verification-report.json
├── semantic-review-report.json
├── study-index-fa.md
├── study-index-en.md
└── .split-session.json

A PDF or both run may also include .pdf files per chunk.

Important Files

Study Index Structure

Each agent-authored study-index-*.md should include:

1. Overview — session count, PDF page coverage, total study time

2. Session table — topic links, page ranges, estimated time, and Study Focus

3. Optional grouping — chapters or sections only when they naturally fit the document

4. Document-specific study guidance — only when useful for that source

Host-Agent Workflow

Boundary Loop

1. ducsplit parses the document into ordered elements.

2. It computes a window starting at the current cursor.

3. It finds safe cut candidates after complete headings, paragraphs, lists, or tables.

4. The host agent reads the content window and candidate list.

5. The host agent chooses cut with an allowed element_id, or extend if the concept is incomplete.

6. ducsplit validates the decision and moves the cursor.

7. The loop continues until the document is fully covered.

Example context returned to the host agent:

{
  "status": "needs_agent_decision",
  "chunk_number": 4,
  "content_window": "...",
  "safe_candidates": [
    {
      "element_id": "el-042",
      "after_type": "paragraph",
      "snippet": "This completes the comparison of...",
      "cumulative_word_count": 9850
    }
  ],
  "min_pages": 5,
  "max_pages": 10
}

The LLM must not invent boundaries. Committing an element_id not in safe_candidates fails.

Analysis Loop

1. analysis-context returns full chunk content and neighboring titles.

2. The host agent writes a concise bilingual topic and practical study focus. topic_en becomes the final chunk filename slug.

3. commit-analysis stores the result in .split-session.json, renames the chunk from the agent-selected topic, and updates manifest.json.

4. Chunks marked needs_review appear in semantic-review-report.json and block final index commit until the boundary/coherence issue is resolved.

5. Every get_chunk call is tracked. The agent must read all chunk files before commit-index is accepted.

6. index returns context for the final index-writing pass, including chunks_unread so the agent knows which files remain.

7. The host agent writes both complete Markdown indexes and stores them with commit-index.

The study focus should answer: What should the learner master in this session?

Good:

{
  "topic_en": "Insulin Signaling and Glucose Uptake",
  "study_focus_en": "Master the insulin receptor cascade, GLUT4 translocation, and how defects in the pathway explain insulin resistance."
}

Weak:

{
  "topic_en": "Insulin",
  "study_focus_en": "This section is about insulin."
}

Output Format Trade-Offs

For PDF output, conceptual chunks are decided by element boundaries, but written PDFs include whole pages. --overlap-pages reduces context loss at boundaries at the cost of duplicated pages between neighbors.

Edge Cases and Limitations

• Scanned PDFs — OCR is disabled by default. Pages without extractable text are skipped and reported.

• Password-protected PDFs — not supported.

• DOCX page numbers — estimated from word count; DOCX has no stable rendered page numbers.

• Non-Latin filenames — semantic slugs are ASCII-folded; fallback is section-N.

• Very short documents — may produce fewer or smaller chunks than the 5–10 page target.

• Very long concepts — the host agent should choose extend until a coherent cut appears.

• Tables and lists — treated atomically; the tool avoids cutting through them.

• OpenDataLoader failures — parsing continues with pymupdf4llm; recorded in reconciliation notes.

• PDF boundary precision — PDF chunks are page-level extracts, not element-level visual crops.

Troubleshooting

Development

ducsplit/
├── server.js                         # MCP server
├── pyproject.toml                    # Python package metadata
├── package.json                      # Node/MCP dependencies
├── scripts/install-mcp.sh            # MCP registration helper
├── src/doc_splitter/
│   ├── cli.py                        # CLI entry point
│   ├── orchestrator.py               # Pipeline coordination
│   ├── config.py                     # SplitConfig defaults
│   ├── format_detector.py            # PDF/DOCX detection
│   ├── ir/                           # Intermediate representation
│   ├── parsers/                      # PDF/DOCX parsers and reconciliation
│   ├── boundary/                     # Safe candidates and boundary session
│   ├── writers/                      # Markdown and PDF writers
│   ├── content/                      # Chunk analysis workflow
│   ├── verifier.py                   # Integrity verification
│   └── index_generator.py            # Agent-authored index context/commit
└── tests/

Run tests:

source .venv/bin/activate
python -m pytest -q

Run the MCP server directly (waits for stdio MCP protocol messages):

node server.js

Recommended workflow:

1. Add or update parser/writer behavior.

2. Add focused tests under tests/.

3. Run python -m pytest -q.

4. Test a small real PDF or DOCX through the CLI.

5. Inspect ir.json, manifest.json, and verification-report.json.

6. If MCP behavior changed, test through an MCP host with DOC_SPLITTER_PYTHON pointing at the local venv.

Design Principles

• LLM judgment is constrained — the agent decides among safe options; it does not rewrite parser state freely.

• Agent laziness is rejected — auto-generated reasons (auto-cut, auto from section_headings), template study-focus, and index commits without reading chunk files are caught by validators and refused.

• Verification is mandatory — generated chunks are auditable and checked for loss or duplication.

• Concepts beat page counts — 5–10 pages is a target, not a hard rule.

• No bundled LLM — ducsplit does not call an external LLM API; the host agent does.

• PDF fidelity and semantic precision differ — Markdown is best for semantic processing; PDF output is best for visual study continuity.

License

No license file is currently included. Until one is added, the repository is public but not explicitly open-source licensed. Recommended next step: add an OSI-approved license (MIT or Apache-2.0).