docshelf-mcp
This server turns a folder of PDFs/Markdown into an AI-friendly docshelf with a compact INDEX.md, and gives MCP tools to add, read, search, organize, and maintain documents.
Initialize a shelf —
docshelf_init_shelfcreates the layout (docs/,INDEX.md,.docshelf.json), default categories, and remote/provider settings.Add documents —
docshelf_add_documentingests PDF/Markdown/DOCX/HTML/EPUB, converts PDFs to Markdown, auto-splits large files by H2 heading, writes per-document metadata, and regeneratesINDEX.md.Add directories in batch —
docshelf_add_directoryscans a folder for supported files, adds all of them under a category, and re-indexes once.Read content over MCP —
docshelf_read_documentreturns document/section content directly (useful for private/local shelves), with paging and path confinement todocs/.Search the shelf —
docshelf_searchdoes plain-text token search, returns snippets and raw fetch URLs, ranks heading/title matches, and falls back to any-token matching.List and organize —
docshelf_list_documentslists by category;docshelf_remove_documentanddocshelf_rename_documentdelete or retitle/recategorize documents with dry-run and no-clobber protection.Maintain integrity —
docshelf_rebuild_indexregenerates the index after manual edits;docshelf_doctordetects drift and can apply safe fixes.Convert PDFs standalone —
docshelf_convert_pdfconverts a PDF to Markdown outside any shelf, optionally splitting by H2.MCP resources — shelf files are exposed as read-only
docshelf:///...resources, refreshed after mutations and capped at 1 MB reads.Use as a Python library —
from docshelf_mcp import Shelflets you do the same operations programmatically without MCP.
Enables hosting a docshelf repository on GitHub, with INDEX.md containing raw.githubusercontent.com URLs for AI agents to fetch individual document sections on demand.
docshelf-mcp
Put your manuals on a shelf, hand the AI the index.
📖 Docs & landing page: https://ignatenkofi.github.io/docshelf-mcp/
_ _ _ __
__| | ___ ___ ___| |__ ___| |/ _|
/ _` |/ _ \ / __/ __| '_ \ / _ \ | |_
| (_| | (_) | (__\__ \ | | | __/ | _|
\__,_|\___/ \___|___/_| |_|\___|_|_|
MCP server for AI-friendly doc shelvesAn MCP server that turns a folder of PDFs and Markdown into a chat-project-friendly document collection: AI agents see a single INDEX.md and pull individual sections by raw GitHub URL on demand — instead of choking on a 4 MB datasheet.
Why?
You have 30 hardware manuals, or 200 cooking recipes, or a stack of research PDFs.
You want Claude / ChatGPT / whatever to be able to answer questions across them — but:
❌ You can't dump 80 MB of PDFs into a chat project. It won't fit, and you'd burn the context window even if it did.
❌ You can manually copy-paste the relevant pages, but only after you remember which manual mentioned the thing you need.
❌ Long files mean retrieval is wasteful — the model loads the whole RouterOS guide just to answer a question about VLANs.
docshelf-mcp solves it like this:
You drop a PDF onto the shelf.
The shelf converts it to Markdown, splits big files chapter-by-chapter, and regenerates a navigation
INDEX.md.You commit and push to a public GitHub repo.
Add only
INDEX.mdto your Claude project. When the model needs a section, it fetches it viaraw.githubusercontent.com.
Result: a 5 KB index pointing at a 50 MB collection. The model reads exactly the chapter it needs.
Related MCP server: Obsidian MCP Server
📦 Install
From PyPI:
# uv (recommended)
uv pip install docshelf-mcp
# or plain pip
pip install docshelf-mcpOr straight from main (always-latest, no PyPI required):
pip install "git+https://github.com/ignatenkofi/docshelf-mcp"That gives you Markdown shelves. Input formats beyond Markdown are extras — pick the ones you ingest:
pip install "docshelf-mcp[pdf]" # PDF via pymupdf4llm (the default engine)
pip install "docshelf-mcp[formats]" # PDF + DOCX + HTML + EPUB; or [docx] / [html] / [epub]The pdf extra pulls the PyMuPDF chain (~260 MB installed); a consumer that
only reads and writes Markdown does not need it, which is why it is not core.
Breaking since 0.5.0: PDF conversion needs the
[pdf]extra. Up to 0.4.1 a plainpip install docshelf-mcpconverted PDFs becausepymupdf4llmwas a core dependency; it is now optional (#93). Upgrading from 0.4.x and still adding PDFs? Installdocshelf-mcp[pdf](or[formats]), otherwiseadd_documenton aConversionErrortelling you the same.
Optional high-quality PDF engine (pulls ~2 GB of PyTorch — only if you need it):
pip install "docshelf-mcp[high-quality]"Python 3.14 (checked 2026-10-07):
[high-quality]does not install from wheels — everymarker-pdf1.x requirespillow<11, and Pillow 10 has no 3.14 wheels, so pip would have to build it from source. Use Python 3.10–3.13 for it.[pdf]and[formats]install from wheels on 3.14 on Linux, Windows x64 and Apple-silicon macOS 14+, but not on Intel Macs or macOS 13:onnxruntime, whichpymupdf4llmpulls in, has no 3.14 wheels there. To re-check from a checkout (exit 0 means wheels exist):uv pip compile pyproject.toml --extra high-quality --python-version 3.14 --only-binary :all:
📋 Project Prompt
Drop this into the Custom Instructions of any Claude project that consumes
a docshelf-style INDEX.md:
This project uses the docshelf pattern.
INDEX.mdis the entry point. When answering: read INDEX → fetch ONLY the needed section file via its GitHub raw URL (use WebFetch / fetch / curl). Don't load full source files into context. For large manuals split into chapters, follow INDEX → chapter SUBINDEX → section file.
Medium (~150 words) and full (~400 words) versions, plus how-to snippets for
Claude Code, Claude Desktop, and the Anthropic API, live in
docs/PROJECT_PROMPT.md.
Quickstart (Python library)
from docshelf_mcp import Shelf
shelf = Shelf("~/Documents/my-homelab-docs").init(
name="My HomeLab Docs",
remote="https://github.com/me/my-homelab-docs",
default_categories=["routers", "switches", "psu", "motherboards"],
)
shelf.add_document(
"~/Downloads/MIKROTIK_RouterOS.pdf",
category="routers",
title="Mikrotik RouterOS — full manual",
description="Official RouterOS reference, split by chapter.",
)
# → docs/routers/mikrotik-routeros-full-manual.md + docs/routers/.../001-..md, 002-..md, ...
# → INDEX.md is regenerated automatically.Then in the shelf directory: git add . && git commit -m "docs: add RouterOS" && git push.
In your Claude project, attach only INDEX.md. Done.
Quickstart (MCP server)
1. Add to Claude Desktop
Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%/Claude/claude_desktop_config.json (Windows):
{
"mcpServers": {
"docshelf": {
"command": "docshelf-mcp",
"env": {
"DOCSHELF_ROOT": "/Users/me/Documents/my-homelab-docs"
}
}
}
}Restart Claude Desktop. You now have eleven new tools available:
Tool | What it does |
| Bootstrap a new shelf directory. |
| Add a file (MD/PDF/DOCX/HTML/EPUB). Converts, splits, re-indexes. |
| Add every supported file (MD/PDF/DOCX/HTML/EPUB) in a folder in one call. Re-indexes once. |
| Read a document/section's content over MCP (works on private shelves). |
| Remove a document, its sections, and metadata. Re-indexes. |
| Retitle / recategorize a document (moves file, sections, meta) — no re-conversion. |
| Regenerate |
| Check shelf integrity; optionally auto-fix safe drift. |
| Plain-text search across the shelf, with raw URLs. |
| List documents by category. |
| Standalone PDF → Markdown (no shelf). |
The shelf files are also exposed as read-only MCP resources, so a client can browse and attach them natively — see MCP Resources below.
2. Add to Claude Code
claude mcp add docshelf -- docshelf-mcp
# Optional: pin a default shelf for this server
printf 'Path to your docshelf shelf: '; read -r shelf
shelf=${shelf/#\~/$HOME} # read does not expand a typed ~
shelf=$(cd "$shelf" && pwd) || exit 1 # absolute, and fails loudly if absent
claude mcp add docshelf --env DOCSHELF_ROOT="$shelf" -- docshelf-mcp3. Test from the command line
# Sanity check — should print the server version then wait on stdin
docshelf-mcpMCP Resources
Alongside the tools, every shelf file is exposed as a read-only MCP resource, so an MCP client (Claude Desktop, Claude Code, …) can browse and attach shelf content natively — no tool call required.
Scheme:
docshelf:///<relative-path>, e.g.docshelf:///INDEX.mdordocshelf:///docs/routers/mikrotik/003-firewall.md.What's exposed:
INDEX.mdplus every document and every split section underdocs/— one resource each. A split document exposes both its whole-file parent and its individual section files.Size cap: a resource read is capped at 1 MB (1,000,000 bytes). A larger file is truncated at a UTF-8 character boundary and ends with a notice pointing at the
docshelf_read_documenttool, which pages the rest.Freshness: content is read from disk on every access, and the resource set is re-synced when the server starts and after each mutating tool call (
add_document,add_directory,remove_document,rename_document,rebuild_index,init_shelf) — so newly added files appear and removed ones drop out. Reads are confined to the shelf root.
Resources are only registered for an initialized shelf (one that has a .docshelf.json); a non-shelf DOCSHELF_ROOT simply exposes none.
The shelf layout
my-shelf/
├── .docshelf.json ← shelf metadata: name, remote, category order
├── INDEX.md ← auto-generated navigation (your chat-project file)
├── .gitignore
└── docs/
├── routers/
│ ├── .meta.json ← per-document title/description overrides
│ ├── mikrotik-routeros.md (full document, lightly cleaned)
│ └── mikrotik-routeros/ (auto-split sections)
│ ├── SUBINDEX.md (per-document navigation page)
│ ├── 001-overview.md
│ ├── 002-bridging.md
│ └── 003-firewall.md
└── switches/
└── cudy-gs1010pe.mdEverything in docs/ is committed; everything is fetchable via raw URL once you push to GitHub.
How splitting works
A document is split when both conditions hold:
UTF-8 size > 50 KB (configurable via
.docshelf.json:split_threshold_bytes).The document has at least two
##(H2) headings.
The splitter:
Cleans PDF-extraction noise (collapses runaway blank lines, demotes CLI dumps mistaken for H1s).
Slices on H2 boundaries.
Names files
NNN-<slug>.mdso they sort naturally and survive title changes.Wipes the previous split directory before regenerating — fully idempotent.
Writes a
SUBINDEX.mdnavigation page into the split directory (title, description, per-section links) — regenerated on everyrebuild_index.
In INDEX.md, split documents with up to 10 sections list every section
inline; bigger splits get a single link to their SUBINDEX.md so the index
stays small. Control this via .docshelf.json:
"index_style": "auto" | "inline" | "subindex" and
"subindex_threshold_sections": 10.
If you want to keep a document whole, pass split=False.
Examples
See the examples/ directory for three concrete use cases:
examples/homelab/— original use case, hardware manuals for a home lab.examples/recipes/— a cookbook with one recipe per file.examples/research-papers/— academic PDFs with abstracts in.meta.json.
Each example shows the directory layout and the INDEX.md you'd end up with.
Optional: high-quality PDF conversion
The default engine (pymupdf4llm, installed by the pdf extra) is fast and good enough for ~95% of technical documents. For papers with complex tables, math, or scanned content, install the marker-pdf backend:
pip install "docshelf-mcp[high-quality]"Then pass quality="high":
shelf.add_document("paper.pdf", category="research", title="...", quality="high")⚠️ marker-pdf pulls in PyTorch (~2 GB) and is significantly slower (10–60 s per document on CPU). The library import is deferred — if you don't use quality="high", the dependency is never loaded. On Python 3.14 it does not install from wheels yet — see the 3.14 note under Install.
FAQ
Why GitHub raw URLs and not embeddings / RAG? Because it's dead simple, costs nothing to host, and the AI is already good at chasing links. You can layer embedding search on top later if you want — the on-disk shape is a normal git repo.
Does this work with private repos?
Partly. The raw-URL trick needs a public repo — raw.githubusercontent.com won't serve private ones without auth. But docshelf_search and docshelf_read_document both work over MCP on private (or purely local, non-git) shelves: the model searches, then reads the exact section's content directly from the server, no raw URL required. You only lose the ability to hand a bare INDEX.md to a chat project and have it fetch by URL — with the MCP server attached, the full flow works either way. Make the doc repo public if you want the URL-fetch path too.
Do I have to use GitHub?
No. Set provider in .docshelf.json (or at init_shelf): github (default), gitlab, gitea, custom, or none. The github provider also covers GitHub Enterprise Server: a self-hosted github.<company>.com remote gets the GHES raw form (https://<host>/<owner>/<repo>/raw/<branch>/<path>) automatically. custom takes a url_template with {owner}, {repo}, {branch}, {path} placeholders, so you can point at S3, Cloudflare R2, GitLab/Gitea raw, a GHES deployment on a fully custom domain, or any static host — the generated URLs are correct everywhere, no post-processing. none renders relative links in INDEX.md, which stay navigable offline / in a local checkout.
Does it edit the source PDFs?
No. PDFs are converted on add_document and the source is left in place. The shelf only writes inside its own directory.
What about non-English documents?
Slugify is Unicode-aware (NFKD-normalized, with \w under re.UNICODE). Cyrillic / CJK titles slug down to ASCII-ish forms; the body Markdown is preserved as-is.
Can I use it without MCP?
Yes — from docshelf_mcp import Shelf and use the class directly. See docs/USAGE.md.
Limitations
Public GitHub only for the raw-URL trick (or whatever public static host you wire up).
Single repo per shelf. If you outgrow one repo, run multiple shelves and attach multiple
INDEX.mds.Heuristic splitting. The PDF→Markdown extract isn't always clean enough to split cleanly. For pathological cases (some 4+ MB datasheets), keep the file whole and rely on
docshelf_search.No automatic git commit. Tools regenerate
INDEX.mdon disk, but the caller (you, or an agent) is responsible forgit add / commit / push. This is intentional — staying out of git's way keeps the tool safe to call from agents.
Demo — does it actually save tokens?
Measured on two real shelves (24 hardware manuals; a full novel split by chapter): answering a question the docshelf way costs ~3.7K tokens vs 1.22M to dump the collection — 99.7% fewer — and the biggest manual (RouterOS, ~1.05M tokens) doesn't even fit in a 200K context window, while a section fetch always does.
📊 Full write-up with the numbers, chart, and a reproducible benchmark:
docs/demo.md (run benchmarks/token_savings.py on your own shelf).
Architecture
For a deeper dive, see docs/ARCHITECTURE.md — module layout, data flow, design rationale.
Related projects
memshelf-mcp — the sibling project: the same index-and-fetch pattern applied to an AI agent's own working memory. Conversation episodes get offloaded to a private shelf with LLM-written digests; the agent keeps only
INDEX.mdin context and recalls sections on demand. Born as RFC-0001 in this repo; uses docshelf as its storage/index layer.
Contributing
Bug reports and PRs welcome. To set up a dev env:
git clone https://github.com/ignatenkofi/docshelf-mcp
cd docshelf-mcp
uv pip install -e ".[dev]"
ruff check src tests
pytest -vLicense
MIT — see LICENSE.
Origin
docshelf-mcp started life as a 350-line Python script (homelab-encyclopedia.py) that managed a single homelab manuals repo. The split / index / clean logic is the same code, generalised to work for any category-organised document collection.
Available Tools
11 toolsdocshelf_add_directoryAIdempotent
Add every matching file in a directory, rebuilding INDEX.md once.
Scans source_dir (non-recursively) for patterns — every supported
input type by default (Markdown, PDF, DOCX, HTML, EPUB) — adds each under
category with a title derived from its filename, and regenerates
INDEX.md a single time. A corrupt or unreadable file is reported in
failed without aborting the batch.
Inputs: params (required). Returns: result
docshelf_add_documentAIdempotent
Add a PDF or Markdown file to the shelf and refresh INDEX.md.
.pdfis converted to Markdown (pymupdf4llmby default; passquality='high'to usemarker-pdf).Documents larger than 50 KB with multiple H2 headings are split into one file per section (turn this off with
split=False).If a different title/category slugifies onto a path an existing document already occupies, the call errors instead of overwriting it — pass
overwrite=trueto replace it. Re-adding the same title updates in place. The response reportsoverwritten.The response
warningsinclude suspicious section headings and anempty-conversionwarning when the source yields little or no text (e.g. a scanned / image-only PDF — considerquality='high'/ OCR).INDEX.md is regenerated automatically. The caller still owns the git commit / push step.
Inputs: params (required). Returns: result
docshelf_convert_pdfAIdempotent
Standalone PDF → Markdown conversion (no shelf, no INDEX update).
Use when you want the converted file but don't yet want to commit it to a shelf. Optionally splits the result by H2.
Inputs: params (required). Returns: result
docshelf_doctorAIdempotent
Check the shelf for drift and optionally apply the safe fixes.
Reports stale .meta.json entries, orphaned split directories, split
sections out of sync with their parent, a stale INDEX.md, duplicate
titles, and empty categories. Read-only by default; pass fix=true to
prune stale meta entries, delete orphaned split dirs, and rebuild the
index (other findings stay report-only). Findings are sorted for stable
diffing.
Inputs: params (required). Returns: result
docshelf_init_shelfAIdempotent
Bootstrap a new document shelf at shelf_path.
Creates the directory layout (docs/, INDEX.md, .docshelf.json,
.gitignore), pre-creates any default_categories, and stores the
github_remote so generated INDEX entries link to raw GitHub URLs.
Idempotent — safe to call on an existing shelf to update metadata.
Inputs: params (required). Returns: result
docshelf_list_documentsARead-onlyIdempotent
List documents grouped by category.
Pass a category to filter; omit it to list everything.
Inputs: params (required). Returns: result
docshelf_read_documentARead-onlyIdempotent
Read a document or section file from inside the shelf's docs/.
Returns the file content directly over MCP — useful for private or
purely-local shelves where the raw.githubusercontent.com fetch
trick doesn't apply. Pass a relative_path from search /
list_documents. Large files are truncated to max_bytes (default
100 KB) with truncated: true; page with the returned next_offset
(slices snap to UTF-8 character boundaries, so it may differ from
offset + max_bytes — using it avoids splitting a multibyte character)
or read the individual split sections. Paths that escape docs/ are
rejected.
Inputs: params (required). Returns: result
docshelf_rebuild_indexAIdempotent
Regenerate INDEX.md from the current on-disk shelf state.
Useful after manual edits to docs/ or .docshelf.json.
Inputs: params (required). Returns: result
docshelf_remove_documentADestructive
Remove a document — its file, split sections, and metadata entry.
Accepts the filename, the slug, or the human title used at add time.
INDEX.md is regenerated automatically. Pass dry_run=true to see
what would be deleted without touching anything. The caller still owns
the git commit / push step.
Inputs: params (required). Returns: result
docshelf_rename_documentADestructive
Retitle, recategorize, or re-describe a document — no re-conversion.
Moves the document .md, its split-section directory, and its
.meta.json entry (changing the slug when the title changes, or the
directory when the category changes), then regenerates INDEX.md. Give at
least one of new_title / new_category / new_description. Refuses
to clobber an existing target. Pass dry_run=true to preview. The caller
still owns the git commit / push step.
Inputs: params (required). Returns: result
docshelf_searchARead-onlyIdempotent
Plain-text search across every Markdown file in the shelf.
Tokens are space-split; each must appear (case-insensitive) for a hit
to count. If no file contains all tokens, the search falls back to
any-token matching and the response reports match_mode: "any".
Heading/title matches are ranked above body-only matches, and for a
split document the section files are returned rather than the
whole-file parent. Results include the relative path, a
word-boundary-trimmed snippet, and — if a remote is configured — the
fetch URL so the model can pull the matching file directly.
Inputs: params (required). Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 11 tools
Each tool targets a distinct operation: init, add single, add batch, read, remove, rename, rebuild index, doctor check, search, list, and standalone conversion. While add_document and convert_pdf both handle PDF conversion, the former commits to the shelf while the latter does not, making their purposes clearly separable. No two tools appear to overlap in function.
All tools follow a consistent `docshelf_<verb>_<noun>` snake_case pattern, with clear, descriptive verbs like add, remove, rename, search, list, convert. The only slightly unusual verb is 'doctor', but it is intuitive for a health-check tool. The prefix is uniformly applied across all 11 tools, creating a strong, predictable convention.
11 tools is well within the ideal 3–15 range for a domain-specific server. Each tool serves a distinct need in the document shelf lifecycle, from initialization and document ingestion to search, maintenance, and standalone conversion. No tool feels superfluous, and the surface is neither too sparse nor overloaded.
The tool set covers the full lifecycle of a document shelf: create (init), add documents (both single and batch), read, remove, rename (update metadata), list, search, and rebuild the index. Additionally, a 'doctor' tool addresses drift repair and a standalone PDF converter supports pre-shelf workflows. There are no obvious gaps or dead ends for the stated purpose.
Maintenance
Related MCP Connectors
Your personal library, readable by AI agents: durable snapshots, hybrid search, 46 MCP tools.
Cloud or self-hosted knowledge for AI agents: hybrid search, reranking, GraphRAG, scoped MCP tools.
Self-hosted AI workspace: semantic RAG document search, long-term memory, and AI chat via MCP.
One place for every AI agent's pages and docs: versioned links to share, search and update.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables interaction with Notion databases through the Notion API, supporting full CRUD operations on pages and databases. Supports advanced querying, filtering, sorting, and all property types with Docker deployment for easy integration with Cursor and Claude.8MIT
- AlicenseAqualityDmaintenanceConnects to Obsidian vaults via the Local REST API plugin, enabling AI-assisted Zettelkasten workflows including creating atomic notes, searching content, managing links and tags, and performing precise content editing operations.13MIT
- AlicenseNot gradedqualityDmaintenanceAn MCP server that enables AI agents to query specialized, domain-specific knowledge bases built using the LightRAG framework for enhanced retrieval-augmented generation. It allows for managing and searching knowledge graphs and vector embeddings to provide accurate, context-aware information during an AI assistant's reasoning process.34 PyPI59MIT
- AlicenseNot gradedqualityBmaintenanceZero-config knowledge base for AI coding agents. Loads your markdown docs into a searchable database and exposes them as MCP tools — search, read, and manage documentation without leaving your editor. Works instantly with SQLite (no setup), upgrades to PostgreSQL + pgvector for hybrid semantic search. 6 MCP tools, 3 resources, FTS5 keyword search, 176 tests.79 PyPI31MIT