mcp-agent-docparser

SDK documentation extractor as a local MCP server. Point an MCP-aware agent harness at a documentation site, and it fetches the pages, strips navigation noise, converts HTML to clean markdown, and returns (or writes) a .md file ready to drop into any LLM context window.

Pipeline: fetch → extract → emit.

This is the terminal-menu docparser (a standalone project) rebuilt as an MCP tool for the agent-tools stack. The interactive REPL commands became MCP tools: receipt management, parse, and probe are now callable by any harness (Claude, opencode, etc.) over stdio.

Tool surface

Related MCP server: read-website-fast

Receipts

A receipt is a named profile that tells docparser how to fetch and extract one documentation site. Receipts live in receipts.json — never in the Python source. The repo ships a host of prefilled examples covering a wide range of real documentation stacks: Memvid (memvid-python, static Mintlify), LM Studio (lmstudio-python and lmstudio-bionic, JS-rendered Next.js with a Copy-as-Markdown button), Mojo, Rust, React.dev (Sandpack), Hermes Agent, MCP Specification, Pydantic AI, HuggingFace smolagents, Google ADK, LangChain DeepAgents, Invidious and SearXNG — see the urls, selectors, and strip_tags in receipts.json for ready-made examples per site theme. Delete them or keep them; they are regular receipts.

Receipt schema

{
  "my-sdk-python": {
    // Required
    "name":      "My SDK (Python)",          // human label
    "language":  "python",                   // used in code blocks and header
    "urls":      ["https://docs.example.com/overview"],  // fetched in order
    "selectors": ["#content", ".prose", "article", "body"],

    // Optional
    "strip_tags":           ["nav", "footer", "header", "script", "style"],
    "section":              null,            // restrict to content after this H2 text
    "js_render":            false,           // use Playwright instead of requests
    "markdown_passthrough": false,           // treat extracted <pre> as raw markdown
    "notes":                "",              // free-text notes about the site

    // Managed automatically
    "last_fetched": null,                    // ISO date of last successful parse
    "last_output":  null                     // filename of last emitted .md
  }
}

Workflow for a new site

1. doc_probe           → paste the docs URL, see which selectors match and
   (or doc_probe_js      what H2s exist. Nothing matched? The site is
    for JS-rendered)     probably JS-rendered — retry with doc_probe_js.
2. receipt_add         → save the findings as a receipt (best_selector +
                         strip_tags / js_render / markdown_passthrough).
3. doc_parse(key)      → fetch, extract, emit .md into doc_output/. Returns
                         the rendered markdown directly. doc_parse with
                         dry_run=true previews what will be fetched.

For a single unfamiliar page that matches an existing receipt's template, doc_parse_url(template_key, url) parses it without persisting a receipt.

Whole-site ingestion with doc_crawl

1. doc_probe / doc_probe_js → confirm the site renders + pick a template
   receipt whose selectors fit.
2. doc_crawl(key, url) with dry_run=true → discovery only: sitemaps/robots
   are read, candidate URLs are listed, nothing is fetched. Scope with
   max_pages and path_prefix if the list is too big.
3. doc_crawl(key, url) → fetches the discovered pages concurrently, emits one
   combined {name}_{timestamp}.md, and returns the markup inline.

Crawl discovery order: robots.txt Sitemap: directives → sitemap index → leaf sitemaps (a .xml seed is parsed directly) → same-host link walk as a fallback for sites without a sitemap. Politeness is robots-strict: every URL passes through the site's robots.txt before fetching, a declared Crawl-delay throttles the workers, and results are deduped, same-origin, and capped at max_pages (default 200). Crawls never modify the registry — they are one-shot like doc_parse_url.

Requirements

• Python 3.13+

• uv

• Playwright Chromium (only for js_render receipts): uv sync && uv run playwright install chromium

Install

git clone <repo-url> mcp_agent_docparser
cd mcp_agent_docparser
uv sync

Usage

Run as an MCP server (stdio — what harnesses expect)

uv run mcp-agent-docparser

Override paths via environment variables (defaults resolve relative to the project root, never the process CWD):

To serve over HTTP (streamable-http) instead:

uv run mcp-agent-docparser serve --http --port 8000

Register in an MCP client

{
  "mcpServers": {
    "mcp-agent-docparser": {
      "command": "uv",
      "args": ["--project", "C:/path/to/mcp_agent_docparser", "run", "mcp-agent-docparser"]
    }
  }
}

CLI commands

Output

Files are written to doc_output/ as {receipt_name}_{YYYYMMDD_HHMMSS}.md. Each file contains a header block (language, fetch timestamp, source URLs), the receipt's notes if set, and one <!-- SOURCE: url --> section per fetched page. doc_parse also returns the rendered markdown in its result, so the model can use the content even if the harness has no filesystem view.

Development

uv run ruff check .
uv run pytest -q

Tests cover the receipt registry, config path resolution, extraction, the parse pipeline (mocked fetch), and the full tool surface — no network needed.

Agent skill: fixing noisy fetched pages

When a fetched page comes out malformed (wrong code-fence language, blank lines in code blocks, leaked navigation noise, heading-anchor junk, CRLF copy-markdown), opencode loads the docparser-page-fixes skill from .opencode/skills/docparser-page-fixes/SKILL.md. It teaches the reproduce → isolate → fix → test → restart loop, maps the real symptom/cause/fix cases from this repo's history, and lists the extractor's gotchas (MULTILINE regexes, trailing-newline traps, the stale-MCP-module restart requirement). Extend it whenever the extractor grows a new fix.

License

MIT © 2026 Christof Milius