🩺 webdocs-mcp

Crawl any documentation site, index it into DuckDB with hybrid (semantic + BM25) search, and expose the whole thing to LLM agents as an MCP server — so your coding assistant reasons over current docs instead of its training cutoff.

Runs completely offline by default: the built-in hashing embedder needs no API key, and every test runs without touching the network. Set one environment variable to switch to OpenAI embeddings in production.

Why

LLM agents hallucinate stale APIs. The fix is retrieval over the actual docs of the actual version you use. webdocs-mcp is the smallest honest stack that does this end to end: point it at a docs site, it crawls with hierarchy tracking, chunks and embeds every page, and serves search_docs / get_doc_page tools over the Model Context Protocol to Cursor, VS Code, or Claude Code.

Related MCP server: LocalDocs MCP

Architecture

flowchart LR
    U["You / CI"] -->|POST /fetch_url| API["FastAPI"]
    API --> JOBS["Job runner<br/>(background threads)"]
    JOBS --> CRAWL["Crawler<br/>BFS, same-domain,<br/>parent/child hierarchy"]
    CRAWL --> CHUNK["Chunker<br/>paragraph-aware + overlap"]
    CHUNK --> EMB["Embedder<br/>hashing (offline) / OpenAI"]
    EMB --> DB[("DuckDB<br/>pages + chunks + embeddings")]
    AGENT["LLM agent<br/>(Cursor / VS Code / Claude Code)"] -->|MCP JSON-RPC| MCP["/mcp endpoint"]
    MCP --> SEARCH["Hybrid search<br/>cosine + BM25"]
    SEARCH --> DB
    BROWSER["Browser"] -->|/map| MAPS["Site maps<br/>pure HTML tree"]
    MAPS --> DB

Design choices worth calling out:

• DuckDB over a vector-DB server — a doc index for one team is thousands of chunks, not billions. One portable file, zero ops.

• BM25 + embeddings, always both — embeddings catch paraphrases ("how do I get my money back" → refund policy); BM25 catches exact identifiers (ERR_LOCK_TIMEOUT) that embeddings smear. Scores are min-max normalised and blended 60/40.

• Injectable fetcher — the crawler takes a plain (url) -> html callable. Tests inject a dict-backed fake site; production uses httpx; a headless browser slots in without touching crawl logic.

• MCP implemented directly — the streamable-HTTP core of MCP is JSON-RPC dispatch (initialize, tools/list, tools/call). Owning those ~100 lines keeps the stack dependency-light and fully offline-testable.

• Threads now, Redis when needed — jobs run on daemon threads behind a JobManager interface shaped like a queue consumer. docker-compose already ships the Redis service for the scale-out path.

Quickstart

git clone https://github.com/saianthireddy/webdocs-mcp.git
cd webdocs-mcp

python -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt

pytest            # 23 tests, fully offline
ruff check src tests

PYTHONPATH=src uvicorn webdocs.api:app --port 9111

Then:

# start a crawl
curl -X POST localhost:9111/fetch_url -H 'Content-Type: application/json' \
     -d '{"url": "https://docs.astral.sh/ruff", "max_pages": 30}'

# watch progress
curl localhost:9111/job_progress

# search what was indexed
curl 'localhost:9111/search_docs?query=how+do+I+ignore+a+rule&top_k=3'

Interactive OpenAPI docs at http://localhost:9111/docs, site maps at http://localhost:9111/map.

Docker

docker compose up --build
# app on :9111, index persisted in the webdocs-data volume

MCP integration

Add to Cursor / VS Code / Claude Code MCP configuration:

{
  "webdocs": {
    "type": "http",
    "url": "http://localhost:9111/mcp"
  }
}

Exposed tools:

API reference

Site maps

Crawled pages keep parent/child relationships, so /map renders a real navigable tree per site — breadcrumbs from root, sibling navigation, children listing — in pure HTML (no JavaScript). Pages fetched individually from the same domain group under one root.

Configuration

All optional (see .env.example):

Testing

pytest tests/ -v      # 23 tests: crawler, chunker, embedder, DB, hybrid search, API, MCP

No test touches the network — the crawler tests run against an in-memory fake site injected through the fetcher interface, and each test gets a throwaway DuckDB file.

Roadmap

• Respect robots.txt and add per-domain rate limiting

• Incremental re-crawls (etag/last-modified) instead of full re-index

• Redis-backed worker pool using the existing compose service

• OCR/text extraction for linked PDFs

• DuckDB VSS extension (HNSW) once the index outgrows brute-force cosine

License

MIT — see LICENSE.