🩺 webdocs-mcp

Crawl any documentation site, index it into DuckDB with hybrid (semantic + BM25) search, and expose the whole thing to LLM agents as an MCP server — so your coding assistant reasons over current docs instead of its training cutoff.

Runs completely offline by default: the built-in hashing embedder needs no API key, and every test runs without touching the network. Set one environment variable to switch to OpenAI embeddings in production.

Why

LLM agents hallucinate stale APIs. The fix is retrieval over the actual docs of the actual version you use. webdocs-mcp is the smallest honest stack that does this end to end: point it at a docs site, it crawls with hierarchy tracking, chunks and embeds every page, and serves search_docs / get_doc_page tools over the Model Context Protocol to Cursor, VS Code, or Claude Code.

Related MCP server: LocalDocs MCP

Architecture

flowchart LR
    U["You / CI"] -->|POST /fetch_url| API["FastAPI"]
    API --> JOBS["Job runner<br/>(background threads)"]
    JOBS --> CRAWL["Crawler<br/>BFS, same-domain,<br/>parent/child hierarchy"]
    CRAWL --> CHUNK["Chunker<br/>paragraph-aware + overlap"]
    CHUNK --> EMB["Embedder<br/>hashing (offline) / OpenAI"]
    EMB --> DB[("DuckDB<br/>pages + chunks + embeddings")]
    AGENT["LLM agent<br/>(Cursor / VS Code / Claude Code)"] -->|MCP JSON-RPC| MCP["/mcp endpoint"]
    MCP --> SEARCH["Hybrid search<br/>cosine + BM25"]
    SEARCH --> DB
    BROWSER["Browser"] -->|/map| MAPS["Site maps<br/>pure HTML tree"]
    MAPS --> DB

Design choices worth calling out:

• DuckDB over a vector-DB server — a doc index for one team is thousands of chunks, not billions. One portable file, zero ops.

• BM25 + embeddings, always both — embeddings catch paraphrases ("how do I get my money back" → refund policy); BM25 catches exact identifiers (ERR_LOCK_TIMEOUT) that embeddings smear. Scores are min-max normalised and blended 60/40.

• Injectable fetcher — the crawler takes a plain (url) -> html callable. Tests inject a dict-backed fake site; production uses httpx; a headless browser slots in without touching crawl logic.

• MCP implemented directly — the streamable-HTTP core of MCP is JSON-RPC dispatch (initialize, tools/list, tools/call). Owning those ~100 lines keeps the stack dependency-light and fully offline-testable.

• Threads now, Redis when needed — jobs run on daemon threads behind a JobManager interface shaped like a queue consumer. docker-compose already ships the Redis service for the scale-out path.

Quickstart

git clone https://github.com/saianthireddy/webdocs-mcp.git
cd webdocs-mcp

python -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt

pytest            # 23 tests, fully offline
ruff check src tests

PYTHONPATH=src uvicorn webdocs.api:app --port 9111

Then:

# start a crawl
curl -X POST localhost:9111/fetch_url -H 'Content-Type: application/json' \
     -d '{"url": "https://docs.astral.sh/ruff", "max_pages": 30}'

# watch progress
curl localhost:9111/job_progress

# search what was indexed
curl 'localhost:9111/search_docs?query=how+do+I+ignore+a+rule&top_k=3'

Interactive OpenAPI docs at http://localhost:9111/docs, site maps at http://localhost:9111/map.

Docker

docker compose up --build
# app on :9111, index persisted in the webdocs-data volume

MCP integration

Add to Cursor / VS Code / Claude Code MCP configuration:

{
  "webdocs": {
    "type": "http",
    "url": "http://localhost:9111/mcp"
  }
}

Exposed tools:

API reference

Site maps

Crawled pages keep parent/child relationships, so /map renders a real navigable tree per site — breadcrumbs from root, sibling navigation, children listing — in pure HTML (no JavaScript). Pages fetched individually from the same domain group under one root.

Configuration

All optional (see .env.example):

Re-crawling incrementally

Re-crawling used to duplicate the index. Page ids were random uuid4s, so the INSERT OR REPLACE had nothing to replace — crawling a two-page site three times left six page rows, three identical roots, and every chunk indexed three times, which quietly triples what search ranks over. Identity is now sha256(root_url + url), so a re-crawl updates rows in place.

With identity stable, cache validators do useful work:

• ETag and Last-Modified are stored per page and sent back as If-None-Match / If-Modified-Since on the next crawl.

• A 304 skips re-indexing — no body downloaded, no chunking, no embedding. Jobs report it separately as pages_unchanged.

• Traversal continues past a 304 using the links that page yielded last crawl. A 304 has no body, so there is nothing to parse for links; the stored parent/child edges are the only way onward.

• Pages that vanish are pruned. If a URL is no longer linked anywhere on the site, its page row and its chunks are deleted, so a restructured docs site does not leave the index answering from URLs that no longer exist.

• Older index files are migrated on open, not rejected: the etag and last_modified columns are added if missing, and legacy rows simply have no validators so they re-fetch once.

The fetcher contract widened without breaking. It was (url) -> html; it can now also be (url, etag, last_modified) -> FetchResult, and as_conditional detects which one it was handed. A plain fetcher never reports not_modified, so passing one behaves exactly as before — correct, just not saving work.

crawl(url)                      # plain fetcher, full re-read
crawl(url, validators=..., known_links=..., on_unchanged=...)   # conditional

When pruning deliberately does nothing

Deleting indexed content is the only operation here that loses data, so it only runs when the crawl can positively explain every URL it knew about. A page is kept — not pruned — if it:

• failed to fetch. The fetcher contract returns str and surfaces no status code, so a 404 and a 503 are indistinguishable. Deleting a page because of a transient error is much worse than leaving a dead one indexed.

• was disallowed by robots.txt. Being told not to look is not evidence of absence.

• was never reached because max_pages cut the crawl short. Then absence is unexplained rather than meaningful, and pruning is skipped entirely for that run.

Pruning is also scoped to one root, derived from the root URL, so re-crawling site A can never delete site B's pages. CrawlResult carries the four outcome sets and the truncated flag that make this decidable; test_pruning.py asserts each refusal case separately.

Crawling politely

Pointing a crawler at someone else's site is the one thing here with consequences outside this repo, so it is on by default rather than opt-in:

• robots.txt is fetched once per crawl and every candidate URL is checked against it, both before queueing and before fetching. Disallowed paths are skipped and logged.

• Requests to a host are spaced by WEBDOCS_CRAWL_DELAY (1s default). If robots.txt declares a larger Crawl-delay, the site's number wins.

• Missing or unreachable robots.txt means allow-all, per RFC 9309 for a

  404. This is a deliberate deviation for 5xx — the injected fetcher surfaces no status code, so a 503 and a DNS failure look identical, and blocking every crawl on a transient blip is the worse failure mode.

• Fractional Crawl-delay is honoured. urllib.robotparser guards on str.isdigit() and silently discards Crawl-delay: 0.5, so robots.py re-reads the value itself and takes whichever is larger.

• Known deviation: conflicting rules resolve first-match-wins (urllib.robotparser) rather than RFC 9309's longest-match, so Disallow: /docs/ followed by Allow: /docs/public/ blocks the latter. The error is always over-restrictive — pages get skipped, never wrongly fetched.

Both pieces are injected, not baked in: RobotsPolicy takes the same (url) -> html fetcher the crawler uses, and Throttle takes clock and sleep callables, so the whole thing is asserted against a fake clock instead of spending real seconds in CI.

crawl(url, respect_robots=False)   # sites you control
crawl(url, crawl_delay=5.0)        # be extra gentle

Testing

pytest tests/ -v      # 65 tests: crawler, robots/throttle, incremental re-crawl, pruning, chunker, embedder, DB, hybrid search, API, MCP

No test touches the network — the crawler tests run against an in-memory fake site injected through the fetcher interface, and each test gets a throwaway DuckDB file.

Roadmap

• Redis-backed worker pool using the existing compose service

• OCR/text extraction for linked PDFs

• DuckDB VSS extension (HNSW) once the index outgrows brute-force cosine

License

MIT — see LICENSE.