<p align="center">
  <img src="docs/assets/banner.png" alt="webdocs-mcp" width="100%" />
</p>

<p align="center">
  <a href="https://github.com/saianthireddy/webdocs-mcp/actions/workflows/ci.yml"><img src="https://github.com/saianthireddy/webdocs-mcp/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <img src="https://img.shields.io/badge/python-3.10%20%7C%203.12-blue" alt="Python">
  <img src="https://img.shields.io/badge/license-MIT-green" alt="License">
</p>

# 🩺 webdocs-mcp

Crawl any documentation site, index it into DuckDB with hybrid (semantic + BM25) search, and expose the whole thing to LLM agents as an **MCP server** — so your coding assistant reasons over *current* docs instead of its training cutoff.

Runs completely **offline by default**: the built-in hashing embedder needs no API key, and every test runs without touching the network. Set one environment variable to switch to OpenAI embeddings in production.

## Why

LLM agents hallucinate stale APIs. The fix is retrieval over the *actual* docs of the *actual* version you use. webdocs-mcp is the smallest honest stack that does this end to end: point it at a docs site, it crawls with hierarchy tracking, chunks and embeds every page, and serves `search_docs` / `get_doc_page` tools over the Model Context Protocol to Cursor, VS Code, or Claude Code.

## Architecture

```mermaid
flowchart LR
    U["You / CI"] -->|POST /fetch_url| API["FastAPI"]
    API --> JOBS["Job runner<br/>(background threads)"]
    JOBS --> CRAWL["Crawler<br/>BFS, same-domain,<br/>parent/child hierarchy"]
    CRAWL --> CHUNK["Chunker<br/>paragraph-aware + overlap"]
    CHUNK --> EMB["Embedder<br/>hashing (offline) / OpenAI"]
    EMB --> DB[("DuckDB<br/>pages + chunks + embeddings")]
    AGENT["LLM agent<br/>(Cursor / VS Code / Claude Code)"] -->|MCP JSON-RPC| MCP["/mcp endpoint"]
    MCP --> SEARCH["Hybrid search<br/>cosine + BM25"]
    SEARCH --> DB
    BROWSER["Browser"] -->|/map| MAPS["Site maps<br/>pure HTML tree"]
    MAPS --> DB
```

Design choices worth calling out:

- **DuckDB over a vector-DB server** — a doc index for one team is thousands of chunks, not billions. One portable file, zero ops.
- **BM25 + embeddings, always both** — embeddings catch paraphrases ("how do I get my money back" → refund policy); BM25 catches exact identifiers (`ERR_LOCK_TIMEOUT`) that embeddings smear. Scores are min-max normalised and blended 60/40.
- **Injectable fetcher** — the crawler takes a plain `(url) -> html` callable. Tests inject a dict-backed fake site; production uses httpx; a headless browser slots in without touching crawl logic.
- **MCP implemented directly** — the streamable-HTTP core of MCP is JSON-RPC dispatch (`initialize`, `tools/list`, `tools/call`). Owning those ~100 lines keeps the stack dependency-light and fully offline-testable.
- **Threads now, Redis when needed** — jobs run on daemon threads behind a `JobManager` interface shaped like a queue consumer. docker-compose already ships the Redis service for the scale-out path.

## Quickstart

```bash
git clone https://github.com/saianthireddy/webdocs-mcp.git
cd webdocs-mcp

python -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt

pytest            # 23 tests, fully offline
ruff check src tests

PYTHONPATH=src uvicorn webdocs.api:app --port 9111
```

Then:

```bash
# start a crawl
curl -X POST localhost:9111/fetch_url -H 'Content-Type: application/json' \
     -d '{"url": "https://docs.astral.sh/ruff", "max_pages": 30}'

# watch progress
curl localhost:9111/job_progress

# search what was indexed
curl 'localhost:9111/search_docs?query=how+do+I+ignore+a+rule&top_k=3'
```

Interactive OpenAPI docs at `http://localhost:9111/docs`, site maps at `http://localhost:9111/map`.

## Docker

```bash
docker compose up --build
# app on :9111, index persisted in the webdocs-data volume
```

## MCP integration

Add to Cursor / VS Code / Claude Code MCP configuration:

```json
{
  "webdocs": {
    "type": "http",
    "url": "http://localhost:9111/mcp"
  }
}
```

Exposed tools:

| Tool | Purpose |
|---|---|
| `search_docs` | Hybrid search over every indexed chunk; returns text + source URLs |
| `list_doc_pages` | List indexed pages so the agent can pick one |
| `get_doc_page` | Full extracted text of one page |

## API reference

| Endpoint | Description |
|---|---|
| `POST /fetch_url` | Start a crawl job (`{url, max_pages?, max_depth?}`); `?sync=true` blocks until done |
| `GET /job_progress` | All jobs, or one via `?job_id=` |
| `GET /search_docs` | `?query=&top_k=` hybrid search |
| `GET /list_doc_pages` | Every indexed page |
| `GET /get_doc_page` | `?page_id=` full page text |
| `GET /map` | Index of crawled sites (pure HTML, no JS) |
| `GET /map/site/{root_id}` | Hierarchical tree of one site |
| `GET /map/page/{id}` | Page view with breadcrumbs, siblings, children |
| `GET /map/page/{id}/raw` | Raw extracted text |
| `POST /mcp` | MCP JSON-RPC (initialize, tools/list, tools/call) |
| `GET /health` | Status + page/chunk counts |

## Site maps

Crawled pages keep parent/child relationships, so `/map` renders a real navigable tree per site — breadcrumbs from root, sibling navigation, children listing — in pure HTML (no JavaScript). Pages fetched individually from the same domain group under one root.

## Configuration

All optional (see `.env.example`):

| Variable | Default | Meaning |
|---|---|---|
| `WEBDOCS_DB_PATH` | `data/webdocs.duckdb` | Index location (falls back to in-memory if unwritable) |
| `WEBDOCS_EMBEDDER` | `hashing` | `hashing` (offline) or `openai` |
| `OPENAI_API_KEY` | — | Only for `openai` embedder |
| `WEBDOCS_MAX_PAGES` / `WEBDOCS_MAX_DEPTH` | 50 / 3 | Crawl limits |
| `WEBDOCS_CHUNK_SIZE` / `WEBDOCS_CHUNK_OVERLAP` | 1200 / 150 | Chunking |
| `WEBDOCS_RESPECT_ROBOTS` | `true` | Obey `robots.txt`. Only disable for sites you own |
| `WEBDOCS_PRUNE_MISSING` | `true` | Delete indexed pages no longer linked from the site |
| `WEBDOCS_CRAWL_DELAY` | `1.0` | Seconds between requests to a host (floor — a site asking for more wins) |

## Re-crawling incrementally

Re-crawling used to duplicate the index. Page ids were random `uuid4`s, so the
`INSERT OR REPLACE` had nothing to replace — crawling a two-page site three
times left six page rows, three identical roots, and every chunk indexed three
times, which quietly triples what search ranks over. Identity is now
`sha256(root_url + url)`, so a re-crawl updates rows in place.

With identity stable, cache validators do useful work:

- **`ETag` and `Last-Modified` are stored per page** and sent back as
  `If-None-Match` / `If-Modified-Since` on the next crawl.
- **A 304 skips re-indexing** — no body downloaded, no chunking, no embedding.
  Jobs report it separately as `pages_unchanged`.
- **Traversal continues past a 304** using the links that page yielded last
  crawl. A 304 has no body, so there is nothing to parse for links; the stored
  parent/child edges are the only way onward.
- **Pages that vanish are pruned.** If a URL is no longer linked anywhere on the
  site, its page row and its chunks are deleted, so a restructured docs site
  does not leave the index answering from URLs that no longer exist.
- **Older index files are migrated on open**, not rejected: the `etag` and
  `last_modified` columns are added if missing, and legacy rows simply have no
  validators so they re-fetch once.

The fetcher contract widened without breaking. It was `(url) -> html`; it can
now also be `(url, etag, last_modified) -> FetchResult`, and `as_conditional`
detects which one it was handed. A plain fetcher never reports `not_modified`,
so passing one behaves exactly as before — correct, just not saving work.

```python
crawl(url)                      # plain fetcher, full re-read
crawl(url, validators=..., known_links=..., on_unchanged=...)   # conditional
```

### When pruning deliberately does nothing

Deleting indexed content is the only operation here that loses data, so it only
runs when the crawl can positively explain every URL it knew about. A page is
kept — not pruned — if it:

- **failed to fetch.** The fetcher contract returns `str` and surfaces no status
  code, so a 404 and a 503 are indistinguishable. Deleting a page because of a
  transient error is much worse than leaving a dead one indexed.
- **was disallowed by `robots.txt`.** Being told not to look is not evidence of
  absence.
- **was never reached because `max_pages` cut the crawl short.** Then absence is
  unexplained rather than meaningful, and pruning is skipped entirely for that
  run.

Pruning is also scoped to one root, derived from the root URL, so re-crawling
site A can never delete site B's pages. `CrawlResult` carries the four outcome
sets and the `truncated` flag that make this decidable; `test_pruning.py` asserts
each refusal case separately.

## Crawling politely

Pointing a crawler at someone else's site is the one thing here with
consequences outside this repo, so it is on by default rather than opt-in:

- **`robots.txt` is fetched once per crawl** and every candidate URL is checked
  against it, both before queueing and before fetching. Disallowed paths are
  skipped and logged.
- **Requests to a host are spaced** by `WEBDOCS_CRAWL_DELAY` (1s default). If
  `robots.txt` declares a larger `Crawl-delay`, the site's number wins.
- **Missing or unreachable `robots.txt` means allow-all**, per RFC 9309 for a
  404. This is a deliberate deviation for 5xx — the injected fetcher surfaces
  no status code, so a 503 and a DNS failure look identical, and blocking every
  crawl on a transient blip is the worse failure mode.
- **Fractional `Crawl-delay` is honoured.** `urllib.robotparser` guards on
  `str.isdigit()` and silently discards `Crawl-delay: 0.5`, so `robots.py`
  re-reads the value itself and takes whichever is larger.
- **Known deviation:** conflicting rules resolve first-match-wins
  (`urllib.robotparser`) rather than RFC 9309's longest-match, so
  `Disallow: /docs/` followed by `Allow: /docs/public/` blocks the latter. The
  error is always over-restrictive — pages get skipped, never wrongly fetched.

Both pieces are injected, not baked in: `RobotsPolicy` takes the same
`(url) -> html` fetcher the crawler uses, and `Throttle` takes clock and sleep
callables, so the whole thing is asserted against a fake clock instead of
spending real seconds in CI.

```python
crawl(url, respect_robots=False)   # sites you control
crawl(url, crawl_delay=5.0)        # be extra gentle
```

## Testing

```bash
pytest tests/ -v      # 65 tests: crawler, robots/throttle, incremental re-crawl, pruning, chunker, embedder, DB, hybrid search, API, MCP
```

No test touches the network — the crawler tests run against an in-memory fake site injected through the fetcher interface, and each test gets a throwaway DuckDB file.

## Roadmap

- [ ] Redis-backed worker pool using the existing compose service
- [ ] OCR/text extraction for linked PDFs
- [ ] DuckDB VSS extension (HNSW) once the index outgrows brute-force cosine

## License

MIT — see [LICENSE](LICENSE).