# AskDocs

Ask a set of docs a question and get an answer that cites the exact passages it came
from, so you can check it. Ships with the FastAPI documentation, works with any folder
of Markdown, text or PDF files, and is also an **MCP server**, so Claude Code or
Cursor can search the same docs.

![AskDocs answering a FastAPI question with numbered citations and source passages](docs/screenshot.png)

## What it does

- **Hybrid retrieval:** BM25 keyword search plus Gemini embeddings, combined with
  Reciprocal Rank Fusion. Falls back to keyword-only if no embedding key is set or the
  API fails.
- **Grounded answers:** the model may only use the numbered sources and must cite them
  as `[n]`. Each citation links to the passage and to the section on the docs website.
- **Checks the model's work:** citations to sources that don't exist are removed and
  flagged. Answers with no citations are marked as unverified.
- **Knows when to say no:** if retrieval finds nothing relevant, it refuses *without
  calling the LLM*, so quota isn't spent on a confident wrong answer.
- **MCP server:** `search_docs`, `read_doc` and (locally) `ask_docs` tools, over stdio
  or Streamable HTTP at `/mcp`.
- **Gemini or Groq:** choose with an environment variable. With no key at all it runs
  in search-only mode.
- **Ready for a public demo:** a per-IP rate limit, a daily cap on LLM calls, and a
  cache for repeated questions.

## How it works

```
docs/*.md ──▶ chunk by heading ──▶ BM25 index ─────────┐
              (code blocks kept     Gemini embeddings ─┤─▶ RRF ─▶ top 5 ─▶ in scope? ──no──▶ "not in these docs"
               whole, anchors →     (cached on disk)   │                     │yes
               citation URLs)                          │                     ▼
question ──────────────────────────────────────────────┘     prompt with [1]..[5] ─▶ Gemini / Groq
                                                                                       │
                              answer + verified citations + source cards ◀── check citations
```

1. **Ingest** (`chunking.py`): split Markdown on headings (levels 1–4). Code blocks are
   never split. Each chunk keeps its title, heading path and a URL to the exact section.
2. **Retrieve** (`retrieval.py`): BM25 over title + heading + text, with code
   identifiers indexed whole and split (`response_model` → `response`, `model`). When a
   Gemini key is set, cosine similarity over embeddings is fused in with RRF (k = 60).
3. **Decide** (`retrieval.py`): a question is out of scope when the top passages cover
   less than 45% of its meaning, weighting each word by rarity (IDF), and (in hybrid
   mode) the best embedding similarity is also low.
4. **Answer** (`rag.py`): a prompt with numbered sources. Citations are parsed (ignoring
   code like `items[0]`), checked against the sources, and returned with the answer.

## Retrieval quality

`evals/questions.jsonl` has 35 questions with the document that should answer each (25
use the docs' own words, 10 are paraphrased), plus 6 off-topic questions that should be
refused. `askdocs eval` runs it, and CI fails if it regresses.

Keyword mode (no API key):

| Metric | Value |
| --- | --- |
| Hit@1 | 0.77 |
| Hit@5 | 0.89 |
| MRR | 0.81 |
| Off-topic questions refused | 0.67 |
| On-topic questions wrongly refused | 0.00 |
| Hit@5, keyword questions | 0.96 |
| Hit@5, paraphrased questions | 0.70 |

Keyword search is strong when the question uses the docs' words, and weaker on
paraphrases ("run code after the response is sent" vs "background tasks"). That gap
is what the embedding half of hybrid search is for. The refusal threshold favours never
refusing a real question; [docs/agent-log.md](docs/agent-log.md) explains the trade-off.

## Run it

Requires Python 3.11+.

```bash
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
cp .env.example .env               # add GEMINI_API_KEY or GROQ_API_KEY (both have free tiers)
askdocs doctor                     # checks the key and that the model names exist
askdocs serve                      # http://127.0.0.1:8000
```

Other commands:

```bash
askdocs ask "How do I add CORS middleware?"
askdocs search "upload files"      # retrieval only, no LLM
askdocs eval -v                    # retrieval metrics + misses
```

### Use it from Claude Code or Cursor (MCP)

```bash
claude mcp add askdocs -- askdocs mcp                              # local, stdio
claude mcp add --transport http askdocs https://YOUR-HOST/mcp     # deployed app
```

Then ask your agent something like "using askdocs, how do I close a DB session after each
request?". It will call `search_docs`, then `read_doc` on the best page. The
public HTTP endpoint only exposes search and read: the calling agent is already an LLM,
and a public server shouldn't spend its owner's quota.

### Your own docs

Point `ASKDOCS_CORPUS_DIR` at any folder of `.md`, `.txt` or `.pdf` files. Add a
`corpus.json` with `{"name": "...", "base_url": "https://..."}` to get citation links.
The bundled corpus is rebuilt from the FastAPI repo with `scripts/build_corpus.py`.

## Tests

```bash
pytest          # 66 tests: chunking, BM25, RRF, prompts, citations, providers, API, MCP, eval gate
pytest e2e      # 7 Playwright browser tests against the running app
```

Tests never call a real model. Providers are tested against `httpx.MockTransport`,
and the app uses a fake LLM, so the suite runs offline with no keys. The browser
tests include one that feeds the UI a model answer containing `<img onerror>` and
`<script>`, and checks that nothing executes.

## Project layout

```
src/askdocs/      chunking, corpus, lexical (BM25), embeddings, retrieval, llm, rag,
                  api (FastAPI), mcp_server, evals, cli, static/ (HTML/CSS/JS)
corpus/fastapi/   114 pages of the FastAPI docs (MIT), built by scripts/build_corpus.py
evals/            retrieval test questions
specs/            one spec per feature, with acceptance criteria
tests/, e2e/      pytest and Playwright suites
```

## How this was built

Spec-first, with Claude Code as the coding agent: each feature started as a spec in
[`specs/`](specs/) with acceptance criteria, was implemented by the agent, and then
went through tests, the retrieval eval and screenshot review. [`AGENTS.md`](AGENTS.md)
is the instruction file every coding agent gets. [`docs/agent-log.md`](docs/agent-log.md)
records what went wrong during the build and how each problem was caught.

## License

MIT. The bundled FastAPI documentation is © Sebastián Ramírez, MIT licensed
(`corpus/fastapi/LICENSE`).