AskDocs

Ask a set of docs a question and get an answer that cites the exact passages it came from, so you can check it. Ships with the FastAPI documentation, works with any folder of Markdown, text or PDF files, and is also an MCP server, so Claude Code or Cursor can search the same docs.

What it does

• Hybrid retrieval: BM25 keyword search plus Gemini embeddings, combined with Reciprocal Rank Fusion. Falls back to keyword-only if no embedding key is set or the API fails.

• Grounded answers: the model may only use the numbered sources and must cite them as [n]. Each citation links to the passage and to the section on the docs website.

• Checks the model's work: citations to sources that don't exist are removed and flagged. Answers with no citations are marked as unverified.

• Knows when to say no: if retrieval finds nothing relevant, it refuses without calling the LLM, so quota isn't spent on a confident wrong answer.

• MCP server: search_docs, read_doc and (locally) ask_docs tools, over stdio or Streamable HTTP at /mcp.

• Gemini or Groq: choose with an environment variable. With no key at all it runs in search-only mode.

• Ready for a public demo: a per-IP rate limit, a daily cap on LLM calls, and a cache for repeated questions.

Related MCP server: Document Search MCP Server

How it works

docs/*.md ──▶ chunk by heading ──▶ BM25 index ─────────┐
              (code blocks kept     Gemini embeddings ─┤─▶ RRF ─▶ top 5 ─▶ in scope? ──no──▶ "not in these docs"
               whole, anchors →     (cached on disk)   │                     │yes
               citation URLs)                          │                     ▼
question ──────────────────────────────────────────────┘     prompt with [1]..[5] ─▶ Gemini / Groq
                                                                                       │
                              answer + verified citations + source cards ◀── check citations

1. Ingest (chunking.py): split Markdown on headings (levels 1–4). Code blocks are never split. Each chunk keeps its title, heading path and a URL to the exact section.

2. Retrieve (retrieval.py): BM25 over title + heading + text, with code identifiers indexed whole and split (response_model → response, model). When a Gemini key is set, cosine similarity over embeddings is fused in with RRF (k = 60).

3. Decide (retrieval.py): a question is out of scope when the top passages cover less than 45% of its meaning, weighting each word by rarity (IDF), and (in hybrid mode) the best embedding similarity is also low.

4. Answer (rag.py): a prompt with numbered sources. Citations are parsed (ignoring code like items[0]), checked against the sources, and returned with the answer.

Retrieval quality

evals/questions.jsonl has 35 questions with the document that should answer each (25 use the docs' own words, 10 are paraphrased), plus 6 off-topic questions that should be refused. askdocs eval runs it, and CI fails if it regresses.

Keyword mode (no API key):

Keyword search is strong when the question uses the docs' words, and weaker on paraphrases ("run code after the response is sent" vs "background tasks"). That gap is what the embedding half of hybrid search is for. The refusal threshold favours never refusing a real question; docs/agent-log.md explains the trade-off.

Run it

Requires Python 3.11+.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
cp .env.example .env               # add GEMINI_API_KEY or GROQ_API_KEY (both have free tiers)
askdocs doctor                     # checks the key and that the model names exist
askdocs serve                      # http://127.0.0.1:8000

Other commands:

askdocs ask "How do I add CORS middleware?"
askdocs search "upload files"      # retrieval only, no LLM
askdocs eval -v                    # retrieval metrics + misses

Use it from Claude Code or Cursor (MCP)

claude mcp add askdocs -- askdocs mcp                              # local, stdio
claude mcp add --transport http askdocs https://YOUR-HOST/mcp     # deployed app

Then ask your agent something like "using askdocs, how do I close a DB session after each request?". It will call search_docs, then read_doc on the best page. The public HTTP endpoint only exposes search and read: the calling agent is already an LLM, and a public server shouldn't spend its owner's quota.

Your own docs

Point ASKDOCS_CORPUS_DIR at any folder of .md, .txt or .pdf files. Add a corpus.json with {"name": "...", "base_url": "https://..."} to get citation links. The bundled corpus is rebuilt from the FastAPI repo with scripts/build_corpus.py.

Tests

pytest          # 66 tests: chunking, BM25, RRF, prompts, citations, providers, API, MCP, eval gate
pytest e2e      # 7 Playwright browser tests against the running app

Tests never call a real model. Providers are tested against httpx.MockTransport, and the app uses a fake LLM, so the suite runs offline with no keys. The browser tests include one that feeds the UI a model answer containing <img onerror> and <script>, and checks that nothing executes.

Project layout

src/askdocs/      chunking, corpus, lexical (BM25), embeddings, retrieval, llm, rag,
                  api (FastAPI), mcp_server, evals, cli, static/ (HTML/CSS/JS)
corpus/fastapi/   114 pages of the FastAPI docs (MIT), built by scripts/build_corpus.py
evals/            retrieval test questions
specs/            one spec per feature, with acceptance criteria
tests/, e2e/      pytest and Playwright suites

How this was built

Spec-first, with Claude Code as the coding agent: each feature started as a spec in specs/ with acceptance criteria, was implemented by the agent, and then went through tests, the retrieval eval and screenshot review. AGENTS.md is the instruction file every coding agent gets. docs/agent-log.md records what went wrong during the build and how each problem was caught.

License

MIT. The bundled FastAPI documentation is © Sebastián Ramírez, MIT licensed (corpus/fastapi/LICENSE).