Ask your documents: an MCP server with citations, and the numbers to prove it

An MCP server that lets Claude (or ChatGPT, or any MCP client) answer questions from a set of long, messy documents, and cite the exact passage every answer comes from. The documents here are eight SEC annual reports (10-K) from Costco, Nike, Delta Air Lines and Netflix, about 360,000 words of HTML with tables, footnotes and no real headings.

Most RAG demos stop at "it answered my question". This one also ships the part that decides whether it works on real questions: a retrieval eval that compares chunking strategies, embedding models and search modes against a fixed question set, so changes are made on evidence, not on vibes.

Results

37 real-world questions, each with a known answer in the filings. "Right passage in top 5" is what the model actually gets to read; "right passage first" is how often the best hit is on top.

Full grid of 12 configurations: eval/results.md.

What the numbers said, and what a gut call would have got wrong:

• Chunking mattered more than the model. Swapping to the bigger embedding model alone left top-5 accuracy at 70%; cutting along the filing's sections with that same model took it to 81%. The first version of structure-aware chunking actually lost to plain windows. It produced 800+ fragments under 40 words (signature lines, footnotes), and short chunks win searches on length alone. Merging fragments and keeping sub-headings in the text fixed it.

• The small default model truncates. all-MiniLM-L6-v2 reads at most 256 tokens, so the end of a 300-word chunk is invisible to it. bge-base reads 512.

• Hybrid search is a trade, not an upgrade. Adding keyword search puts the right passage first more often (46% → 59%) but pushes it out of the top five more often too (81% → 68%). Since the model reads all five, the server defaults to vector search; set DOCS_MODE=hybrid if your users ask for exact figures and codes.

• The reranker didn't pay for itself. A popular off-the-shelf cross-encoder (ms-marco MiniLM, trained on web search) recovered hybrid's top-5 losses, but only back to the 81% plain vector search already gets, at about 90× the latency on CPU. On fixed-window chunks it made results worse. On a GPU, or with a reranker tuned for financial text, that may change; the eval is how you'd know.

• Graded on the fact, not the paragraph. A first pass graded on exact sentences and marked correct answers wrong when a filing stated the same number in a table and in the text. The eval now checks for the answer itself (e.g. "103,000" employees).

The misses are the hardest paraphrases, e.g. "What was Nike's headcount?" when the filing says "employees", or "How exposed is Costco to a single US state?" when it says "California operations comprised 26%". In a chat the model can rephrase and search again; the eval scores the first try only.

Related MCP server: edgar-mcp

What Claude can do with it

Every result carries the company, fiscal year, section path (Item 7 > Membership Fees) and the SEC URL, and the server instructions tell the model to cite them and say so when the passages don't contain the answer.

Run it in five minutes

Requires Python 3.11+. No API keys, no GPU: embeddings run locally on CPU.

git clone https://github.com/mariacparra/mcp-docs-demo
cd mcp-docs-demo
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .
cp .env.example .env                                 # put your email in SEC_USER_AGENT

python -m docsdemo.fetch_sec      # downloads the 8 filings from SEC EDGAR (~17 MB)
python -m docsdemo.ingest         # parses, chunks and embeds them (a few minutes on a laptop)
python eval/run_eval.py           # reproduces the table above
python tests/smoke_test.py        # starts the server and calls every tool

Connect it to Claude Desktop

Add this to claude_desktop_config.json (Settings > Developer > Edit Config), with your own paths:

{
  "mcpServers": {
    "sec-filings": {
      "command": "C:\\path\\to\\mcp-docs-demo\\.venv\\Scripts\\python.exe",
      "args": ["-m", "docsdemo.server"]
    }
  }
}

Or in Claude Code: claude mcp add sec-filings -- /path/to/.venv/bin/python -m docsdemo.server.

For a remote client, python -m docsdemo.server --http serves streamable HTTP at http://localhost:8000/mcp.

How it works

SEC EDGAR ──> fetch_sec ──> parse ──> chunking ──> embeddings ──┬──> local index (numpy + BM25)
                                                                └──> MongoDB Atlas ($vectorSearch + Atlas Search)
                                                                          │
                       Claude ──MCP──> server: list_filings / search_filings / get_passage

Parsing. 10-K filings are inline XBRL: thousands of nested divs and styled spans, hidden XBRL metadata, and headings that are just bold text. The parser strips the hidden blocks, finds "Item 7" style section breaks, treats short bold lines as headings, and turns tables into label | 2026 | 2025 rows, re-joining the $, number and % cells that XBRL splits apart.

Two chunking strategies, so the eval has something to compare:

• fixed: 300-word sliding windows with 50 words of overlap. The common default.

• structure: paragraphs packed within one section up to 350 words, tables kept whole, and every chunk prefixed with a context line like Costco 10-K FY2026 > Item 7 > Membership Fees, so a chunk that says "increased 11%" still knows what increased.

Two embedding models, both free and local through fastembed: all-MiniLM-L6-v2 (the most common default) and BAAI/bge-base-en-v1.5.

Four search modes: BM25 keyword search, dense vector search, hybrid (reciprocal rank fusion of both), and hybrid followed by a cross-encoder reranker over the top 20. Dense search understands "how many people work at Costco"; keyword search nails "$7.4 billion" and "SkyMiles". Which mix wins is an empirical question, answered in the results above.

The eval

eval/questions.jsonl holds 37 questions written the way a person would ask them ("Does Delta produce any of its own jet fuel?"), each with a snippet of the passage that answers it ("operate the Trainer refinery"). A question counts as answered at rank r when the r-th result is from the right company and contains the snippet. Matching on text rather than chunk ids means the same questions grade any chunking strategy.

• Hit@1: the first result is the right passage.

• Hit@5: the right passage is somewhere in the top five (what the model actually sees).

• MRR: mean reciprocal rank; 1.0 means always first.

Full results: eval/results.md.

MongoDB Atlas (optional)

The local index is enough to run everything. To serve from Atlas instead (the free M0 tier works):

1. Create a free cluster at mongodb.com/atlas, add a database user and allow your IP.

2. Put the connection string in .env as MONGODB_URI.

3. python -m docsdemo.ingest --mongo uploads the chunks and creates a vector index (with ticker and fiscal_year as filters) and a text index.

4. DOCS_BACKEND=mongo python -m docsdemo.server.

Search then runs $vectorSearch and $search in Atlas and fuses them with the same RRF as locally.

Adapting this to your documents

The SEC part is two files (fetch_sec.py, parse.py). The rest (chunking, hybrid search, citations, the eval, the MCP server) works the same for contracts, policies, manuals, research or support tickets. The step that matters most is the one people skip: write 30 to 50 real questions with known answers before tuning anything, then let the numbers pick the configuration.

Layout

src/docsdemo/
  fetch_sec.py     download filings from SEC EDGAR
  parse.py         10-K HTML -> blocks with section paths
  chunking.py      fixed and structure-aware chunking
  retrieval.py     embeddings, BM25, hybrid search (local)
  mongo_store.py   the same search on MongoDB Atlas
  ingest.py        build every index
  server.py        the MCP server
eval/              questions, runner, results
tests/             end-to-end smoke test over stdio

License

MIT. The filings are public documents from SEC EDGAR.