# Ask your documents: an MCP server with citations, and the numbers to prove it

An MCP server that lets Claude (or ChatGPT, or any MCP client) answer questions from a set of long,
messy documents, and cite the exact passage every answer comes from. The documents here are eight
SEC annual reports (10-K) from Costco, Nike, Delta Air Lines and Netflix, about 360,000 words of
HTML with tables, footnotes and no real headings.

Most RAG demos stop at "it answered my question". This one also ships the part that decides whether
it works on real questions: **a retrieval eval** that compares chunking strategies, embedding models
and search modes against a fixed question set, so changes are made on evidence, not on vibes.

## Results

37 real-world questions, each with a known answer in the filings. "Right passage in top 5" is what
the model actually gets to read; "right passage first" is how often the best hit is on top.

| Setup | Right passage in top 5 | Right passage first | Latency (laptop CPU) |
|---|---:|---:|---:|
| Tutorial default: 300-word windows + all-MiniLM-L6-v2 | 70% | 30% | 9 ms |
| Keyword search only (BM25) | 54% | 41% | 4 ms |
| Structure-aware chunks + bge-base, vector search **(default)** | **81%** | 46% | 41 ms |
| Structure-aware chunks + bge-base, hybrid | 68% | **59%** | 46 ms |
| … plus a cross-encoder reranker | 81% | 54% | 3,839 ms |

Full grid of 12 configurations: [`eval/results.md`](eval/results.md).

What the numbers said, and what a gut call would have got wrong:

- **Chunking mattered more than the model.** Swapping to the bigger embedding model alone left
  top-5 accuracy at 70%; cutting along the filing's sections with that same model took it to 81%. The first version of structure-aware
  chunking actually *lost* to plain windows. It produced 800+ fragments under 40 words (signature
  lines, footnotes), and short chunks win searches on length alone. Merging fragments and keeping
  sub-headings in the text fixed it.
- **The small default model truncates.** all-MiniLM-L6-v2 reads at most 256 tokens, so the end of a
  300-word chunk is invisible to it. bge-base reads 512.
- **Hybrid search is a trade, not an upgrade.** Adding keyword search puts the right passage first
  more often (46% → 59%) but pushes it out of the top five more often too (81% → 68%). Since the
  model reads all five, the server defaults to vector search; set `DOCS_MODE=hybrid` if your users
  ask for exact figures and codes.
- **The reranker didn't pay for itself.** A popular off-the-shelf cross-encoder (ms-marco MiniLM,
  trained on web search) recovered hybrid's top-5 losses, but only back to the 81% plain vector
  search already gets, at about 90× the latency on CPU. On fixed-window chunks it made results
  worse. On a GPU, or with a reranker tuned for financial text, that may change; the eval is how
  you'd know.
- **Graded on the fact, not the paragraph.** A first pass graded on exact sentences and marked
  correct answers wrong when a filing stated the same number in a table and in the text. The eval
  now checks for the answer itself (e.g. "103,000" employees).

The misses are the hardest paraphrases, e.g. "What was Nike's headcount?" when the filing says
"employees", or "How exposed is Costco to a single US state?" when it says "California operations
comprised 26%". In a chat the model can rephrase and search again; the eval scores the first
try only.

## What Claude can do with it

| Tool | What it does |
|---|---|
| `list_filings` | Lists the filings available, with fiscal year and SEC link |
| `search_filings(query, ticker?, fiscal_year?, limit?)` | Searches one company and year, or all filings; returns passages with section and SEC link |
| `get_passage(chunk_id, context?)` | Reads a passage in full with the passages around it, to check context before answering |

Every result carries the company, fiscal year, section path (`Item 7 > Membership Fees`) and the
SEC URL, and the server instructions tell the model to cite them and say so when the passages
don't contain the answer.

## Run it in five minutes

Requires Python 3.11+. No API keys, no GPU: embeddings run locally on CPU.

```bash
git clone https://github.com/mariacparra/mcp-docs-demo
cd mcp-docs-demo
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e .
cp .env.example .env                                 # put your email in SEC_USER_AGENT

python -m docsdemo.fetch_sec      # downloads the 8 filings from SEC EDGAR (~17 MB)
python -m docsdemo.ingest         # parses, chunks and embeds them (a few minutes on a laptop)
python eval/run_eval.py           # reproduces the table above
python tests/smoke_test.py        # starts the server and calls every tool
```

### Connect it to Claude Desktop

Add this to `claude_desktop_config.json` (Settings > Developer > Edit Config), with your own paths:

```json
{
  "mcpServers": {
    "sec-filings": {
      "command": "C:\\path\\to\\mcp-docs-demo\\.venv\\Scripts\\python.exe",
      "args": ["-m", "docsdemo.server"]
    }
  }
}
```

Or in Claude Code: `claude mcp add sec-filings -- /path/to/.venv/bin/python -m docsdemo.server`.

For a remote client, `python -m docsdemo.server --http` serves streamable HTTP at `http://localhost:8000/mcp`.

## How it works

```
SEC EDGAR ──> fetch_sec ──> parse ──> chunking ──> embeddings ──┬──> local index (numpy + BM25)
                                                                └──> MongoDB Atlas ($vectorSearch + Atlas Search)
                                                                          │
                       Claude ──MCP──> server: list_filings / search_filings / get_passage
```

**Parsing.** 10-K filings are inline XBRL: thousands of nested `div`s and styled `span`s, hidden
XBRL metadata, and headings that are just bold text. The parser strips the hidden blocks, finds
"Item 7" style section breaks, treats short bold lines as headings, and turns tables into
`label | 2026 | 2025` rows, re-joining the `$`, number and `%` cells that XBRL splits apart.

**Two chunking strategies**, so the eval has something to compare:

- `fixed`: 300-word sliding windows with 50 words of overlap. The common default.
- `structure`: paragraphs packed within one section up to 350 words, tables kept whole, and every
  chunk prefixed with a context line like `Costco 10-K FY2026 > Item 7 > Membership Fees`, so a
  chunk that says "increased 11%" still knows what increased.

**Two embedding models**, both free and local through [fastembed](https://github.com/qdrant/fastembed):
`all-MiniLM-L6-v2` (the most common default) and `BAAI/bge-base-en-v1.5`.

**Four search modes**: BM25 keyword search, dense vector search, hybrid (reciprocal rank fusion
of both), and hybrid followed by a cross-encoder reranker over the top 20. Dense search understands
"how many people work at Costco"; keyword search nails "$7.4 billion" and "SkyMiles". Which mix
wins is an empirical question, answered in the results above.

## The eval

[`eval/questions.jsonl`](eval/questions.jsonl) holds 37 questions written the way a person would
ask them ("Does Delta produce any of its own jet fuel?"), each with a snippet of the passage that
answers it ("operate the Trainer refinery"). A question counts as answered at rank *r* when the
*r*-th result is from the right company and contains the snippet. Matching on text rather than
chunk ids means the same questions grade any chunking strategy.

- **Hit@1**: the first result is the right passage.
- **Hit@5**: the right passage is somewhere in the top five (what the model actually sees).
- **MRR**: mean reciprocal rank; 1.0 means always first.

Full results: [`eval/results.md`](eval/results.md).

## MongoDB Atlas (optional)

The local index is enough to run everything. To serve from Atlas instead (the free M0 tier works):

1. Create a free cluster at [mongodb.com/atlas](https://www.mongodb.com/atlas), add a database user
   and allow your IP.
2. Put the connection string in `.env` as `MONGODB_URI`.
3. `python -m docsdemo.ingest --mongo` uploads the chunks and creates a vector index (with
   `ticker` and `fiscal_year` as filters) and a text index.
4. `DOCS_BACKEND=mongo python -m docsdemo.server`.

Search then runs `$vectorSearch` and `$search` in Atlas and fuses them with the same RRF as locally.

## Adapting this to your documents

The SEC part is two files (`fetch_sec.py`, `parse.py`). The rest (chunking, hybrid search,
citations, the eval, the MCP server) works the same for contracts, policies, manuals, research
or support tickets. The step that matters most is the one people skip: write 30 to 50 real
questions with known answers *before* tuning anything, then let the numbers pick the configuration.

## Layout

```
src/docsdemo/
  fetch_sec.py     download filings from SEC EDGAR
  parse.py         10-K HTML -> blocks with section paths
  chunking.py      fixed and structure-aware chunking
  retrieval.py     embeddings, BM25, hybrid search (local)
  mongo_store.py   the same search on MongoDB Atlas
  ingest.py        build every index
  server.py        the MCP server
eval/              questions, runner, results
tests/             end-to-end smoke test over stdio
```

## License

MIT. The filings are public documents from [SEC EDGAR](https://www.sec.gov/edgar).