Financial Filings RAG

Retrieval-augmented generation over SEC filings (10-K, 10-Q, 8-K, earnings releases), evaluated on FinanceBench – 150 analyst questions written by financial professionals about 84 real filings of 32 large publicly traded companies.

The project covers the whole stack an LLM application needs before the model writes a word: PDF ingestion, hybrid search, query routing, a learned reranker, an MCP server that exposes the search to Claude or any MCP client, and a tool-calling agent that cites the pages it read. Every retrieval number below comes from a reproducible evaluation with confidence intervals.

Key results

Retrieval is measured per question: did a page that contains the gold evidence come back in the top k, out of 12,013 pages?

• Plain search fails on financial documents. BM25 over the whole corpus finds the right page in the top 5 for only 13% of questions: the same words ("revenue", "operating income", "2022") appear in dozens of filings.

• Knowing which filing to look in matters most. Routing the question to the right company and weighting the filing that matches the asked period lifts Hit@5 from 12% to 33%; each step is significant in a paired bootstrap (p < 0.001).

• A learned reranker more than triples top-1 accuracy (13% → 43%). It is a LightGBM LambdaRank model evaluated with 5-fold cross-validation grouped by company, so it is always tested on companies it has never seen.

• Metric questions benefit most: for questions like "What is 3M's FY2018 capex?" Hit@5 goes from 2% (BM25) to 78%.

Reranker metrics are the mean of 5 cross-validation runs with different fold assignments (Hit@1 43.2% ± 1.7, Hit@5 59.3% ± 0.5). The chart above shows a single run.

Related MCP server: call-copilot

Architecture

flowchart LR
    Q[Analyst question] --> R[Query router<br/>company · fiscal period · filing type]
    R --> B[BM25<br/>+ financial synonym expansion]
    R --> D[Dense retrieval<br/>WordLlama embeddings]
    B --> F[Reciprocal Rank Fusion<br/>+ period / filing-type prior]
    D --> F
    F -->|top 50 pages| RR[LambdaRank reranker<br/>17 features]
    RR --> T[search_filings · read_page · list_filings]
    T --> MCP[MCP server<br/>Claude Desktop / Claude Code / any client]
    T --> AG[Claude tool-calling agent<br/>answers with page citations]

Results in detail

How many pages does the model need to read?

With the reranker, an LLM that reads the top 5 pages has the evidence in front of it for 59% of questions, versus 33% without it and 13% with plain BM25.

Which components matter? (leave-one-out)

Starting from the routed pipeline (before the reranker), each component is removed in turn:

• Routing and the period prior carry most of the gain.

• Query expansion only helps once the search is routed. On the whole corpus it changes nothing (the step in the chart above is flat), but inside the right company's filings it adds 6 points of Hit@10.

• The dense retriever does not improve the top 5, where BM25 alone is as good. It does widen the candidate pool for the reranker (79.3% vs 77.3% at depth 50), which is why the hybrid is kept. Static word embeddings are a weak semantic model, and this is the first component to upgrade (see Limitations).

By question type

FinanceBench has three question types of 50 questions each. Metrics-generated questions ask for figures from the financial statements; the reranker learns to recognise statement pages and finds them 78% of the time. Domain-relevant questions (e.g. "What drove operating margin change?") point at narrative sections and remain the hardest.

What the reranker learned

The most useful signal is how table-like a page is (numeric density), followed by the routed hybrid rank and query-term coverage. The model is small and interpretable: 17 features, no neural network.

Guarding against an optimistic evaluation

• Leakage check: 32 questions share a gold page with another question. If folds were split by question only, page-identifying features could leak from training to test. All reported reranker numbers therefore use folds grouped by company. For reference, question-grouped folds give a Hit@5 of 58.3% – almost identical, so the gain is not an artefact of leakage.

• No tuning on the test set: the reranker's hyper-parameters, the RRF constant (60) and the prior weight (0.5) were fixed before evaluation. The prior weight is not a knife-edge choice: Hit@5 is 29.3% at 0.25 and stays at 32.7% from 0.5 to 4.

• Uncertainty is reported: 95% bootstrap CIs and paired bootstrap tests are in results/metrics.json.

• The ceiling is explicit: the reranker can only reorder the 50 candidates it receives, which contain a gold page for 79.3% of questions.

Using it with Claude (MCP)

Install the package, build the index (see Reproduce), then add the server to Claude Desktop's config file (Windows: %APPDATA%\Claude\claude_desktop_config.json, macOS: ~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "financial-filings": {
      "command": "python",
      "args": ["-m", "finrag.mcp_server"]
    }
  }
}

If you use a virtual environment, put the full path to its Python as command (e.g. C:\\path\\to\\financial-filings-rag\\.venv\\Scripts\\python.exe).

Claude can then call search_filings, read_page and list_filings directly. The server was verified end-to-end over stdio against the full index. Example call:

search_filings("By how much did PepsiCo increase its unsecured five year revolving credit agreement on May 26, 2023?")
→ PEPSICO_2023_8K_dated-2023-05-30 p.2   (the 8-K announcing the new credit agreement)

Command line

finrag search "What is the FY2018 capital expenditure amount for 3M? Use the cash flow statement."
# 1. 3M_2018_10K p.60  ← Consolidated Statement of Cash Flows (the gold page)

export ANTHROPIC_API_KEY=...          # only for the agent; Windows PowerShell: $env:ANTHROPIC_API_KEY="..."
finrag ask "What was 3M's FY2018 capital expenditure?"

The agent prints its answer, the pages it read, and any citation that points to a page it never opened.

What is verified – and what is not

Limitations

• Embeddings: Hugging Face models were not downloadable in the build environment, so dense retrieval uses WordLlama, a static embedding model that ships inside its pip package. A modern sentence-embedding model and a neural cross-encoder reranker are the obvious next step; the evaluation harness is ready for them.

• Small test set: 150 questions. The confidence interval for Hit@5 of the final system is 51–67%.

• Learned reranker, small data: it is trained on 150 questions. Grouped cross-validation shows it generalises to unseen companies, but not that it would transfer to other document types.

• Router coverage: company aliases cover the 32 companies in the benchmark. A production system would resolve company names against a full ticker / CIK database.

• Scanned pages: 65 of 12,013 pages have no extractable text (scanned images); OCR is not applied.

Reproduce

Python 3.10 or newer.

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

python scripts/build_corpus.py   # downloads FinanceBench + 84 PDFs (~160 MB), extracts 12k pages  (~1 min)
python scripts/build_index.py    # BM25 + embeddings over 40k chunks                                 (~1 min)
python scripts/evaluate.py       # all experiments, figures, tables, trains models/reranker          (~1-2 min)
pytest -q                        # 26 tests, no data or API key needed

Everything runs on a laptop CPU; no GPU or paid API is needed for retrieval and evaluation. A full rebuild from an empty folder took 3 minutes and reproduced every number in results/metrics.json exactly.

Project structure

src/finrag/
  data.py         download, PDF -> pages, evidence alignment, chunking
  text.py         tokenizer, stemming, financial synonym expansion
  bm25.py         sparse-matrix BM25
  embeddings.py   WordLlama dense index
  router.py       company / period / filing-type detection and document prior
  index.py        hybrid search pipeline (RRF, routing, prior), page-level aggregation
  rerank.py       LambdaRank features, grouped cross-validation
  search.py       production search API (pipeline + reranker)
  tools.py        tool implementations and JSON schemas shared by MCP and the agent
  mcp_server.py   MCP server (stdio)
  agent.py        Claude tool-calling agent with citation checks
  metrics.py      Hit@k, MRR, nDCG, bootstrap CIs, paired bootstrap tests
scripts/          build_corpus.py, build_index.py, evaluate.py
results/          metrics.json, tables/, figures/
models/           trained reranker used by the search API
tests/            retrieval, router, metrics, MCP server and agent-loop tests

References

• Islam, P., Kannappan, A., Kiela, D., Qian, R., Scherrer, N., Vidgen, B. (2023). FinanceBench: A New Benchmark for Financial Question Answering. arXiv:2311.11944. Data: patronus-ai/financebench.

• Robertson, S. & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in IR.

• Cormack, G., Clarke, C. & Büttcher, S. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods. SIGIR.

• Burges, C. (2010). From RankNet to LambdaRank to LambdaMART: An Overview. Microsoft Research.

• Ke, G. et al. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. NeurIPS.

• Miller, D. L. (2024). WordLlama. github.com/dleemiller/WordLlama.

• Model Context Protocol. modelcontextprotocol.io.

FinanceBench is not redistributed here; the build script downloads it from the authors' repository. Check the dataset's terms before any commercial use.

Author

Marcin Ossoliński – Data Engineering student, Rzeszów University of Technology · Automation & AI Intern, Asseco Poland · GitHub