Agentic RAG over Indian IT-sector annual reports

A question-answering agent over 30 annual reports (TCS, Infosys, Wipro, HCLTech, Tech Mahindra, LTIMindtree — FY2020 to FY2024; ~10,900 pages, 50,079 indexed chunks).

The distinction this project cares about: retrieval is a tool the agent decides to use, not a fixed pipeline stage. The agent classifies the question, decomposes it, chooses filters, judges whether what came back is sufficient, searches again if not, and refuses when the corpus cannot support an answer.

Q: "What was TCS attrition in FY2023?"                        [4 turns, ~55s]

  search_filings(query="attrition rate IT services workforce FY2023",
                 company="tcs", year=2023)          <- the agent chose this

  TCS's IT services attrition (LTM basis) for FY2023 was 20.1%. Industry-wide
  churn drove it to an all-time high in H1, then it trended down...
  (TCS FY2023, p.11; p.92; p.19)

  CITATIONS: [{tcs,2023,11}, {tcs,2023,92}, {tcs,2023,19}]

Q: "What was Apple's iPhone revenue in FY2023?"               [1 turn, ~14s]

  (no search issued)
  Apple is not an Indian IT-services company, and the available corpus only
  covers TCS, Infosys, Wipro, HCLTech, Tech Mahindra and LTIMindtree...

The second case matters as much as the first: the agent decided not to retrieve and refused, rather than answering from world knowledge.

Architecture

30 PDFs ──> parse ──> chunk ──> embed ──> pgvector ─┐
        (pdfplumber   (~1200    (Vertex             │
         + cid         chars,    gemini-            ├──> search_filings
         recovery)     tables    embedding-001,     │    (MCP tool)
                       kept      1536-dim)          │         │
                       whole)                       │         │
                            └──> BM25 (rank_bm25) ──┘         │
                                                              v
                                              dense + sparse -> RRF (k=60)
                                                     -> cross-encoder rerank
                                                              │
                                                              v
                              claude -p  (agent loop: classify -> plan ->
                              retrieve -> evaluate -> answer -> self-check)
                                                              │
                                                              v
                                            answer + citations + retrieval trace

Dimension note: Gemini returns 3072 dims, but pgvector's HNSW/IVFFlat indexes cap at 2000, so we truncate via MRL to 1536. Truncated Gemini vectors come back unnormalised (measured ~0.70), and cosine search assumes unit norm — so embed.py L2-normalises explicitly. Skipping that silently corrupts every score.

Quickstart

Full command reference: run.txt.

# prerequisites: Python 3.11, Postgres 17 + pgvector, gcloud ADC
brew services start postgresql@17
gcloud auth application-default login

.venv/bin/pytest -q -m "not live"                          # 58 tests

PYTHONPATH=src .venv/bin/python -m ragfilings.index.build  # resumable

PYTHONPATH=src .venv/bin/python -c "
from ragfilings.agent.run import run_agent
r = run_agent('What was TCS attrition in FY2023?')
print(r.answer); print(r.citations); print(r.trace)"

Evaluation

Questions live in data/eval/questions.yaml — 31 across five categories:

Beyond answer quality, the harness measures agent decisions:

• retrieval-decision accuracy — did it retrieve when it should have, and abstain when it shouldn't? Derived from actual tool calls, not citation count: an agent can retrieve and then correctly refuse, citing nothing.

• refusal accuracy — on out-of-corpus questions, did it decline?

• n_searches — how many retrievals per question type.

Status: questions are drafted with # src: page pointers but are not yet human-verified, and the eval has not been run. Numbers are deliberately absent rather than estimated.

Engineering notes

Three bugs worth recording, because each was silent — nothing crashed:

1. Font-encoding corruption — 17% of the index was unreadable. Several reports embed subset fonts with no ToUnicode CMap, so pdfplumber emitted raw glyph ids: (cid:49)(cid:82)(cid:87)(cid:72)(cid:86). HCLTech was 59% garbage. The glyph id turned out to be ASCII minus 29 — confirmed by decoding the corpus and matching the result against English letter frequency (cid:3→ space, cid:72→e, cid:68→a, cid:87→t). The decoder maps them back, expands ligatures (191→fi, 192→fl), and drops a second symbol font. It runs per token, not per page, because pages mix broken and correct spans — a blanket shift corrupts the text that was already fine. Result: 57,308 → 50,079 chunks, 0 corrupted; the count fell while readable content rose, since each (cid:NN) was 9 characters standing in for one.

2. The eval was measuring the wrong things. did_retrieve was inferred from len(citations) > 0, so retrieve-then-refuse scored as "never retrieved". Refusal detection matched the literal string "i don't know", so a correct live refusal — "Apple is not one of the companies covered..." — scored as a failure. Both bugs understated the agent.

3. Tests were pointed at the production database. The pgvector fixture runs TRUNCATE chunks. One pytest run would have erased a 20-minute index build. Tests now use a separate ragfilings_test database.

Also: the agent's instructions were originally concatenated into the -p user turn, which made the model treat them as a document to discuss and answer meta. They belong in --system-prompt. And the retrieval trace only exists under --output-format stream-json; the plain json format carries no message history.

Known limitations

• ~55s per question. The local 568M cross-encoder dominates; a smaller reranker or smaller candidate pool would cut it substantially.

• Tool-discovery overhead. The agent spends ~2 of 4 turns rediscovering its own MCP tool before calling it.

• Multi-column prose interleaves. Narrative pages read across columns. Tables use a separate extraction path and are unaffected, so financial figures are intact — the damage is to qualitative sections.

• Some HCLTech table cells extract right-to-left.

• Page-header boilerplate ranks high on some queries.

• Contextual headers are metadata-only. Per-chunk LLM headers over ~50k chunks were not viable; deferred to a batched v2 experiment.

Layout

src/ragfilings/
  ingest/    parse.py (+ decode_cid), chunk.py, context.py
  index/     embed.py, pgvector_store.py, bm25_store.py, build.py
  retrieval/ fusion.py (RRF), rerank.py, search.py
  mcp/       server.py          # exposes search_filings over MCP
  agent/     run.py, system_prompt.md
  eval/      questions.py, metrics.py, runner.py, baseline.py
  ui/        app.py             # Reflex
docs/STATUS.md                  # resume guide, decisions, gotchas
run.txt                         # every command

Corpus PDFs and the BM25 artifact are git-ignored; rebuild with ragfilings.index.build.