# Agentic RAG over Indian IT-sector annual reports

A question-answering agent over **30 annual reports** (TCS, Infosys, Wipro,
HCLTech, Tech Mahindra, LTIMindtree — FY2020 to FY2024; ~10,900 pages, 50,079
indexed chunks).

The distinction this project cares about: retrieval is **a tool the agent
decides to use**, not a fixed pipeline stage. The agent classifies the question,
decomposes it, chooses filters, judges whether what came back is sufficient,
searches again if not, and refuses when the corpus cannot support an answer.

```
Q: "What was TCS attrition in FY2023?"                        [4 turns, ~55s]

  search_filings(query="attrition rate IT services workforce FY2023",
                 company="tcs", year=2023)          <- the agent chose this

  TCS's IT services attrition (LTM basis) for FY2023 was 20.1%. Industry-wide
  churn drove it to an all-time high in H1, then it trended down...
  (TCS FY2023, p.11; p.92; p.19)

  CITATIONS: [{tcs,2023,11}, {tcs,2023,92}, {tcs,2023,19}]

Q: "What was Apple's iPhone revenue in FY2023?"               [1 turn, ~14s]

  (no search issued)
  Apple is not an Indian IT-services company, and the available corpus only
  covers TCS, Infosys, Wipro, HCLTech, Tech Mahindra and LTIMindtree...
```

The second case matters as much as the first: the agent decided **not to
retrieve** and refused, rather than answering from world knowledge.

## Architecture

```
30 PDFs ──> parse ──> chunk ──> embed ──> pgvector ─┐
        (pdfplumber   (~1200    (Vertex             │
         + cid         chars,    gemini-            ├──> search_filings
         recovery)     tables    embedding-001,     │    (MCP tool)
                       kept      1536-dim)          │         │
                       whole)                       │         │
                            └──> BM25 (rank_bm25) ──┘         │
                                                              v
                                              dense + sparse -> RRF (k=60)
                                                     -> cross-encoder rerank
                                                              │
                                                              v
                              claude -p  (agent loop: classify -> plan ->
                              retrieve -> evaluate -> answer -> self-check)
                                                              │
                                                              v
                                            answer + citations + retrieval trace
```

| Layer | Choice | Why |
|---|---|---|
| Agent | Claude Code `claude -p` + MCP | Runs on a Claude subscription, no API key; MCP makes the retriever a real tool |
| Embeddings | Vertex `gemini-embedding-001`, 1536-dim | Local BGE-M3 took 140 min for the corpus and thrashed swap; Vertex is ~5 min |
| Vector store | Postgres 17 + pgvector | Metadata filters (`company`, `year`) live next to the vectors |
| Sparse | `rank_bm25` + RRF | Annual reports are full of exact tokens (line items, years) that dense retrieval blurs |
| Rerank | `bge-reranker-v2-m3` cross-encoder | Precision at low k |
| UI | Reflex | Python end-to-end |

**Dimension note:** Gemini returns 3072 dims, but pgvector's HNSW/IVFFlat indexes
cap at 2000, so we truncate via MRL to 1536. Truncated Gemini vectors come back
**unnormalised** (measured ~0.70), and cosine search assumes unit norm — so
`embed.py` L2-normalises explicitly. Skipping that silently corrupts every score.

## Quickstart

Full command reference: **[run.txt](run.txt)**.

```bash
# prerequisites: Python 3.11, Postgres 17 + pgvector, gcloud ADC
brew services start postgresql@17
gcloud auth application-default login

.venv/bin/pytest -q -m "not live"                          # 58 tests

PYTHONPATH=src .venv/bin/python -m ragfilings.index.build  # resumable

PYTHONPATH=src .venv/bin/python -c "
from ragfilings.agent.run import run_agent
r = run_agent('What was TCS attrition in FY2023?')
print(r.answer); print(r.citations); print(r.trace)"
```

## Evaluation

Questions live in [data/eval/questions.yaml](data/eval/questions.yaml) — 31
across five categories:

| Category | n | Tests |
|---|---|---|
| `pointed_fact` | 15 | single figure, single company-year |
| `comparative` | 5 | multiple retrievals across companies |
| `trend` | 5 | multiple years of one company |
| `general_knowledge` | 2 | must answer **without** retrieving |
| `out_of_corpus` | 4 | must **refuse** |

Beyond answer quality, the harness measures agent decisions:

- **retrieval-decision accuracy** — did it retrieve when it should have, and
  abstain when it shouldn't? Derived from actual tool calls, not citation count:
  an agent can retrieve and then correctly refuse, citing nothing.
- **refusal accuracy** — on out-of-corpus questions, did it decline?
- **n_searches** — how many retrievals per question type.

> **Status:** questions are drafted with `# src:` page pointers but are **not yet
> human-verified**, and the eval has not been run. Numbers are deliberately absent
> rather than estimated.

## Engineering notes

Three bugs worth recording, because each was silent — nothing crashed:

**1. Font-encoding corruption — 17% of the index was unreadable.**
Several reports embed subset fonts with no ToUnicode CMap, so pdfplumber emitted
raw glyph ids: `(cid:49)(cid:82)(cid:87)(cid:72)(cid:86)`. HCLTech was 59%
garbage. The glyph id turned out to be **ASCII minus 29** — confirmed by decoding
the corpus and matching the result against English letter frequency (`cid:3`→
space, `cid:72`→`e`, `cid:68`→`a`, `cid:87`→`t`). The decoder maps them back,
expands ligatures (`191`→`fi`, `192`→`fl`), and drops a second symbol font.
It runs **per token, not per page**, because pages mix broken and correct spans —
a blanket shift corrupts the text that was already fine.
Result: 57,308 → **50,079 chunks, 0 corrupted**; the count fell while readable
content rose, since each `(cid:NN)` was 9 characters standing in for one.

**2. The eval was measuring the wrong things.**
`did_retrieve` was inferred from `len(citations) > 0`, so retrieve-then-refuse
scored as "never retrieved". Refusal detection matched the literal string
`"i don't know"`, so a correct live refusal — *"Apple is not one of the companies
covered..."* — scored as a failure. Both bugs **understated** the agent.

**3. Tests were pointed at the production database.**
The pgvector fixture runs `TRUNCATE chunks`. One `pytest` run would have erased
a 20-minute index build. Tests now use a separate `ragfilings_test` database.

Also: the agent's instructions were originally concatenated into the `-p` user
turn, which made the model treat them as a document to discuss and answer meta.
They belong in `--system-prompt`. And the retrieval trace only exists under
`--output-format stream-json`; the plain `json` format carries no message history.

## Known limitations

- **~55s per question.** The local 568M cross-encoder dominates; a smaller
  reranker or smaller candidate pool would cut it substantially.
- **Tool-discovery overhead.** The agent spends ~2 of 4 turns rediscovering its
  own MCP tool before calling it.
- **Multi-column prose interleaves.** Narrative pages read across columns.
  Tables use a separate extraction path and are unaffected, so financial figures
  are intact — the damage is to qualitative sections.
- **Some HCLTech table cells extract right-to-left.**
- **Page-header boilerplate** ranks high on some queries.
- **Contextual headers are metadata-only.** Per-chunk LLM headers over ~50k
  chunks were not viable; deferred to a batched v2 experiment.

## Layout

```
src/ragfilings/
  ingest/    parse.py (+ decode_cid), chunk.py, context.py
  index/     embed.py, pgvector_store.py, bm25_store.py, build.py
  retrieval/ fusion.py (RRF), rerank.py, search.py
  mcp/       server.py          # exposes search_filings over MCP
  agent/     run.py, system_prompt.md
  eval/      questions.py, metrics.py, runner.py, baseline.py
  ui/        app.py             # Reflex
docs/STATUS.md                  # resume guide, decisions, gotchas
run.txt                         # every command
```

Corpus PDFs and the BM25 artifact are git-ignored; rebuild with
`ragfilings.index.build`.