# MMWRAG

![CI](https://github.com/mikrominiw/scientific-rag-mcp/actions/workflows/ci.yml/badge.svg)
![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
![Python](https://img.shields.io/badge/python-3.12-blue.svg)

A bilingual (Russian/English) RAG system over scientific literature (textbooks/papers):
vision PDF parsing, **BGE-M3 hybrid (dense+sparse) retrieval** with a **cross-encoder
reranker**, an **MCP tool server**, a **LangGraph agent layer**, and an **HTTP serving
API** with async client-driven ingestion. Every non-trivial decision here — retrieval
pool composition, reranker choice, context budget, LLM provider, whether the agent
layer even helps — was made by measurement against a gold set, not intuition. See
**[DECISIONS.md](DECISIONS.md)** for the full log.

## Features

- **Vision PDF parsing** behind a swappable interface (cloud PaddleOCR-VL / local PP-StructureV3) — required because the text layer doesn't encode formula structure.
- **Structure-aware chunking** (~512-token packing over blocks, page spans kept for citations).
- **BGE-M3** dense + sparse embeddings; **Qdrant** hybrid search with server-side RRF.
- **Cross-encoder reranker** (`bge-reranker-v2-m3`) — measured `hit@5` 0.761 → 0.848.
- **Book-aware cross-lingual routing** — `search(book_id=...)` targets a specific book/language.
- **MCP server** (`search`, `list_books`) over streamable HTTP — no answer generation.
- **LangGraph agent** — multi-round, book-routed tool-calling search over MCP; beats single-shot answers on multi-hop questions.
- **HTTP serving API** (`src/serve/`) — SSE-streamed `/answer`/`/answer/agentic`, async `POST /ingest` + status polling, health/readiness/Prometheus.
- **Self-hosted observability** (Langfuse) — full span tree, a true no-op when disabled.
- **Two LLM providers** behind one interface (OpenRouter, GigaChat), retry/backoff always on.
- **Eval harness** — page-level `hit@k`/`MRR`/`recall@k`, plus an LLM-judge harness for end-to-end answer quality.

## Architecture

```
INDEXING   PDF ─parse─> Page[] ─chunk─> Chunk[] ─BGE-M3 (dense+sparse)─> Qdrant
QUERY      question ─HybridRetriever (RRF)─> top-N ─rerank─> top-k ─LLM─> Answer
MCP        client ─/mcp─> search(query, top_k, book_id) / list_books()
AGENT      question ─LangGraph tool-calling loop (book-routed)─> synthesize ─> Answer
SERVING    POST /answer, /answer/agentic (SSE) · POST /ingest, GET /ingest/{job_id}
```

Full package layout, swappable interfaces, and the agent/serving/observability design
are in **[ARCHITECTURE.md](ARCHITECTURE.md)**.

## Quickstart

```bash
# 1. dependencies (paddlepaddle-gpu is a manual prereq for the PARSING path only —
#    everything else, including embedding/reranking, runs on CPU too, just slower)
uv sync

# 2. vector database (+ optional embed-service/app containers — see docker-compose.yml)
docker compose up -d            # Qdrant on :6333

# 3. bring your own PDF and index it
#    parsing needs PADDLEOCR_TOKEN in .env (see .env.example);
#    pipeline: parse(pdf) -> chunk_pages(...) -> index_chunks(...)  (see notebooks/ for examples)

# 4. run the MCP server (for an LLM/agent client)...
uv run python -m src.mcp.server        # streamable-http on 127.0.0.1:8000
# ...or the HTTP serving API (for ordinary clients)
uv run python -m src.serve.app         # SERVE_API_KEY required, see .env.example
```

> The corpus is **not included** (copyright). Search/MCP/serving need Qdrant + the
> local models (BGE-M3, the reranker); CPU works (slower), GPU is faster. Parsing
> additionally needs a PaddleOCR-VL cloud token. A Docker image is also published —
> see [Docker image](#docker-image) below.

## Results

Six of the decisions in [DECISIONS.md](DECISIONS.md), each backed by a gold-set
measurement (own metric/gold set per panel — bars aren't comparable across panels):

![Six panels: reranker uplift hit@5 0.76→0.85; hybrid vs dense-only pool hit@10 0.92 vs 0.69; cross-encoder vs ColBERT MRR 0.86 vs 0.73; context budget faithfulness 0.96→0.99; agent vs single-shot relevance 0.62→1.00; OpenRouter vs GigaChat faithfulness 0.99 vs 0.82](assets/measured-decisions.png)

The single most interesting result — and the one with the most involved diagnosis —
is why cross-lingual retrieval was failing:

![RU→EN hit@10 across pool × scope: 0.00/0.06 cross-book, 0.71/0.71 target-filtered](assets/crosslingual-crowding.png)

### Technical deep dive: diagnosing RU→EN ≈ 0

A Russian→English cross-lingual query scored **`hit@10` ≈ 0** in a cross-book search —
by far the worst slice in the eval suite. There were three plausible causes: sparse
retrieval dragging in Russian noise, BGE-M3 having a hard ceiling on cross-script
matching, or the correct English chunk being crowded out by a same-language, same-
concept competitor. A single before/after number can't tell those apart, so the fix
was a **2×2 diagnostic**: pool (hybrid vs. dense-only) × search scope (whole index vs.
scoped to the target book), reranker held out entirely so it couldn't mask the result.

**What the four numbers show:** scoping the search to the target English book alone
lifts `hit@10` from **0.00 to 0.71** — ruling out an embedding ceiling (BGE-M3 clearly
*can* match Russian queries to English text). And dense-only vs. hybrid in the
cross-book condition (0.06 vs 0.00) is far too small a gap to explain a 0.71-point
swing — ruling out sparse drag as the dominant cause. What's left is **crowding**: in
a cross-book search, the same concept's Russian source (present in the much larger
Russian corpus) systematically outranks the correct English one.

That result killed a planned "script-aware fusion" fix (it targeted sparse
reweighting — the wrong cause) and replaced it with **book-aware routing**
(`search(book_id=...)`), which is why the MCP tool takes a `book_id` parameter at all
and why the agent layer routes sub-searches per book instead of always searching the
whole index. Full walkthrough, including why the reranker was deliberately excluded
from the diagnostic run, is in [DECISIONS.md](DECISIONS.md#diagnosing-the-ruen-gap).

## Demo

A real session against the MCP server (`notebooks/mcp_smoke.py`, output trimmed to
metadata):

```
tools: ['search', 'list_books']

list_books:
  {'book_id': 'zorich_v1', 'title': 'Zorich — Mathematical Analysis I', 'language': 'ru', 'chunks': 1472}
  {'book_id': 'zorich_v2', 'title': 'Zorich — Mathematical Analysis II', 'language': 'ru', 'chunks': 2526}
  {'book_id': 'lebl', 'title': 'Lebl — Basic Analysis I', 'language': 'en', 'chunks': 722}

search RU (all books), top 3:
  zorich_v1 159 2.125
  zorich_v1 158–159 0.297
  zorich_v2 517 -0.357

search RU routed to lebl (cross-lingual), top 3:
  lebl 135–136 0.123
  lebl 167 -0.047
  lebl 208 -0.141
```

The last call shows **book-aware cross-lingual routing**: a Russian query with
`book_id="lebl"` returns the English source (Lebl, p.135–136) that a plain cross-book
search buries behind the Russian equivalent — the exact failure mode diagnosed above.

## Docker image

A `Dockerfile`/`docker-compose.yml` for the HTTP serving app (`src/serve/`) are
included; a pre-built image is also published to GHCR on release:

```bash
docker pull ghcr.io/mikrominiw/mmwrag:latest
```

GPU is optional (see [docker-compose.yml](docker-compose.yml)); the image runs the
serving API on CPU too, just slower for embedding/reranking.

## Project structure

```
src/
  parse/           vision PDF -> Page[]   (cloud / local engines, idempotent cache)
  chunk/           Page[] -> Chunk[]      (structure-aware packing, page spans)
  index/           Chunk[] -> BGE-M3 -> Qdrant   (Embedder / VectorStore interfaces)
  query/           HybridRetriever + RerankingRetriever + ContextBuilder; answer()
  mcp/             MCP server: search / list_books (pure core + thin FastMCP server)
  agent/           LangGraph agent: multi-round, book-routed tool-calling search
  serve/           FastAPI HTTP API: SSE /answer(/agentic), async /ingest, health/metrics
  ingest/          async ingestion job: Job/JobStore, sequential worker, pipeline
  embed_service/   BGE-M3 embedder + cross-encoder reranker as a standalone HTTP service
  observability/   Langfuse tracing, a true no-op when disabled
  eval/            page-level hit@k/MRR/recall@k; end-to-end LLM-judge eval
tests/             unit tests (pure logic on fakes; integration tests skip offline)
notebooks/         runnable examples & measurement runners (*_smoke, eval_*, diag_*)
```

## Status & roadmap

The full pipeline (parse → chunk → index → hybrid+rerank retrieval → MCP → agent →
HTTP serving → observability → network embed-service → async ingestion) is done and
measured. GraphRAG was considered and explicitly not pursued — see
[DECISIONS.md](DECISIONS.md#roadmap) for why.

## License

[MIT](LICENSE) © 2026 mikrominiw