🇵🇱 [Polska wersja](README.pl.md)

# RAG Pipeline

A local, GPU-accelerated retrieval pipeline for research papers — parses PDFs with layout-aware chunking, pulls real bibliographic metadata (DOI → CrossRef), embeds with hybrid dense + sparse vectors, and stores everything in Qdrant for hybrid search with reranking. Built to be queried directly from Claude through an MCP server.

## Why

Dropping papers into a folder makes them invisible to search. This turns a pile of PDFs into something queryable: proper section/paragraph-aware chunks (not a naive character splitter), citation-ready metadata pulled automatically from CrossRef, and hybrid dense+sparse retrieval so both semantic and exact-term queries work — all exposed as MCP tools so Claude can search and ingest papers directly.

## Features

- **Layout-aware PDF parsing** via [Docling](https://github.com/DS4SD/docling) — chunks by paragraph/section instead of a naive character splitter, keeps page numbers and section headings per chunk.
- **Real bibliographic metadata** — regex-extracts a DOI from page 1, resolves full citation data (title, authors, journal, year, volume/issue/pages) via the CrossRef API, falls back to the PDF's own metadata when no DOI is found.
- **Hybrid embeddings** — dense (`BAAI/bge-m3`) + sparse (SPLADE, `prithvida/Splade_PP_EN_v1`) vectors per chunk, stored together in Qdrant for hybrid retrieval.
- **Background ingestion** — a Redis + [RQ](https://python-rq.org/) worker so large PDFs don't block a request; GPU-enabled Docker container for embedding.
- **MCP server** exposing two tools to Claude — `search_papers` (hybrid search with reranking) and `ingest_paper` (drop a PDF straight into the index) — meant to work as a research-paper memory Claude can query directly.
- **Dockerized** — Qdrant + Redis + GPU-enabled API/worker containers via `docker-compose`.

## Tech stack

Python · [Docling](https://github.com/DS4SD/docling) (PDF parsing) · `sentence-transformers` (BGE-M3 dense embeddings) · `fastembed` (SPLADE sparse embeddings) · [Qdrant](https://qdrant.tech/) (hybrid vector search) · Redis + RQ (background job queue) · FastAPI (planned API layer) · MCP (Model Context Protocol server for Claude) · Docker Compose, CUDA 12.9 GPU container.

## Status

Work in progress — not fully wired up end to end yet.

**Works today:** `ingest.py` runs standalone from the CLI — parse a PDF, pull its metadata, embed it (dense + sparse), and upsert into Qdrant. Point it at a running Qdrant instance and it works.

**Missing:** `main.py`, the FastAPI service that's supposed to expose `/search`, `/ingest`, and `/health` — the Dockerfile copies it, the Docker Compose healthcheck pings it, the RQ worker expects something to be enqueuing jobs for it, and `mcp_server.py`'s two Claude tools both call it over HTTP at `localhost:8000`. Without it, the worker has nothing to consume, the MCP tools have no backend to talk to, and `docker-compose up` won't build (the API/worker image's build step copies a file that isn't there).

The `docker-compose.yml` bind mounts also still point at a Windows path (`S:\RAG-data\...`) from an earlier setup — adjust those to wherever you want Qdrant/Redis/model cache data to live before running it.

## Running it (today)

```bash
uv sync

# Qdrant needs to be running somewhere ingest.py can reach:
docker run -p 6333:6333 qdrant/qdrant

python ingest.py path/to/paper.pdf
```

The full pipeline (Docker Compose stack, `/search` and `/ingest` API, MCP tools) will work once `main.py` exists.