🇵🇱 Polska wersja

RAG Pipeline

A local, GPU-accelerated retrieval pipeline for research papers — parses PDFs with layout-aware chunking, pulls real bibliographic metadata (DOI → CrossRef), embeds with hybrid dense + sparse vectors, and stores everything in Qdrant for hybrid search with reranking. Built to be queried directly from Claude through an MCP server.

Why

Dropping papers into a folder makes them invisible to search. This turns a pile of PDFs into something queryable: proper section/paragraph-aware chunks (not a naive character splitter), citation-ready metadata pulled automatically from CrossRef, and hybrid dense+sparse retrieval so both semantic and exact-term queries work — all exposed as MCP tools so Claude can search and ingest papers directly.

Related MCP server: Personal Research Assistant MCP

Features

• Layout-aware PDF parsing via Docling — chunks by paragraph/section instead of a naive character splitter, keeps page numbers and section headings per chunk.

• Real bibliographic metadata — regex-extracts a DOI from page 1, resolves full citation data (title, authors, journal, year, volume/issue/pages) via the CrossRef API, falls back to the PDF's own metadata when no DOI is found.

• Hybrid embeddings — dense (BAAI/bge-m3) + sparse (SPLADE, prithvida/Splade_PP_EN_v1) vectors per chunk, stored together in Qdrant for hybrid retrieval.

• Background ingestion — a Redis + RQ worker so large PDFs don't block a request; GPU-enabled Docker container for embedding.

• MCP server exposing two tools to Claude — search_papers (hybrid search with reranking) and ingest_paper (drop a PDF straight into the index) — meant to work as a research-paper memory Claude can query directly.

• Dockerized — Qdrant + Redis + GPU-enabled API/worker containers via docker-compose.

Tech stack

Python · Docling (PDF parsing) · sentence-transformers (BGE-M3 dense embeddings) · fastembed (SPLADE sparse embeddings) · Qdrant (hybrid vector search) · Redis + RQ (background job queue) · FastAPI (planned API layer) · MCP (Model Context Protocol server for Claude) · Docker Compose, CUDA 12.9 GPU container.

Status

Work in progress — not fully wired up end to end yet.

Works today: ingest.py runs standalone from the CLI — parse a PDF, pull its metadata, embed it (dense + sparse), and upsert into Qdrant. Point it at a running Qdrant instance and it works.

Missing: main.py, the FastAPI service that's supposed to expose /search, /ingest, and /health — the Dockerfile copies it, the Docker Compose healthcheck pings it, the RQ worker expects something to be enqueuing jobs for it, and mcp_server.py's two Claude tools both call it over HTTP at localhost:8000. Without it, the worker has nothing to consume, the MCP tools have no backend to talk to, and docker-compose up won't build (the API/worker image's build step copies a file that isn't there).

The docker-compose.yml bind mounts also still point at a Windows path (S:\RAG-data\...) from an earlier setup — adjust those to wherever you want Qdrant/Redis/model cache data to live before running it.

Running it (today)

uv sync

# Qdrant needs to be running somewhere ingest.py can reach:
docker run -p 6333:6333 qdrant/qdrant

python ingest.py path/to/paper.pdf

The full pipeline (Docker Compose stack, /search and /ingest API, MCP tools) will work once main.py exists.