rag-pipeline
Resolves bibliographic metadata by extracting a DOI from the first page of a PDF and fetching citation data (title, authors, journal, year, volume/issue/pages) from the CrossRef API, falling back to the PDF's own metadata when no DOI is found.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@rag-pipeline find papers on transformer architectures from 2023"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
π΅π± Polska wersja
RAG Pipeline
A local, GPU-accelerated retrieval pipeline for research papers β parses PDFs with layout-aware chunking, pulls real bibliographic metadata (DOI β CrossRef), embeds with hybrid dense + sparse vectors, and stores everything in Qdrant for hybrid search with reranking. Built to be queried directly from Claude through an MCP server.
Why
Dropping papers into a folder makes them invisible to search. This turns a pile of PDFs into something queryable: proper section/paragraph-aware chunks (not a naive character splitter), citation-ready metadata pulled automatically from CrossRef, and hybrid dense+sparse retrieval so both semantic and exact-term queries work β all exposed as MCP tools so Claude can search and ingest papers directly.
Related MCP server: Personal Research Assistant MCP
Features
Layout-aware PDF parsing via Docling β chunks by paragraph/section instead of a naive character splitter, keeps page numbers and section headings per chunk.
Real bibliographic metadata β regex-extracts a DOI from page 1, resolves full citation data (title, authors, journal, year, volume/issue/pages) via the CrossRef API, falls back to the PDF's own metadata when no DOI is found.
Hybrid embeddings β dense (
BAAI/bge-m3) + sparse (SPLADE,prithvida/Splade_PP_EN_v1) vectors per chunk, stored together in Qdrant for hybrid retrieval.Background ingestion β a Redis + RQ worker so large PDFs don't block a request; GPU-enabled Docker container for embedding.
MCP server exposing two tools to Claude β
search_papers(hybrid search with reranking) andingest_paper(drop a PDF straight into the index) β meant to work as a research-paper memory Claude can query directly.Dockerized β Qdrant + Redis + GPU-enabled API/worker containers via
docker-compose.
Tech stack
Python Β· Docling (PDF parsing) Β· sentence-transformers (BGE-M3 dense embeddings) Β· fastembed (SPLADE sparse embeddings) Β· Qdrant (hybrid vector search) Β· Redis + RQ (background job queue) Β· FastAPI (planned API layer) Β· MCP (Model Context Protocol server for Claude) Β· Docker Compose, CUDA 12.9 GPU container.
Status
Work in progress β not fully wired up end to end yet.
Works today: ingest.py runs standalone from the CLI β parse a PDF, pull its metadata, embed it (dense + sparse), and upsert into Qdrant. Point it at a running Qdrant instance and it works.
Missing: main.py, the FastAPI service that's supposed to expose /search, /ingest, and /health β the Dockerfile copies it, the Docker Compose healthcheck pings it, the RQ worker expects something to be enqueuing jobs for it, and mcp_server.py's two Claude tools both call it over HTTP at localhost:8000. Without it, the worker has nothing to consume, the MCP tools have no backend to talk to, and docker-compose up won't build (the API/worker image's build step copies a file that isn't there).
The docker-compose.yml bind mounts also still point at a Windows path (S:\RAG-data\...) from an earlier setup β adjust those to wherever you want Qdrant/Redis/model cache data to live before running it.
Running it (today)
uv sync
# Qdrant needs to be running somewhere ingest.py can reach:
docker run -p 6333:6333 qdrant/qdrant
python ingest.py path/to/paper.pdfThe full pipeline (Docker Compose stack, /search and /ingest API, MCP tools) will work once main.py exists.
Available Tools
2 toolsingest_paperA
Ingest a PDF file from the local filesystem into the search index. The file is chunked, embedded (dense + sparse), and stored in Qdrant.
Inputs: file_path (required).
search_papersA
Search the ingested paper collection using a natural-language query. Returns the most relevant passages with page numbers and sources, ranked by a cross-encoder reranker.
Inputs: query (required), limit.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogβ¦
TDQS
Scored across 2 tools
The two tools have completely distinct roles: one adds documents to the index, the other queries it. There is no overlap or ambiguity between ingestion and search.
Both tools follow a verb_noun pattern and are recognizable at a glance. The only minor inconsistency is that one uses the plural 'papers' while the other uses singular 'paper'.
Two tools is a thin surface, falling below the typical 3-15 well-scoped range. Each tool is essential for the core RAG workflow, but the set still feels minimal for anything beyond basic ingest-and-query usage.
The core ingest and search operations are present, giving agents the main RAG loop. However, there is no way to list, delete, or update ingested papers, which leaves obvious workflow gaps around managing the collection.
Related MCP Connectors
Search arXiv/Semantic Scholar/OpenAlex + medical evidence (PubMed/Europe PMC) + LaTeX/PDF tools.
Private, multimodal knowledge store for agents: hybrid search with cited, provenance-backed results.
Research paper search with real citations and reference formatting for AI assistants
arXiv and OpenAlex/Crossref paper search (an open Google Scholar alternative) as agent tools.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables Claude to perform hybrid search across local documents by combining semantic vector retrieval and BM25 keyword matching for optimal context recovery. It supports multiple file formats including PDF, CSV, and Markdown, leveraging local Ollama models for private and efficient document querying.5MIT
- FlicenseNot gradedqualityDmaintenanceEnables semantic search and conversational querying across a personal research library of PDFs, DOCX, and other documents using a vector database. It provides tools for document summarization, finding related papers, and high-accuracy retrieval for AI clients like Claude Desktop.-
- AlicenseNot gradedqualityDmaintenanceEnables Claude Code to index and semantically search through PDFs, code, and documents with exact citations and zero hallucinations.MIT
- AlicenseAqualityCmaintenanceEnables searching for scientific papers across OpenAlex, CrossRef, and Unpaywall, and downloading open-access PDFs directly through Claude Desktop.51MIT