Skip to main content
Glama

πŸ‡΅πŸ‡± Polska wersja

RAG Pipeline

A local, GPU-accelerated retrieval pipeline for research papers β€” parses PDFs with layout-aware chunking, pulls real bibliographic metadata (DOI β†’ CrossRef), embeds with hybrid dense + sparse vectors, and stores everything in Qdrant for hybrid search with reranking. Built to be queried directly from Claude through an MCP server.

Why

Dropping papers into a folder makes them invisible to search. This turns a pile of PDFs into something queryable: proper section/paragraph-aware chunks (not a naive character splitter), citation-ready metadata pulled automatically from CrossRef, and hybrid dense+sparse retrieval so both semantic and exact-term queries work β€” all exposed as MCP tools so Claude can search and ingest papers directly.

Related MCP server: Personal Research Assistant MCP

Features

  • Layout-aware PDF parsing via Docling β€” chunks by paragraph/section instead of a naive character splitter, keeps page numbers and section headings per chunk.

  • Real bibliographic metadata β€” regex-extracts a DOI from page 1, resolves full citation data (title, authors, journal, year, volume/issue/pages) via the CrossRef API, falls back to the PDF's own metadata when no DOI is found.

  • Hybrid embeddings β€” dense (BAAI/bge-m3) + sparse (SPLADE, prithvida/Splade_PP_EN_v1) vectors per chunk, stored together in Qdrant for hybrid retrieval.

  • Background ingestion β€” a Redis + RQ worker so large PDFs don't block a request; GPU-enabled Docker container for embedding.

  • MCP server exposing two tools to Claude β€” search_papers (hybrid search with reranking) and ingest_paper (drop a PDF straight into the index) β€” meant to work as a research-paper memory Claude can query directly.

  • Dockerized β€” Qdrant + Redis + GPU-enabled API/worker containers via docker-compose.

Tech stack

Python Β· Docling (PDF parsing) Β· sentence-transformers (BGE-M3 dense embeddings) Β· fastembed (SPLADE sparse embeddings) Β· Qdrant (hybrid vector search) Β· Redis + RQ (background job queue) Β· FastAPI (planned API layer) Β· MCP (Model Context Protocol server for Claude) Β· Docker Compose, CUDA 12.9 GPU container.

Status

Work in progress β€” not fully wired up end to end yet.

Works today: ingest.py runs standalone from the CLI β€” parse a PDF, pull its metadata, embed it (dense + sparse), and upsert into Qdrant. Point it at a running Qdrant instance and it works.

Missing: main.py, the FastAPI service that's supposed to expose /search, /ingest, and /health β€” the Dockerfile copies it, the Docker Compose healthcheck pings it, the RQ worker expects something to be enqueuing jobs for it, and mcp_server.py's two Claude tools both call it over HTTP at localhost:8000. Without it, the worker has nothing to consume, the MCP tools have no backend to talk to, and docker-compose up won't build (the API/worker image's build step copies a file that isn't there).

The docker-compose.yml bind mounts also still point at a Windows path (S:\RAG-data\...) from an earlier setup β€” adjust those to wherever you want Qdrant/Redis/model cache data to live before running it.

Running it (today)

uv sync

# Qdrant needs to be running somewhere ingest.py can reach:
docker run -p 6333:6333 qdrant/qdrant

python ingest.py path/to/paper.pdf

The full pipeline (Docker Compose stack, /search and /ingest API, MCP tools) will work once main.py exists.

Available Tools

2 tools
ingest_paperA

Ingest a PDF file from the local filesystem into the search index. The file is chunked, embedded (dense + sparse), and stored in Qdrant.

Inputs: file_path (required).

search_papersA

Search the ingested paper collection using a natural-language query. Returns the most relevant passages with page numbers and sources, ranked by a cross-encoder reranker.

Inputs: query (required), limit.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4/5.0

    Scored across 2 tools

    Disambiguation5/5

    The two tools have completely distinct roles: one adds documents to the index, the other queries it. There is no overlap or ambiguity between ingestion and search.

    Naming Consistency4/5

    Both tools follow a verb_noun pattern and are recognizable at a glance. The only minor inconsistency is that one uses the plural 'papers' while the other uses singular 'paper'.

    Tool Count3/5

    Two tools is a thin surface, falling below the typical 3-15 well-scoped range. Each tool is essential for the core RAG workflow, but the set still feels minimal for anything beyond basic ingest-and-query usage.

    Completeness3/5

    The core ingest and search operations are present, giving agents the main RAG loop. However, there is no way to list, delete, or update ingested papers, which leaves obvious workflow gaps around managing the collection.

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables Claude to perform hybrid search across local documents by combining semantic vector retrieval and BM25 keyword matching for optimal context recovery. It supports multiple file formats including PDF, CSV, and Markdown, leveraging local Ollama models for private and efficient document querying.
      5
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      Enables semantic search and conversational querying across a personal research library of PDFs, DOCX, and other documents using a vector database. It provides tools for document summarization, finding related papers, and high-accuracy retrieval for AI clients like Claude Desktop.
      -