Skip to main content
Glama

🇵🇱 波兰语版本

RAG Pipeline

一个面向研究论文的本地、GPU 加速检索管道——通过布局感知分块解析 PDF,获取真实的书目元数据(DOI → CrossRef),使用混合稠密 + 稀疏向量进行嵌入,并将所有内容存储在 Qdrant 中,用于带重排序的混合搜索。构建目的是通过 MCP 服务器让 Claude 直接查询。

为什么

把论文丢进文件夹,它们就搜索不到了。这个管道把一堆 PDF 变成可查询的东西:按章节/段落感知的合理分块(而不是朴素的字符切分器)、从 CrossRef 自动拉取的可用于引用的元数据,以及混合稠密+稀疏检索,让语义查询和精确词项查询都能工作——所有这些都作为 MCP 工具暴露出来,让 Claude 可以直接搜索和摄取论文。

Related MCP server: Personal Research Assistant MCP

特性

  • 布局感知的 PDF 解析,通过 Docling ——按段落/章节分块,而不是朴素的字符切分器,保留每个块的页码和章节标题。

  • 真实的书目元数据 ——从第 1 页用正则提取 DOI,通过 CrossRef API 解析完整的引文数据(标题、作者、期刊、年份、卷/期/页码),当未找到 DOI 时回退到 PDF 自身的元数据。

  • 混合嵌入 ——每个块包含稠密(BAAI/bge-m3)+ 稀疏(SPLADE、prithvida/Splade_PP_EN_v1)向量,一起存储在 Qdrant 中用于混合检索。

  • 后台摄取 ——一个 Redis + RQ 工作进程,让大的 PDF 不会阻塞请求;支持 GPU 的 Docker 容器用于嵌入。

  • MCP 服务器向 Claude 暴露两个工具 ——search_papers(带重排序的混合搜索)和 ingest_paper(把 PDF 直接放入索引)——旨在作为 Claude 可以直接查询的研究论文记忆。

  • Docker 化 ——通过 docker-compose 运行 Qdrant + Redis + 启用 GPU 的 API/工作进程容器。

技术栈

Python · Docling(PDF 解析)· sentence-transformers(BGE-M3 稠密嵌入)· fastembed(SPLADE 稀疏嵌入)· Qdrant(混合向量搜索)· Redis + RQ(后台任务队列)· FastAPI(计划中的 API 层)· MCP(面向 Claude 的 Model Context Protocol 服务器)· Docker Compose、CUDA 12.9 GPU 容器。

状态

正在进行中 ——尚未完全端到端打通。

目前已可用: ingest.py 可以从 CLI 独立运行 ——解析 PDF,获取其元数据,进行嵌入(稠密 + 稀疏),然后 upsert 到 Qdrant。把它指向一个正在运行的 Qdrant 实例就能工作。

缺失: main.py ——本应暴露 /search、/ingest 和 /health 的 FastAPI 服务。Dockerfile 会复制它,Docker Compose 健康检查会 ping 它,RQ 工作进程期待有东西为它入队任务,mcp_server.py 的两个 Claude 工具也都通过 HTTP 在 localhost:8000 调用它。没有它,工作进程就没有可消费的任务,MCP 工具就没有后端可对接,docker-compose up 也无法构建(API/工作进程镜像的构建步骤会复制一个不存在的文件)。

docker-compose.yml 的绑定挂载也仍然指向早期设置中的 Windows 路径(S:\RAG-data\...)——在运行之前,请将这些路径调整为你希望 Qdrant/Redis/模型缓存数据存放的位置。

当前运行方式

uv sync

# Qdrant needs to be running somewhere ingest.py can reach:
docker run -p 6333:6333 qdrant/qdrant

python ingest.py path/to/paper.pdf

一旦 main.py 存在,完整管道(Docker Compose 栈、/search 和 /ingest API、MCP 工具)就能工作。

Available Tools

2 tools
ingest_paperA

Ingest a PDF file from the local filesystem into the search index. The file is chunked, embedded (dense + sparse), and stored in Qdrant.

NameRequiredDescriptionDefault
file_pathYesAbsolute or relative path to the PDF file.

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of disclosing effects. It clearly explains the processing pipeline: the file is chunked, embedded with dense + sparse vectors, and stored in Qdrant. This lets an agent anticipate side effects. It does not mention idempotence, duplicate handling, or failure behavior, but the core mutation is transparently described.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences with no filler. The first sentence states the primary action and target, the second adds the important processing steps. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter ingest operation, the description covers the core workflow and storage destination. It does not describe the return value or failure behavior, but given the simple input and lack of output schema, the definition is reasonably complete for an agent to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already provides 100% coverage for the single parameter file_path, including its type and that it is an absolute or relative path. The description adds little beyond the schema, reinforcing that the file is a local PDF. That meets the baseline of 3 but does not exceed it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Ingest'), a clear resource ('PDF file'), and a destination ('search index'). It also names the sibling (search_papers) implicitly by its focus on adding to an index, so an agent can distinguish ingest from search without extra context.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use this tool (when a PDF needs to be added to the search index) versus the sibling search_papers (when querying is needed), but it never explicitly states 'use this when... use search_papers when...' or mentions any exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_papersA

Search the ingested paper collection using a natural-language query. Returns the most relevant passages with page numbers and sources, ranked by a cross-encoder reranker.

NameRequiredDescriptionDefault
limitNoNumber of results to return (default 5).
queryYesThe search query in natural language.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It reveals that results are passages rather than full papers, include page numbers and sources, and are ranked by a cross-encoder reranker, which meaningfully describes behavior beyond the tool name.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with the main action front-loaded. The second sentence efficiently packs return format and ranking behavior. No unnecessary words or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter search tool with no output schema, the description adequately conveys what results include (passages, page numbers, sources, ranked order) and the scope (ingested collection). It does not specify the exact result data structure, but that is minor given the straightforward contract and the clear sibling relationship.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description reinforces that the query is natural-language but adds no new detail about the limit parameter or query formatting. It does not need to compensate for schema gaps.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('search') and resource ('ingested paper collection'), and clarifies that queries are natural-language. It also distinguishes itself from the sibling 'ingest_paper' by focusing on retrieval of already-ingested content rather than adding papers.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'ingested paper collection' clearly implies this is for searching papers previously added, giving the agent a strong contextual signal about when to use it. However, it does not explicitly name the alternative or state exclusions, so it falls short of full routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv0.1.0
    • First observedingest_paper
    • First observedsearch_papers

TDQS

A4/5.0

Scored across 2 tools

Disambiguation5/5

The two tools have completely distinct roles: one adds documents to the index, the other queries it. There is no overlap or ambiguity between ingestion and search.

Naming Consistency4/5

Both tools follow a verb_noun pattern and are recognizable at a glance. The only minor inconsistency is that one uses the plural 'papers' while the other uses singular 'paper'.

Tool Count3/5

Two tools is a thin surface, falling below the typical 3-15 well-scoped range. Each tool is essential for the core RAG workflow, but the set still feels minimal for anything beyond basic ingest-and-query usage.

Completeness3/5

The core ingest and search operations are present, giving agents the main RAG loop. However, there is no way to list, delete, or update ingested papers, which leaves obvious workflow gaps around managing the collection.

Maintenance

ActivityStale
ResponsivenessNo recent community issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables Claude to perform hybrid search across local documents by combining semantic vector retrieval and BM25 keyword matching for optimal context recovery. It supports multiple file formats including PDF, CSV, and Markdown, leveraging local Ollama models for private and efficient document querying.
    5
    MIT
  • F
    license
    Not graded
    quality
    D
    maintenance
    Enables semantic search and conversational querying across a personal research library of PDFs, DOCX, and other documents using a vector database. It provides tools for document summarization, finding related papers, and high-accuracy retrieval for AI clients like Claude Desktop.
    -