연구 논문을 위한 로컬 GPU 가속 검색 파이프라인 — 레이아웃 인식 청킹으로 PDF를 파싱하고, 실제 서지 메타데이터(DOI → CrossRef)를 가져오며, 하이브리드 밀집 + 희소 벡터로 임베딩하고, 모든 것을 Qdrant에 저장하여 리랭킹이 포함된 하이브리드 검색을 제공합니다. MCP 서버를 통해 Claude에서 직접 쿼리할 수 있도록 설계되었습니다.
폴더에 논문을 넣어두면 검색에서 사라집니다. 이 프로젝트는 PDF 더미를 쿼리 가능한 무언가로 바꿔줍니다: 적절한 섹션/문단 인식 청크(단순한 문자 분할기가 아님), CrossRef에서 자동으로 가져온 인용 준비 완료 메타데이터, 그리고 의미 검색과 정확한 용어 검색이 모두 작동하도록 하는 하이브리드 밀집+희소 검색 — 이 모든 것이 MCP 도구로 노출되어 Claude가 논문을 직접 검색하고 수집할 수 있습니다.
Related MCP server: Personal Research Assistant MCP
레이아웃 인식 PDF 파싱 — Docling 기반. 단순한 문자 분할기 대신 문단/섹션 단위로 청킹하며, 청크별로 페이지 번호와 섹션 제목을 유지합니다.
실제 서지 메타이터 — 1페이지에서 정규식으로 DOI를 추출하고, CrossRef API를 통해 전체 인용 데이터(제목, 저자, 저널, 연도, 권/호/페이지)를 해석하며, DOI가 없으면 PDF 자체 메타데이터로 대체합니다.
하이브리드 임베딩 — 청크별로 밀집(BAAI/bge-m3) + 희소(SPLADE, prithvida/Splade_PP_EN_v1) 벡터를 생성하여 Qdrant에 함께 저장하고이브리드 검색에 사용합니다.
백그라운드 수집 — Redis + RQ커를 사용하여 대용량 PDF가 요청을 차단하지 않도록 합니다. 임베딩을 위한 GPU 지원 Docker 컨테이너를 포함합니다.
MCP 서버 — Claude에 가지 도구를 노출합니다: search_papers(리랭이 포함된 하브리드 검색)와 ingest_paper(PDF를덱스에 직접 투입). Claude가 직접 쿼리할 수 있는 연구 논문 메모리로 작동하도록 설계되었습니다.
ocker화 — docker-compose를 통해 Qdrant + Redis + GPU 지원 API/워커 컨테이너를 제공합니다.
Python Docling(PDF 파싱) · sentence-transformers(BGE-M3 밀집 임베딩) · fastembed(SPLADE 희소 임베딩) · Qdrant(하이브리드 벡터 검색) · Redis + RQ(백그라운드 작업 큐) · FastAPI(계획된 API 레이어) · MCP(Claude용 Model Context Protocol 서버) · Docker Compose, CUDA 12.9 GPU 컨테이너.
진행 중 — 아직 완전히 종단 간 연결되지는 않았습니다.
현재 작동하는 것: ingest.py는 CLI에서 독립 실행됩니다 — PDF 파싱, 메타데이터 가져오기, 임베딩(밀집 + 희소), Qdrant에 업서트. 실행 중인 Qdrant 인스턴스를 가리키면 작동합니다.
누락된 것: /search, /ingest, /health를 노출해야 하는 FastAPI 서비스인 main.py — Dockerfile이 이를 복사하고, Docker Compose healthcheck 이를 핑하며, RQ 워커는 작업을 큐에 넣을 무언가를 기대하고, mcp_server.py의 두 Claude 도구 모두 localhost:8000에서 HTTP로 이를 호출합니다. 이것이 없으면 워커가 소비할 작업이 없고, MCP 도구가 통신할 백엔드가 없으며, docker-compose up이 빌드되지 않습니다(API/워커 이미지의 빌드 단계가 존재하지 않는 파일을 복사하기 때문).
docker-compose.yml의 바인드 마운트도 여전히 이전 설정의 Windows 경로(S:\RAG-data\...)를 가리키고 있습니다 — 실행 전에 Qdrant/Redis/모델 캐시 데이터를 저장할 위치로 조정하세요.
uv sync
# Qdrant needs to be running somewhere ingest.py can reach:
docker run -p 6333:6333 qdrant/qdrant
python ingest.py path/to/paper.pdf전체 파이프라인(Docker Compose 스택, /search 및 /ingest API, MCP 도구)은 main.py가 생성되면 작동할 것입니다.
Ingest a PDF file from the local filesystem into the search index. The file is chunked, embedded (dense + sparse), and stored in Qdrant.
| Name | Required | Description | Default |
|---|---|---|---|
| file_path | Yes | Absolute or relative path to the PDF file. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of disclosing effects. It clearly explains the processing pipeline: the file is chunked, embedded with dense + sparse vectors, and stored in Qdrant. This lets an agent anticipate side effects. It does not mention idempotence, duplicate handling, or failure behavior, but the core mutation is transparently described.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with no filler. The first sentence states the primary action and target, the second adds the important processing steps. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter ingest operation, the description covers the core workflow and storage destination. It does not describe the return value or failure behavior, but given the simple input and lack of output schema, the definition is reasonably complete for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides 100% coverage for the single parameter file_path, including its type and that it is an absolute or relative path. The description adds little beyond the schema, reinforcing that the file is a local PDF. That meets the baseline of 3 but does not exceed it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Ingest'), a clear resource ('PDF file'), and a destination ('search index'). It also names the sibling (search_papers) implicitly by its focus on adding to an index, so an agent can distinguish ingest from search without extra context.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use this tool (when a PDF needs to be added to the search index) versus the sibling search_papers (when querying is needed), but it never explicitly states 'use this when... use search_papers when...' or mentions any exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Search the ingested paper collection using a natural-language query. Returns the most relevant passages with page numbers and sources, ranked by a cross-encoder reranker.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Number of results to return (default 5). | |
| query | Yes | The search query in natural language. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of behavioral disclosure. It reveals that results are passages rather than full papers, include page numbers and sources, and are ranked by a cross-encoder reranker, which meaningfully describes behavior beyond the tool name.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with the main action front-loaded. The second sentence efficiently packs return format and ranking behavior. No unnecessary words or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter search tool with no output schema, the description adequately conveys what results include (passages, page numbers, sources, ranked order) and the scope (ingested collection). It does not specify the exact result data structure, but that is minor given the straightforward contract and the clear sibling relationship.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description reinforces that the query is natural-language but adds no new detail about the limit parameter or query formatting. It does not need to compensate for schema gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('search') and resource ('ingested paper collection'), and clarifies that queries are natural-language. It also distinguishes itself from the sibling 'ingest_paper' by focusing on retrieval of already-ingested content rather than adding papers.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'ingested paper collection' clearly implies this is for searching papers previously added, giving the agent a strong contextual signal about when to use it. However, it does not explicitly name the alternative or state exclusions, so it falls short of full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0ingest_papersearch_papersScored across 2 tools
The two tools have completely distinct roles: one adds documents to the index, the other queries it. There is no overlap or ambiguity between ingestion and search.
Both tools follow a verb_noun pattern and are recognizable at a glance. The only minor inconsistency is that one uses the plural 'papers' while the other uses singular 'paper'.
Two tools is a thin surface, falling below the typical 3-15 well-scoped range. Each tool is essential for the core RAG workflow, but the set still feels minimal for anything beyond basic ingest-and-query usage.
The core ingest and search operations are present, giving agents the main RAG loop. However, there is no way to list, delete, or update ingested papers, which leaves obvious workflow gaps around managing the collection.
Search arXiv/Semantic Scholar/OpenAlex + medical evidence (PubMed/Europe PMC) + LaTeX/PDF tools.
Research paper search with real citations and reference formatting for AI assistants
Search arXiv and ACL Anthology, retrieve citations and references, and browse web sources to accel…
Search 340M+ academic papers — citation graphs, semantic similarity, and AI literature reviews.
