# MCP-based Academic Paper Retrieval RAG System

> A production-oriented, fully pluggable Retrieval-Augmented Generation (RAG) system exposed as an MCP (Model Context Protocol) server — enabling AI coding assistants like GitHub Copilot and Claude Desktop to query private knowledge bases via standard Tool Calling.

---

## Overview

Engineers on R&D teams frequently search academic literature for technical research, but papers are scattered across internal file systems and keyword search fails to understand semantic intent — or integrate with AI coding assistant workflows.

This project solves that by building an **MCP-based academic paper retrieval RAG system**: ingest your PDFs once, then ask questions directly from GitHub Copilot or Claude Desktop via standard Tool Calling. Responses include inline citations and are grounded in your private document collection. The system supports both API-based LLM backends and fully local Ollama deployment. When configured with local embedding and local LLM backends, it can run without external network dependency, making it suitable for privacy-sensitive or air-gapped environments.


**Benchmark results** (10 papers, 1,104 chunks, 100 Golden QA pairs):

| Metric | Hybrid Search (baseline) | + BGE Reranker v2-m3 |
|--------|--------------------------|----------------------|
| Hit Rate@10 | 89% | 89% |
| MRR | 0.61 | **0.83** (+36%) |
| NDCG@10 | 0.68 | **0.84** (+24%) |

> Ablation: replacing BGE Reranker v2-m3 with a general-purpose MS MARCO Cross-Encoder degraded MRR on academic text — domain-aligned reranking matters.

---

## Architecture

```
PDF Documents
     ↓
┌─────────────────────────────────────┐
│         Ingestion Pipeline          │
│  Load → Split → Transform → Embed  │
│           → Upsert                  │
└──────────────────┬──────────────────┘
                   │
         ┌─────────┴──────────┐
         │                    │
   ChromaDB (Dense)      BM25 Index (Sparse)
   BGE-M3 vectors        Term frequencies
         │                    │
         └─────────┬──────────┘
                   │ RRF Fusion
                   ↓
          BGE Reranker v2-m3
                   ↓
┌─────────────────────────────────────┐
│           MCP Server                │
│  JSON-RPC 2.0 + Stdio Transport     │
│                                     │
│  • query_knowledge_hub              │
│  • list_collections                 │
│  • get_document_summary             │
└─────────────────────────────────────┘
         ↓                ↓
  GitHub Copilot    Claude Desktop
```

---

## Key Features

### Two-Stage Hybrid Retrieval
- **Coarse ranking**: BGE-M3 dense vectors + BM25 sparse retrieval run in parallel, fused via Reciprocal Rank Fusion (RRF)
- **Fine ranking**: BGE Reranker v2-m3 (same model family as the embedder) re-scores the top candidates — domain-aligned reranking for academic text
- Graceful fallback: reranker failure automatically falls back to fusion order with `fallback=True` metadata

### MCP-Compliant RAG Server
- Full JSON-RPC 2.0 protocol over stdio transport
- Three tools available to any MCP client:
  - `query_knowledge_hub` — semantic search with inline citations
  - `list_collections` — enumerate available knowledge bases
  - `get_document_summary` — retrieve title, summary, tags for a document
- Structured multi-modal responses with citation blocks

### Intelligent Ingestion Pipeline
- **5 stages**: Load → Split → Transform → Embed → Upsert
- pdfplumber PDF parsing with image extraction and `[IMAGE: id]` placeholder injection
- Optional LLM-driven chunk refinement and metadata enrichment (Title / Summary / Tags)
- SHA-256 content hashing for idempotent incremental ingestion — re-running on unchanged files is a no-op

### Fully Pluggable Architecture
Every core component is swappable via `config/settings.yaml` with zero code changes:

| Component | Supported Backends |
|-----------|-------------------|
| LLM | OpenAI / Azure / DeepSeek / Ollama |
| Embedding | OpenAI / Azure / Ollama (BGE-M3, nomic-embed-text, …) |
| Vector Store | ChromaDB (local persistence) |
| Reranker | BGE / Cross-Encoder / LLM / None |
| Splitter | Fixed / Recursive / Semantic (TODO) |
| Evaluator | Custom (Hit Rate / MRR / NDCG) / Ragas |

When configured with Ollama-based local LLM and embedding backends, the system can run fully locally with no data leaving the environment.

### Observability Dashboard
6-page Streamlit management platform:

| Page | Function |
|------|----------|
| System Overview | Live component configuration view |
| Data Browser | Browse documents and chunks with metadata |
| Ingestion Manager | Upload PDFs, monitor real-time progress, delete documents |
| Ingestion Traces | Per-run stage timing (load / split / transform / embed / upsert) |
| Query Traces | Per-query stage breakdown (dense / sparse / fusion / rerank) |
| Evaluation | Run Golden QA evaluation and view Hit Rate / MRR / NDCG results |

### Evaluation Framework
- Custom evaluator: Hit Rate@K, MRR, NDCG@10
- Ragas integration (Faithfulness / Answer Relevancy / Context Precision) — optional, requires API key
- Golden Test Set regression pipeline — every retrieval strategy change produces quantified before/after metrics

---

## Quick Start

### Prerequisites
- Python 3.10+
- Ollama (for local embedding) or an OpenAI-compatible API key

```bash
git clone <repo-url>
cd Modular-RAG-MCP-Server

python -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
```

### Configure

Edit `config/settings.yaml` to set your embedding provider and model:

```yaml
embedding:
  provider: ollama        # or openai / azure
  model: bge-m3

rerank:
  backend: cross_encoder  # or none / llm
  model: BAAI/bge-reranker-v2-m3
```

Set API keys if using cloud providers:
```bash
export OPENAI_API_KEY="sk-..."
```

### Ingest Documents

```bash
python scripts/ingest.py --path /path/to/papers/ --collection research
```

### Query

```bash
python scripts/query.py --query "How does RRF fusion work?" --verbose
```

### Launch Dashboard

```bash
streamlit run src/observability/dashboard/app.py
# Open http://localhost:8501
```

### Run Evaluation

```bash
python scripts/evaluate.py --collection research --output results/eval.json
```

---

## MCP Integration

### GitHub Copilot (VS Code)

Create `.vscode/mcp.json`:

```json
{
  "servers": {
    "modular-rag": {
      "type": "stdio",
      "command": "python",
      "args": ["src/mcp_server/server.py", "--config", "config/settings.yaml"],
      "env": {
        "OPENAI_API_KEY": "${env:OPENAI_API_KEY}"
      }
    }
  }
}
```

### Claude Desktop

Add to `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "modular-rag": {
      "command": "python",
      "args": ["/absolute/path/to/src/mcp_server/server.py"],
      "env": {
        "OPENAI_API_KEY": "your-api-key"
      }
    }
  }
}
```

---

## Running Tests

```bash
pytest -q                        # full suite (649 tests)
pytest -q tests/unit/            # unit tests only
pytest -q tests/integration/     # integration tests
pytest -q tests/e2e/             # E2E: MCP client, dashboard smoke, recall regression
```

---

## Tech Stack

- **Retrieval**: BGE-M3 · BM25 · RRF · BGE Reranker v2-m3
- **Storage**: ChromaDB · SQLite · JSON Lines
- **Protocol**: MCP / JSON-RPC 2.0 / Stdio Transport
- **LLM Backends**: OpenAI · Azure OpenAI · Ollama · DeepSeek
- **Evaluation**: Custom (Hit Rate / MRR / NDCG) · Ragas
- **Dashboard**: Streamlit
- **Testing**: Pytest · 638 tests across Unit / Integration / E2E
- **Design Patterns**: Abstract Base Class · Factory Pattern · Dependency Injection · TDD