# Modular RAG MCP Server

> A pluggable, observable Retrieval-Augmented Generation (RAG) service that exposes its knowledge base as tools over the **Model Context Protocol (MCP)**, so AI assistants such as GitHub Copilot, Claude Desktop, Claude Code and Cursor can query your private documents directly.

---

## 📖 Table of Contents

- [Overview](#-overview)
- [Architecture](#-architecture)
- [Key Features](#-key-features)
- [Getting Started](#-getting-started)
- [Usage](#-usage)
- [Connecting an MCP Client](#-connecting-an-mcp-client)
- [Configuration](#-configuration)
- [Testing](#-testing)
- [Project Structure](#-project-structure)
- [Extending the System](#-extending-the-system)
- [Roadmap](#-roadmap)
- [License](#-license)

---

## 🏗️ Overview

Modular RAG MCP Server is an end-to-end RAG system covering the full lifecycle of a knowledge base:

| Module | Capability | Details |
|--------|-----------|---------|
| **Ingestion Pipeline** | PDF → Markdown → Chunk → Transform → Embedding → Upsert | Full ingestion pipeline with multimodal image captioning |
| **Hybrid Search** | Dense (vector) + Sparse (BM25) + RRF fusion + Rerank | Two-stage retrieval: broad recall followed by precise re-ranking |
| **MCP Server** | Standard MCP tools | `query_knowledge_hub`, `list_collections`, `get_document_summary` |
| **Dashboard** | Six-page Streamlit admin console | System overview / data browser / ingestion management / ingestion traces / query traces / evaluation |
| **Evaluation** | Ragas + custom evaluators | Regression testing against a golden test set instead of tuning "by feel" |
| **Observability** | End-to-end tracing | Every intermediate state of both the ingestion and query pipelines is inspectable |
| **Agent Skills** | Spec-driven development workflow | `auto-coder`, `qa-tester`, `setup`, `package` and other skills cover coding, testing, setup and packaging |

The complete design — architecture, module interfaces, test strategy and the phased task schedule — is documented in [DEV_SPEC.md](DEV_SPEC.md).

---

## 🧭 Architecture

```
                 ┌──────────────────────────────────────────────┐
  MCP Clients    │  Copilot · Claude Desktop · Claude Code · …  │
                 └───────────────────────┬──────────────────────┘
                                         │ MCP (stdio / JSON-RPC)
                 ┌───────────────────────▼──────────────────────┐
  MCP Server     │ query_knowledge_hub · list_collections ·     │
                 │ get_document_summary                         │
                 └───────────────────────┬──────────────────────┘
                                         │
                 ┌───────────────────────▼──────────────────────┐
  Query Engine   │ Query processing → Dense + BM25 retrieval →  │
                 │ RRF fusion → (optional) rerank → response    │
                 │ builder with citations                       │
                 └───────────────────────┬──────────────────────┘
                                         │
                 ┌───────────────────────▼──────────────────────┐
  Storage        │ Vector store (Chroma) · BM25 index · images  │
                 └───────────────────────▲──────────────────────┘
                                         │
                 ┌───────────────────────┴──────────────────────┐
  Ingestion      │ Load PDF → split → refine / enrich metadata →│
                 │ caption images (Vision LLM) → embed → upsert │
                 └──────────────────────────────────────────────┘

  Cross-cutting: pluggable libs (LLM / Embedding / Reranker / Splitter /
  VectorStore / Evaluator / Loader) · tracing · Streamlit dashboard
```

---

## ✨ Key Features

**🔌 Fully pluggable architecture** — LLM, embedding, reranker, splitter, vector store and evaluator each sit behind an abstract interface with a factory. Swap backends by editing `config/settings.yaml`; no code changes required.

**🔍 Hybrid search + reranking** — BM25 sparse retrieval handles exact matches on proper nouns and terminology, dense embeddings handle synonyms and semantic similarity. Results are merged with Reciprocal Rank Fusion (RRF), then optionally re-ranked by a Cross-Encoder or an LLM to balance recall and precision.

**🖼️ Multimodal image handling** — An image-to-text strategy uses a Vision LLM to caption images and stitches the captions into their chunks, so the plain-text RAG pipeline can retrieve images by text queries.

**📡 MCP ecosystem integration** — Implements the Model Context Protocol standard, so any MCP client can use the knowledge base without building a frontend. Build once, use everywhere.

**📊 Visual management + automated evaluation** — A Streamlit dashboard provides data management and pipeline tracing; Ragas and custom evaluators create a data-driven feedback loop.

**🧪 Three-layer test suite** — Unit, integration and end-to-end tests cover isolated module logic, cross-module interactions and full flows (MCP client, dashboard).

---

## 🚀 Getting Started

### Prerequisites

- Python 3.10+
- An API key for at least one LLM / embedding provider (OpenAI, Azure OpenAI, DeepSeek), or a local [Ollama](https://ollama.com) install

### Installation

```bash
git clone https://github.com/CrstalLi02/modular-rag-mcp-server.git
cd modular-rag-mcp-server

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

pip install -e ".[dev]"
```

### Configure providers

Edit `config/settings.yaml` to choose your LLM, embedding, vision and vector-store providers, and export the matching API keys (for example `OPENAI_API_KEY`).

Alternatively, open the project in an AI coding assistant that supports agent skills (Claude Code, GitHub Copilot) and run the **`setup`** skill. It walks through provider selection, API key configuration, dependency installation, config generation and launching the dashboard.

---

## 🛠️ Usage

**Ingest documents**

```bash
# A single file
python scripts/ingest.py --path documents/report.pdf --collection contracts

# A whole directory
python scripts/ingest.py --path documents/ --collection technical_docs

# Force re-ingestion of already-processed files
python scripts/ingest.py --path documents/report.pdf --collection contracts --force
```

**Query from the command line**

```bash
python scripts/query.py --query "How do I configure Azure OpenAI?" --collection technical_docs
python scripts/query.py --query "What is RRF?" --verbose
python scripts/query.py --query "What is RRF?" --no-rerank
```

**Run the evaluation suite**

```bash
python scripts/evaluate.py
python scripts/evaluate.py --test-set path/to/golden.json --collection technical_docs
```

**Launch the dashboard**

```bash
python scripts/start_dashboard.py            # http://localhost:8501
python scripts/start_dashboard.py --port 8502
```

**Start the MCP server**

```bash
python -m src.mcp_server.server   # run from the project root
```

---

## 🔗 Connecting an MCP Client

The server speaks MCP over stdio, so any MCP-compatible tool can use it.

**Claude Desktop / Claude Code** (`claude_desktop_config.json` or `.mcp.json`):

```json
{
  "mcpServers": {
    "knowledge-hub": {
      "command": "bash",
      "args": ["-c", "cd /absolute/path/to/modular-rag-mcp-server && .venv/bin/python -m src.mcp_server.server"]
    }
  }
}
```

**GitHub Copilot (VS Code)** — add the same server under `servers` in `.vscode/mcp.json`.

**Cursor** — add it under `mcpServers` in `.cursor/mcp.json`.

Exposed tools:

| Tool | Description |
|------|-------------|
| `query_knowledge_hub` | Hybrid search over a collection; returns ranked chunks with source citations |
| `list_collections` | Lists available collections and their document counts |
| `get_document_summary` | Returns the summary and metadata of an ingested document |

---

## ⚙️ Configuration

All behaviour is driven by `config/settings.yaml`:

| Section | Options |
|---------|---------|
| `llm.provider` | `openai`, `azure`, `ollama`, `deepseek` |
| `embedding.provider` | `openai`, `azure`, `ollama` |
| `vision_llm.provider` | `openai`, `azure`, `ollama` |
| `vector_store.provider` | `chroma` (interfaces prepared for `qdrant`, `pinecone`) |
| `rerank.provider` | `none`, `cross_encoder`, `llm` |
| `evaluation.provider` | `ragas`, `deepeval`, `custom` |

Prompt templates for chunk refinement, metadata enrichment, image captioning and LLM reranking live in `config/prompts/`.

---

## 🧪 Testing

```bash
pytest tests/unit                 # fast, no external services
pytest tests/integration          # requires provider credentials
pytest tests/e2e                  # full MCP client / dashboard flows
```

Integration tests read credentials from `config/test_credentials.yaml` (copy it from `config/test_credentials.yaml.example`).

---

## 📂 Project Structure

```
├── config/              # settings.yaml + prompt templates
├── scripts/             # ingest / query / evaluate / dashboard entry points
├── src/
│   ├── core/            # settings, shared types, query engine, response builder, tracing
│   ├── ingestion/       # chunking, transforms, embedding, storage, pipeline
│   ├── libs/            # pluggable backends: llm, embedding, reranker, splitter,
│   │                    #   vector_store, loader, evaluator
│   ├── mcp_server/      # MCP protocol handler, server and tools
│   └── observability/   # logging, evaluation, Streamlit dashboard
├── tests/               # unit / integration / e2e + fixtures
├── .claude/skills/      # agent skills (Claude Code)
├── .github/skills/      # agent skills (GitHub Copilot)
├── DEV_SPEC.md          # full design spec and task schedule
└── src/mcp_server/server.py   # MCP server entry point (stdio)
```

---

## 🧩 Extending the System

**Add a provider (e.g. Qwen, another vector DB)** — every backend in `src/libs/` uses the factory pattern. To add one: (1) implement the base class, (2) register it in the factory, (3) select it in `settings.yaml`. Most providers follow the OpenAI request format, so new LLM backends are usually small.

**Support more document formats** — the loader layer is built around `BaseLoader`. PDF is implemented today; Word, Markdown or HTML support only needs a new loader class registered alongside the PDF loader.

---

## 📌 Roadmap

| Item | Status |
|------|--------|
| Custom evaluator | Framework in place; evaluation methods and golden dataset to be completed |
| Cross-Encoder reranker | Framework in place; needs end-to-end testing with a local model such as `cross-encoder/ms-marco-MiniLM-L-6-v2` |
| Additional loaders (Word / Markdown / HTML) | Planned |
| Qdrant / Pinecone vector stores | Planned |

---

## 📄 License

Released under the [MIT License](LICENSE).