# 🔌 MCP Docs Assistant

**Production-grade Retrieval-Augmented Generation (RAG) pipeline over the official [Model Context Protocol](https://modelcontextprotocol.io) documentation — served over REST, MCP tools, and Docker.**

Ask natural-language questions about MCP (architecture, building servers/clients, tools/resources/prompts, security) and get answers grounded in the real docs, with guardrails, PII masking, reranking, semantic caching, and hallucination checking built in.

---

## ✨ Features

| Capability | Implementation |
|---|---|
| 🔀 **Multi-key LLM gateway** | Portkey-routed, load-balanced across 2 Gemini keys + 2 Groq keys, with automatic provider fallback |
| 📚 **Grounded retrieval** | Official MCP docs, chunked + embedded into a persistent Qdrant vector store |
| 🎯 **Reranking** | Cross-encoder (`ms-marco-MiniLM-L-6-v2`) narrows a wide candidate pool down to the most relevant chunks |
| 🛡️ **Guardrails** | NeMo Guardrails (Colang 2.x) — input/output safety checks, jailbreak + instruction-leak detection |
| 🕵️ **PII masking** | Microsoft Presidio — masks emails, phone numbers, credit cards in both input and output |
| 🧮 **Token budgeting** | Retrieved context is greedily fit to a fixed token budget before hitting the LLM |
| ⚡ **Semantic cache** | Embedding-similarity cache (not exact-match) with TTL + size cap |
| 💬 **Multi-turn conversations** | LangGraph checkpointer + follow-up query condensation ("show a Python example of that") |
| 🔍 **Hallucination check** | Runtime LLM-as-judge verdict (`GROUNDED` / `HALLUCINATED`) on every generated answer |
| 📊 **Offline evaluation** | RAGAS metrics (faithfulness, relevancy, context precision/recall) against 25 reference Q&A pairs |
| 🔌 **MCP-native** | Exposes itself as MCP tools (`ask_mcp_docs`, `search_mcp_docs`, …) — usable directly from Claude Desktop, Claude Code, or any MCP host |
| 🌐 **REST API** | FastAPI endpoints for any regular HTTP client |
| 🐳 **Dockerized** | One-command deploy with `docker compose up` |

---

## 🏗️ Architecture

```mermaid
flowchart TD
    A[User Question] --> B[Guard Input<br/>NeMo Guardrails]
    B -->|blocked| Z[Refusal message]
    B -->|allowed| C[Mask Input PII<br/>Presidio]
    C --> D[Condense Follow-up<br/>into standalone question]
    D --> E{Semantic<br/>Cache Hit?}
    E -->|yes| F[Return cached answer]
    E -->|no| G[Retrieve Top-15<br/>Qdrant Vector Store]
    G --> H[Rerank Top-5<br/>Cross-Encoder]
    H --> I[Fit to Token Budget]
    I --> J[Generate Answer<br/>Portkey: Gemini / Groq]
    J --> K[Guard Output<br/>leak / PII pattern check]
    K --> L[Hallucination Check<br/>LLM-as-judge]
    L --> M[Mask Output PII]
    M --> N[Cache + Store History]
    N --> O[Return Answer]
```

Every node above is a module in [`rag_pipeline/`](./rag_pipeline), wired together as a LangGraph `StateGraph` in [`rag_pipeline/graph.py`](./rag_pipeline/graph.py). [`rag_core.py`](./rag_core.py) builds every dependency once (as a singleton) and exposes a small stable API — `chat()`, `search()`, `get_history()`, `cache_stats()` — consumed identically by both the REST layer ([`main.py`](./main.py)) and the MCP layer ([`mcp_server.py`](./mcp_server.py)), so a single vector store / cache / conversation history is shared no matter which interface a request comes through.

---

## 📁 Project Structure

```
mcp-docs-rag-assistant/
├── main.py                    # FastAPI app — REST endpoints + mounts MCP at /mcp
├── mcp_server.py               # MCP tools (stdio standalone, or mounted in main.py)
├── rag_core.py                 # Singleton facade wiring the whole pipeline together
├── rag_pipeline/
│   ├── config.py                 # Env vars / secrets (single source of truth)
│   ├── logging_setup.py          # Logging + Logfire
│   ├── gateway.py                 # Portkey multi-key LLM gateway
│   ├── errors.py                   # Retry + safe-node error handling
│   ├── ingestion.py                 # MCP docs loader + splitter
│   ├── vectorstore.py                # Embeddings + persistent Qdrant store
│   ├── reranker.py                    # Cross-encoder reranking
│   ├── pii_masking.py                  # Presidio PII masking
│   ├── guardrails.py                    # NeMo Guardrails (Colang 2.x)
│   ├── token_management.py               # Context window budgeting
│   ├── semantic_cache.py                  # Embedding-similarity cache
│   ├── query_condensation.py               # Follow-up question rewriting
│   ├── hallucination.py                     # Runtime hallucination judge
│   └── graph.py                              # LangGraph StateGraph — full pipeline
├── scripts/
│   └── evaluate_ragas.py        # Offline RAGAS evaluation (25 reference Q&A)
├── tests/
│   └── test_pipeline.py         # Fast smoke tests (no API keys needed)
├── configs/guardrails/           # Colang rail files (generated at first run)
├── data/                          # Persisted Qdrant vector store (gitignored)
├── notebooks/                      # Original development notebook
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .env.example
```

---

## 🚀 Getting Started

### Prerequisites

- Python 3.11+
- API keys: [Google AI Studio](https://aistudio.google.com/) (Gemini, ×2), [Groq](https://console.groq.com/) (×2), [Portkey](https://portkey.ai/) (gateway + config id)

### 1. Clone and set up a virtual environment

```bash
git clone https://github.com/<your-username>/mcp-docs-rag-assistant.git
cd mcp-docs-rag-assistant
python -m venv venv
venv\Scripts\activate          # Windows
# source venv/bin/activate     # macOS/Linux
```

### 2. Install dependencies

```bash
pip install -r requirements.txt
python -m spacy download en_core_web_sm   # required by Presidio for PII detection
```

### 3. Configure environment variables

```bash
cp .env.example .env
```

Open `.env` and fill in your real keys (`GEMINI_API_KEY_1/2`, `GROQ_API_KEY_1/2`, `PORTKEY_API_KEY`, `PORTKEY_CONFIG_ID`).

### 4. Run the server

```bash
uvicorn main:app --reload
```

> ⏳ **First run only**: the vector store doesn't exist yet, so the server ingests the MCP docs and embeds them in rate-limited batches — this can take **5–10 minutes**. On every subsequent run it loads the persisted store from `data/qdrant_mcp_db/` instantly.

Once you see `Startup: RAG pipeline ready.`, open:

- **`http://localhost:8000/docs`** — interactive Swagger UI, try `POST /chat`
- **`http://localhost:8000/health`** — health check

---

## 📡 REST API

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/chat` | Ask a question. Body: `{"question": "...", "thread_id": "optional"}` |
| `POST` | `/search` | Retrieve + rerank raw context, no generation. Body: `{"query": "...", "top_n": 5}` |
| `GET` | `/history/{thread_id}` | Get conversation history for a thread |
| `GET` | `/cache/stats` | Semantic cache observability |
| `GET` | `/health` | Health check |

---

## 🔌 Using it as an MCP Server

### Standalone (stdio) — for Claude Desktop

Run directly:

```bash
python mcp_server.py
```

Or point a local MCP host at it, e.g. in Claude Desktop's config (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "mcp-docs-assistant": {
      "command": "python",
      "args": ["E:\\mcp-docs-rag-assistant\\mcp_server.py"]
    }
  }
}
```

### Remote (streamable-http) — via FastAPI

When `main.py` is running, the same MCP tools are reachable at:

```
http://localhost:8000/mcp
```

**Available tools:** `ask_mcp_docs`, `search_mcp_docs`, `get_conversation_history`, `cache_stats`.

---

## 🐳 Docker

The **only prerequisite is Docker Desktop** (which bundles Docker Compose) — you do **not** need to separately install Python, the pip dependencies, or `spacy`'s model on your machine. All of that happens automatically *inside* the image when you build it (see the `Dockerfile` — it runs `pip install -r requirements.txt` and `python -m spacy download en_core_web_sm` as build steps).

```bash
# 1. Make sure .env exists (same as the local setup, step 3 above)
cp .env.example .env   # then fill in real keys

# 2. Build and run
docker compose up --build
```

That single command builds the image, installs everything inside it, and starts the container. The `data/` folder is mounted as a volume (see `docker-compose.yml`), so the vector store persists across container restarts — you only pay the slow first-run ingestion cost once, even with Docker.

Server is reachable the same way as running locally: `http://localhost:8000/docs`.

To stop:
```bash
docker compose down
```

To rebuild after changing code or dependencies:
```bash
docker compose up --build
```

---

## 🧪 Testing

Fast smoke tests — no API keys or network calls needed (uses fake embeddings):

```bash
pip install pytest
pytest tests/ -v
```

---

## 📊 Offline Evaluation (RAGAS)

Scores the pipeline against 25 hand-written MCP questions with reference answers, using [RAGAS](https://docs.ragas.io/):

```bash
python scripts/evaluate_ragas.py
```

This does **not** need the server running — it builds the pipeline itself (same singleton as `main.py`/`mcp_server.py`) and prints a metrics table:

- **Faithfulness** — is the answer grounded in the retrieved context?
- **Response Relevancy** — does the answer actually address the question?
- **Context Precision** — is the retrieved context relevant?
- **Context Recall** — does retrieved context cover what the reference answer needs?

> ⏱️ Takes a few minutes: each of the 25 questions runs a real retrieval + generation pass, then every metric is itself scored by an LLM-as-judge call.

---

## ⚙️ Configuration Reference

All configuration lives in `.env` (see `.env.example`). Key variables:

| Variable | Purpose |
|---|---|
| `GEMINI_API_KEY_1/2`, `GROQ_API_KEY_1/2` | Provider keys, load-balanced by Portkey |
| `PORTKEY_API_KEY`, `PORTKEY_CONFIG_ID` | Portkey gateway credentials + routing config |
| `QDRANT_PATH`, `QDRANT_COLLECTION` | Vector store location/name |
| `LOGFIRE_TOKEN` | Optional — omit to fall back to console-only logging |
| `HOST`, `PORT` | Server bind address |

---

## 🛠️ Tech Stack

`FastAPI` · `LangChain` · `LangGraph` · `Qdrant` · `Portkey` · `Sentence-Transformers` · `Presidio` · `NeMo Guardrails` · `RAGAS` · `MCP Python SDK` · `Docker`

---

## 📄 License

MIT — see [LICENSE](./LICENSE).