🔌 MCP Docs Assistant

Production-grade Retrieval-Augmented Generation (RAG) pipeline over the official Model Context Protocol documentation — served over REST, MCP tools, and Docker.

Ask natural-language questions about MCP (architecture, building servers/clients, tools/resources/prompts, security) and get answers grounded in the real docs, with guardrails, PII masking, reranking, semantic caching, and hallucination checking built in.

✨ Features

Related MCP server: FusionPact MCP Server

🏗️ Architecture

flowchart TD
    A[User Question] --> B[Guard Input<br/>NeMo Guardrails]
    B -->|blocked| Z[Refusal message]
    B -->|allowed| C[Mask Input PII<br/>Presidio]
    C --> D[Condense Follow-up<br/>into standalone question]
    D --> E{Semantic<br/>Cache Hit?}
    E -->|yes| F[Return cached answer]
    E -->|no| G[Retrieve Top-15<br/>Qdrant Vector Store]
    G --> H[Rerank Top-5<br/>Cross-Encoder]
    H --> I[Fit to Token Budget]
    I --> J[Generate Answer<br/>Portkey: Gemini / Groq]
    J --> K[Guard Output<br/>leak / PII pattern check]
    K --> L[Hallucination Check<br/>LLM-as-judge]
    L --> M[Mask Output PII]
    M --> N[Cache + Store History]
    N --> O[Return Answer]

Every node above is a module in rag_pipeline/, wired together as a LangGraph StateGraph in rag_pipeline/graph.py. rag_core.py builds every dependency once (as a singleton) and exposes a small stable API — chat(), search(), get_history(), cache_stats() — consumed identically by both the REST layer (main.py) and the MCP layer (mcp_server.py), so a single vector store / cache / conversation history is shared no matter which interface a request comes through.

📁 Project Structure

mcp-docs-rag-assistant/
├── main.py                    # FastAPI app — REST endpoints + mounts MCP at /mcp
├── mcp_server.py               # MCP tools (stdio standalone, or mounted in main.py)
├── rag_core.py                 # Singleton facade wiring the whole pipeline together
├── rag_pipeline/
│   ├── config.py                 # Env vars / secrets (single source of truth)
│   ├── logging_setup.py          # Logging + Logfire
│   ├── gateway.py                 # Portkey multi-key LLM gateway
│   ├── errors.py                   # Retry + safe-node error handling
│   ├── ingestion.py                 # MCP docs loader + splitter
│   ├── vectorstore.py                # Embeddings + persistent Qdrant store
│   ├── reranker.py                    # Cross-encoder reranking
│   ├── pii_masking.py                  # Presidio PII masking
│   ├── guardrails.py                    # NeMo Guardrails (Colang 2.x)
│   ├── token_management.py               # Context window budgeting
│   ├── semantic_cache.py                  # Embedding-similarity cache
│   ├── query_condensation.py               # Follow-up question rewriting
│   ├── hallucination.py                     # Runtime hallucination judge
│   └── graph.py                              # LangGraph StateGraph — full pipeline
├── scripts/
│   └── evaluate_ragas.py        # Offline RAGAS evaluation (25 reference Q&A)
├── tests/
│   └── test_pipeline.py         # Fast smoke tests (no API keys needed)
├── configs/guardrails/           # Colang rail files (generated at first run)
├── data/                          # Persisted Qdrant vector store (gitignored)
├── notebooks/                      # Original development notebook
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .env.example

🚀 Getting Started

Prerequisites

• Python 3.11+

• API keys: Google AI Studio (Gemini, ×2), Groq (×2), Portkey (gateway + config id)

1. Clone and set up a virtual environment

git clone https://github.com/<your-username>/mcp-docs-rag-assistant.git
cd mcp-docs-rag-assistant
python -m venv venv
venv\Scripts\activate          # Windows
# source venv/bin/activate     # macOS/Linux

2. Install dependencies

pip install -r requirements.txt
python -m spacy download en_core_web_sm   # required by Presidio for PII detection

3. Configure environment variables

cp .env.example .env

Open .env and fill in your real keys (GEMINI_API_KEY_1/2, GROQ_API_KEY_1/2, PORTKEY_API_KEY, PORTKEY_CONFIG_ID).

4. Run the server

uvicorn main:app --reload

⏳ First run only: the vector store doesn't exist yet, so the server ingests the MCP docs and embeds them in rate-limited batches — this can take 5–10 minutes. On every subsequent run it loads the persisted store from data/qdrant_mcp_db/ instantly.

Once you see Startup: RAG pipeline ready., open:

• http://localhost:8000/docs — interactive Swagger UI, try POST /chat

• http://localhost:8000/health — health check

📡 REST API

🔌 Using it as an MCP Server

Standalone (stdio) — for Claude Desktop

Run directly:

python mcp_server.py

Or point a local MCP host at it, e.g. in Claude Desktop's config (claude_desktop_config.json):

{
  "mcpServers": {
    "mcp-docs-assistant": {
      "command": "python",
      "args": ["E:\\mcp-docs-rag-assistant\\mcp_server.py"]
    }
  }
}

Remote (streamable-http) — via FastAPI

When main.py is running, the same MCP tools are reachable at:

http://localhost:8000/mcp

Available tools: ask_mcp_docs, search_mcp_docs, get_conversation_history, cache_stats.

🐳 Docker

The only prerequisite is Docker Desktop (which bundles Docker Compose) — you do not need to separately install Python, the pip dependencies, or spacy's model on your machine. All of that happens automatically inside the image when you build it (see the Dockerfile — it runs pip install -r requirements.txt and python -m spacy download en_core_web_sm as build steps).

# 1. Make sure .env exists (same as the local setup, step 3 above)
cp .env.example .env   # then fill in real keys

# 2. Build and run
docker compose up --build

That single command builds the image, installs everything inside it, and starts the container. The data/ folder is mounted as a volume (see docker-compose.yml), so the vector store persists across container restarts — you only pay the slow first-run ingestion cost once, even with Docker.

Server is reachable the same way as running locally: http://localhost:8000/docs.

To stop:

docker compose down

To rebuild after changing code or dependencies:

docker compose up --build

🧪 Testing

Fast smoke tests — no API keys or network calls needed (uses fake embeddings):

pip install pytest
pytest tests/ -v

📊 Offline Evaluation (RAGAS)

Scores the pipeline against 25 hand-written MCP questions with reference answers, using RAGAS:

python scripts/evaluate_ragas.py

This does not need the server running — it builds the pipeline itself (same singleton as main.py/mcp_server.py) and prints a metrics table:

• Faithfulness — is the answer grounded in the retrieved context?

• Response Relevancy — does the answer actually address the question?

• Context Precision — is the retrieved context relevant?

• Context Recall — does retrieved context cover what the reference answer needs?

⏱️ Takes a few minutes: each of the 25 questions runs a real retrieval + generation pass, then every metric is itself scored by an LLM-as-judge call.

⚙️ Configuration Reference

All configuration lives in .env (see .env.example). Key variables:

🛠️ Tech Stack

FastAPI · LangChain · LangGraph · Qdrant · Portkey · Sentence-Transformers · Presidio · NeMo Guardrails · RAGAS · MCP Python SDK · Docker

📄 License

MIT — see LICENSE.