MCP FastAPI Multi-Agent RAG System

A production-grade AI assistant system that combines retrieval-augmented generation (RAG), multi-agent orchestration, and the Model Context Protocol (MCP) to answer queries across documents, calculations, and live data.

What This System Does

This architecture deploys an agentic system that decides which tool to use for a given query — similar to how a researcher might choose between consulting a reference library (documents), running calculations, or checking live weather data. The system ingests PDF documents, embeds them into a vector database, and serves a ReAct agent via FastAPI that can retrieve document context, perform arithmetic, fetch weather, and persist conversation history to SQLite.

Related MCP server: personal-notes-assistant

Architecture Overview

User Query (FastAPI)
    ↓
ReAct Agent (LangGraph + Ollama qwen2.5)
    ├─→ [Tool 1] Calculator (numexpr)
    ├─→ [Tool 2] Weather (wttr.in API)
    └─→ [Tool 3] Hybrid Retrieval (ChromaDB + BM25)
              ├─ Dense Search (SentenceTransformer embeddings)
              └─ Sparse Search (BM25 keyword ranking)
                   ↓
    Response Persisted to SQLite

Key Components

Hybrid Retrieval — Combines dense semantic search (ChromaDB embeddings) and sparse lexical search (BM25 token scoring) via reciprocal rank fusion (RRF). Think of it as asking both "What does this mean semantically?" and "What keywords appear here?" and trusting neither alone.

MCP Tools — The Model Context Protocol lets the LLM invoke tools declaratively. Each tool (Calculator, Weather, Hybrid_Retrival) is registered with explicit docstrings that guide the agent on when to use it.

LangGraph ReAct Agent — Implements the Reasoning + Acting loop: the model thinks through which tool fits, invokes it, observes the result, and iterates until it produces a final response.

Prerequisites

• Python 3.10+

• Ollama installed locally with qwen2.5:1.5b model pulled

• SQLite (included with Python)

Installation & Setup

1. Clone and Install Dependencies

git clone <repo-url>
cd MCP\ FastAPI\ System
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -r requirements.txt

2. Prepare Documents

Place your PDF files in the project root. The default is Why_Language_Models_Hallucinate_Explainer.pdf. The system will:

• Load and chunk the PDF

• Generate MD5 IDs for each chunk

• Embed chunks with all-MiniLM-L6-v2 (SentenceTransformer)

• Store in ChromaDB (persists to MCP_DB/ directory)

3. Update File Paths

Edit MCP_Client.py and MCP_Server.py to match your environment:

# Example: Update the absolute path to MCP_Server.py
"args": ["/your/path/to/MCP_Server.py"],

Running the System

Option 1: FastAPI Server (Recommended for Production)

uvicorn main:app --reload --host 0.0.0.0 --port 8000

Access the interactive docs at http://localhost:8000/docs.

Option 2: Standalone Client

python MCP_Client.py

This runs an async query directly without the HTTP wrapper.

Option 3: Jupyter Notebook

Run ALL_to_gether.ipynb cell by cell for development and debugging.

API Endpoints

POST /chat — Ask a Question

{
  "Question": "What causes hallucination in language models?",
  "limit": 50
}

Response:

{
  "id": 1,
  "question": "What causes hallucination in language models?",
  "limit": 50,
  "answer": "<Agent's response>"
}

GET /ask — Retrieve All Queries

Returns all stored questions and answers from the database.

PUT /ask — Update a Query

{
  "ID": 1,
  "Question": "Updated question",
  "limit": 75
}

DELETE /ask — Delete a Query

{
  "ID": 1
}

How It Works: A Concrete Example

User Query: "What is 25 * 8?"

1. FastAPI receives the question and invokes MCP_Client.main().

2. MCP Client connects to the MCP Server (subprocess over stdio).

3. Server exposes three tools; the ReAct Agent receives them.

4. Agent reads the query, sees numeric operators, and calls the Calculator tool with "25 * 8".

5. Calculator uses numexpr.evaluate() to return 200.

6. Agent formats the response: "200".

7. FastAPI stores the Q&A in SQLite and returns JSON.

Configuration

Model Selection

Edit the model name in MCP_Server.py or MCP_Client.py:

ollama = ChatOllama(model="qwen2.5:1.5b")  # Change to qwen2.5:7b for higher accuracy

Retrieval Parameters

Adjust in MCP_Server.py:

• chunk_size=1500 — Size of each document chunk

• chunk_overlap=180 — Overlap between consecutive chunks

• threshold=1.0 — Distance threshold for dense retrieval

• k=10 — Number of BM25 results to consider

RRF Fusion

Tune the rrf_tokes calculation (currently using rank+60 denominator) to weight dense vs. sparse results differently.

Project Structure

MCP FastAPI System/
├── MCP_Server.py          # Exposes Calculator, Weather, Hybrid_Retrival tools
├── MCP_Client.py          # Connects to server, builds ReAct agent, runs queries
├── main.py                # FastAPI app entry point
├── Router.py              # API endpoints (POST /chat, GET /ask, etc.)
├── DataBase.py            # SQLAlchemy models and session management
├── ALL_to_gether.ipynb    # Development notebook (client + server in one)
├── requirements.txt       # Dependencies
├── SQL.db                 # SQLite database (auto-created)
├── MCP_DB/                # ChromaDB persistence (auto-created)
└── Why_Language_Models_Hallucinate_Explainer.pdf

Key Design Decisions

1. Ollama (Local LLM) — Runs inference locally; no API costs or latency bottlenecks.

2. Hybrid Retrieval — Single-modality (dense-only) search misses keyword-heavy queries; hybrid fusion catches both semantic and lexical matches.

3. Tool Docstrings — Each tool has explicit, detailed instructions. The agent reads these to decide when to invoke each tool.

4. SQLite Persistence — Simple, file-based; suitable for small to medium deployments. Swap for PostgreSQL for horizontal scaling.

5. MCP over Stdio — Server runs as a subprocess; communication happens via stdin/stdout, ensuring isolation and easy debugging.

Limitations & Future Work

• Ollama Availability — System assumes a running Ollama instance. Graceful fallback to remote endpoints (e.g., Groq, Anthropic) planned.

• Single PDF — Currently loads one PDF; extend to multiple documents via a document registry.

• No Caching — Repeated queries re-run full RAG pipeline; Redis cache layer recommended for production.

• Tool Routing Logic — Relies on LLM judgment; adversarial queries may misroute. Add explicit regex pre-checks for calculator vs. general queries.

Troubleshooting

"MCP adapter working" doesn't print → MCP server subprocess failed. Check:

python /path/to/MCP_Server.py

ChromaDB errors → Delete MCP_DB/ and rerun to rebuild embeddings.

Ollama model not found → Pull the model:

ollama pull qwen2.5:1.5b

Tool not invoked → Check system prompt in MCP_Client.py. Agent follows tool docstrings; ensure they are clear and unambiguous.

Contributing

Fork, experiment, and refactor. The system is designed for extension: add new tools by decorating functions with @mcp.tool() in MCP_Server.py, then rebuild the agent.