MCP-Powered AI Assistant (Local — LlamaIndex + Ollama)

Privacy-first document intelligence: all models run locally via Ollama — no data leaves your machine.

What is MCP?

Model Context Protocol (MCP) is an open standard (by Anthropic) that defines how AI models discover and invoke tools at runtime. Think of it as a "USB-C port for AI" — any MCP-compatible client (Claude Desktop, your own agent, etc.) can connect to any MCP server and immediately use its tools.

MCP vs. Advanced RAG — What's the Difference?

Dimension	Advanced RAG	MCP
Purpose	Improve retrieval accuracy	Standardise tool/capability exposure
Core idea	Better chunking, re-ranking, hybrid search	JSON-RPC tool registry with discovery
What the LLM gets	Retrieved context injected into prompt	A menu of callable functions with schemas
Execution	Single pipeline (query → retrieve → generate)	Multi-step agent loop (plan → pick tool → call → observe → repeat)
Tools	Retrieval only	Any function: retrieval, APIs, databases, code
State	Stateless per query	Stateful agent sessions possible
This project	RAG is one tool inside the MCP server	MCP wraps 8 RAG tools, discoverable at runtime

In short: Advanced RAG makes retrieval smarter. MCP makes the entire AI system composable and interoperable.

Project Architecture

mcp_rag_assistant/
├── config.py               ← Central config (LLM, embed, chunking, server)
├── rag_engine.py           ← LlamaIndex: load docs → build index → query engine
├── main.py                 ← CLI entrypoint (serve / index / query / demo)
├── mcp_client.py           ← Example client that calls server tools
│
├── mcp_server/
│   └── server.py           ← HTTP JSON-RPC server exposing all tools
│
├── tools/
│   └── rag_tools.py        ← 8 MCP tool implementations
│
├── utils/
│   └── logger.py           ← Structured logging
│
├── my_data/                ← ⬅ DROP YOUR FILES HERE (PDF, DOCX, XLSX, CSV)
├── storage/                ← ChromaDB persistence (auto-created)
├── logs/                   ← Log files (auto-created)
│
├── requirements.txt
├── .env.example
├── .gitignore
└── README.md

Data Flow

User Query
    │
    ▼
MCP Client (mcp_client.py or Claude Desktop or your agent)
    │  JSON-RPC POST /mcp  {"method": "tools/call", "params": {...}}
    ▼
MCP Server (mcp_server/server.py)
    │  dispatches to matching tool function
    ▼
Tool Function (tools/rag_tools.py)
    │  calls get_query_engine().query(...)
    ▼
LlamaIndex Query Engine (rag_engine.py)
    │  embeds query with qwen3-embedding:0.6b via Ollama
    ▼
ChromaDB Vector Store
    │  returns top-K similar chunks
    ▼
Ollama LLM (llama3 or mistral)
    │  synthesises answer from retrieved context
    ▼
JSON response back through MCP → Client

Available MCP Tools

Tool	Description
`query_documents`	General Q&A over all indexed documents
`list_indexed_files`	Show files in `my_data/`
`rebuild_index`	Re-index after adding/removing files
`summarize_document`	Summarise a specific file by name
`analyze_data`	Plain-English data analysis (CSV/XLSX)
`generate_report`	Generate summary / detailed / executive report
`compare_documents`	Compare two documents on a given aspect
`extract_entities`	Extract people, orgs, dates, numbers

Prerequisites

Python 3.10+
Ollama running locally — ollama.com
Models already pulled (you have these):
- llama3:latest
- mistral:latest
- qwen3-embedding:0.6b

Setup

# 1. Clone / unzip the project
cd mcp_rag_assistant

# 2. Create virtual environment
python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Configure (optional — defaults work out of the box)
cp .env.example .env
# Edit .env to change models, ports, chunk sizes etc.

# 5. Add your documents
#    Copy PDFs, DOCX, XLSX, CSV files into:
#    my_data/

# 6. Build the index
python main.py index

# 7. Start the MCP server
python main.py serve

Usage

Start the server

python main.py serve
# MCP server listening on http://0.0.0.0:8080

One-shot query (no server needed)

python main.py query "What are the key findings in the Q1 report?"

Run the demo client (server must be running)

# In a second terminal:
python main.py demo

Rebuild index after adding new files

python main.py index
# or via MCP tool:
# call rebuild_index tool from any client

Health check

GET http://localhost:8080/health
GET http://localhost:8080/tools

Switching Models

Edit config.py or your .env:

# Use mistral instead of llama3
LLM_MODEL=mistral:latest

# Use nomic-embed-text for embeddings
EMBED_MODEL=nomic-embed-text:latest

Tuning Chunk Size

In config.py or .env:

Setting	Default	Notes
`CHUNK_SIZE`	256	Tokens per chunk. Smaller = more precise retrieval
`CHUNK_OVERLAP`	25	Overlap between chunks. Helps preserve context at boundaries
`SIMILARITY_TOP_K`	5	Chunks retrieved per query
`RESPONSE_MODE`	compact	`compact` \| `tree_summarize` \| `refine`