MCP RAG Agent

A local-first AI backend that combines Model Context Protocol, Retrieval-Augmented Generation, and an agent runtime to answer questions from a private knowledge base.

The project runs an MCP server that exposes retrieval tools. A client-side agent connects to that server, lets the LLM decide when to call tools, retrieves relevant knowledge chunks, and returns grounded answers with source citations.

Features

MCP server exposing retrieval tools
CLI agent client
Interactive chat mode
RAG pipeline over a local knowledge base
ChromaDB vector store
Pluggable LLM providers (Ollama, AWS Bedrock)
Pluggable embedding providers (Ollama, AWS Bedrock)
Optional reranking pipeline
Source-grounded answers with citations
Modular service layout for retrieval, embeddings, chunking, vector stores, and rerankers

Example Usage

Ask a single question:

uv run python -m client.src.main "Explain how MCP tools work with RAG"

Start interactive mode:

uv run python -m client.src.main --interactive

Example interactive session:

MCP RAG Agent

Type a question and press Enter.
Type 'exit' or 'quit' to stop.

You: Explain how MCP tools work with RAG

Assistant:
MCP tools allow the agent to access external capabilities, such as document retrieval, through a standard protocol...

Sources:
- knowledge/mcp.md#3
- knowledge/rag.md#1

Architecture

flowchart TD
    User[User] --> Agent[LangGraph Agent]
    Agent --> Runtime[Agent Runtime]

    Runtime --> LLM["LLM Provider<br/>(Ollama / Bedrock)"]
    Runtime --> MCPClient[MCP Client]

    MCPClient --> MCPServer[MCP Server]

    MCPServer --> RetrievalTool["retrieve_documents()"]
    MCPServer --> OtherTools["Other MCP Tools"]

    RetrievalTool --> Embed["Embedding Provider<br/>(Ollama / Bedrock)"]
    Embed --> VectorStore[ChromaDB Vector Store]
    VectorStore --> Reranker[Reranker]
    Reranker --> Context[Context Builder]

    Context --> RetrievalTool
    RetrievalTool --> MCPServer
    MCPServer --> MCPClient
    Runtime --> Answer[Grounded Answer with Citations]

How It Works

The user asks a question through the CLI.
The client starts and connects to the MCP server.
The agent discovers available MCP tools.
The LLM decides whether it needs to call a retrieval tool.
The MCP server runs the RAG pipeline:
- embeds the query
- searches ChromaDB
- optionally reranks retrieved chunks
- builds a context payload
- returns source metadata
The agent generates a final answer using the retrieved context.
The final response includes citations such as:

[knowledge/mcp.md#2]

Project Structure

client/
└── src/
    ├── main.py                 # CLI entry point
    ├── agent_runtime.py        # Agent loop and tool-calling runtime
    ├── mcp_client.py           # MCP client wrapper
    ├── tool_executor.py        # Executes MCP tools
    ├── tool_mapper.py          # Converts MCP tools to Ollama tool schema
    └── llm/                    # LLM provider implementations
        ├── base.py
        ├── ollama.py
        └── bedrock.py
        
server/
└── src/
    ├── server.py               # MCP server entry point
    └── tools/                  # MCP tool registrations

services/
├── rag/
├── chunking/                   # Text chunking strategies
├── dto/                        # API response DTOs
├── embedding/                  # Embedding providers
|      ├── base.py
|      ├── ollama_provider.py
|      └── bedrock_provider.py
├── loaders/                    # Document loaders
├── mappers/                    # Domain-to-response mappers
├── models/                     # RAG domain models
├── pipelines/                  # Indexing and retrieval pipelines
├── rerankers/                  # Reranking implementations
└── vectorstores/               # Vector store adapters

knowledge/                      # Local knowledge base documents
scripts/                        # Utility scripts
chroma_db/                      # Local ChromaDB data

Requirements

Required

Python 3.13+
uv

Optional

Ollama (for local inference)
AWS credentials with Bedrock access (for Bedrock providers)

Example:

ollama pull llama3.1

If your embedding provider uses a separate embedding model, pull that model too.

Setup

Install dependencies:

uv sync

Start Ollama:

ollama serve

Run a question:

uv run python -m client.src.main "Explain how MCP tools work with RAG"

Run interactive chat:

uv run python -m client.src.main --interactive

CLI Options

uv run python -m client.src.main --help

Available options:

question              Optional question to ask the agent
-i, --interactive     Start an interactive chat session
--model               Ollama chat model to use
--max-steps           Maximum number of tool-calling steps

Examples:

uv run python -m client.src.main "What is AWS Bedrock?" --model llama3.1

uv run python -m client.src.main --interactive --max-steps 8

Source Citations

The retrieval tool returns source metadata for every retrieved chunk:

{
  "context": "...",
  "sources": [
    {
      "document_id": "knowledge/mcp.md",
      "chunk_index": 2
    }
  ]
}

The agent is instructed to cite retrieved sources in the final answer using this format:

[document_id#chunk_index]

Example:

MCP lets an agent call external tools through a standard protocol, which makes the RAG system easier to separate from the LLM runtime [knowledge/mcp.md#2].

Sources:
- knowledge/mcp.md#2
- knowledge/rag.md#4

Why MCP + RAG?

Traditional RAG systems often tightly couple the agent, retriever, vector store, and application logic. MCP creates a cleaner boundary:

The agent does not need to know how retrieval is implemented.
Retrieval can be exposed as a reusable tool.
Other MCP-compatible clients can use the same backend.
The server can add more tools without rewriting the agent runtime.

Current Retrieval Tool

The MCP server exposes:

retrieve_documents(query: str, top_k: int = 20, top_n: int = 5)

It returns:

retrieved context
document IDs
chunk indexes
vector similarity scores
rerank scores when available

Retrieval Evaluation

Run retrieval evaluation:

uv run python -m scripts.evaluate_retrieval

Run with custom retrieval settings:

uv run python -m scripts.evaluate_retrieval --top-k 8 --top-n 4

Write detailed results to JSON:

uv run python -m scripts.evaluate_retrieval --json-output scripts/eval/results/latest.json

Web API and Browser Demo

Install API dependencies:

 uv add fastapi uvicorn

Run the FastAPI server:

uv run uvicorn api.main:app --reload

Open the browser UI:

[http://localhost:8000](http://localhost:8000)

Open the interactive API docs:

[http://localhost:8000/docs](http://localhost:8000/docs)

API Endpoints

Health check:

curl [http://localhost:8000/health](http://localhost:8000/health)

Non-streaming chat:

curl -X POST [http://localhost:8000/chat](http://localhost:8000/chat)
-H "Content-Type: application/json"
-d '{"message": "Explain how MCP tools work with RAG"}'

Streaming chat:

curl -N -X POST [http://localhost:8000/chat/stream](http://localhost:8000/chat/stream)
-H "Content-Type: application/json"
-d '{"message": "Explain how MCP tools work with RAG"}'

Direct retrieval:

curl -X POST [http://localhost:8000/retrieve](http://localhost:8000/retrieve)
-H "Content-Type: application/json"
-d '{"query": "What is Model Context Protocol?", "top_k": 8, "top_n": 4}'

List available MCP tools:

curl [http://localhost:8000/tools](http://localhost:8000/tools)

The browser UI uses the streaming endpoint so the answer appears token-by-token while the final response is generated.

Configuration

Chat generation and embeddings are configured independently, allowing combinations such as Bedrock + Ollama or Ollama + Bedrock.

Create a local .env file from the example:

cp .env.example .env

Use Ollama for chat and embeddings

CLIENT_LLM_PROVIDER=ollama OLLAMA_CHAT_MODEL=llama3.1 RAG_EMBEDDING_PROVIDER=ollama OLLAMA_EMBEDDING_MODEL=nomic-embed-text

Use Bedrock for chat and Ollama for embeddings

CLIENT_LLM_PROVIDER=bedrock BEDROCK_CHAT_MODEL=anthropic.claude-3-5-sonnet-20240620-v1:0 AWS_REGION=us-east-1
RAG_EMBEDDING_PROVIDER=ollama OLLAMA_EMBEDDING_MODEL=nomic-embed-text

Use Bedrock for chat and embeddings

CLIENT_LLM_PROVIDER=bedrock BEDROCK_CHAT_MODEL=anthropic.claude-3-5-sonnet-20240620-v1:0
RAG_EMBEDDING_PROVIDER=bedrock BEDROCK_EMBEDDING_MODEL=amazon.titan-embed-text-v2:0
AWS_REGION=us-east-1

If you switch embedding providers, rebuild or re-index the vector database because vectors from different embedding models are not compatible.

Architecture Decisions

MCP defines the boundary between the agent runtime and backend capabilities.
LangGraph orchestrates workflows but does not implement retrieval logic.
Retrieval is exposed as an MCP tool, allowing any MCP-compatible client to reuse the backend.
LLM providers are swappable through a common interface (Ollama or AWS Bedrock).
Embedding providers are independent of the chat model.
Vector stores are abstracted behind a common interface (ChromaDB by default).
Retrieval results are exposed as DTOs to avoid leaking internal domain models through the MCP API.

Portfolio Highlights

This project demonstrates:

MCP tool design
RAG architecture
Provider abstraction (Ollama / Bedrock)
vector search
source-grounded generation
agent tool-calling loops
modular backend design
Python async programming
clean CLI UX

Future Improvements

Planned improvements:

Docker Compose setup
streaming responses
structured logging
query rewriting
hybrid keyword/vector retrieval
automated tests
CI workflow

MCP RAG Agent

MCP RAG Agent

Features

Example Usage

Architecture

How It Works

Project Structure

Requirements

Setup

CLI Options

Source Citations

Why MCP + RAG?

Current Retrieval Tool

Retrieval Evaluation

Web API and Browser Demo

API Endpoints

Configuration

Use Ollama for chat and embeddings

Use Bedrock for chat and Ollama for embeddings

Use Bedrock for chat and embeddings

Architecture Decisions

Portfolio Highlights

Future Improvements

Latest Blog Posts

MCP directory API