Skip to main content
Glama

šŸ”’ Privacy-Preserving Local RAG with MCP Integration

A production-grade, fully local Retrieval-Augmented Generation (RAG) system integrated with the Model Context Protocol (MCP). Zero data ever leaves your machine.

Architecture

User CLI  →  Agent Orchestrator  →  MCP Client (stdio)
                                          ↓
                               ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”
                               │     MCP Server        │
                               │  ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”  │
                               │  │  vector_search  │  │
                               │  │  local_db_query │  │
                               │  ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜  │
                               ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜
                                    ↓           ↓
                               Qdrant DB    SQLite DB
                               (Docker)   (employees)

Related MCP server: OpenRAG MCP Server

Tech Stack

Component

Technology

LLM Inference

Ollama (Llama 3)

Embeddings

Ollama (nomic-embed-text, 768d)

Vector Store

Qdrant (Docker)

Re-ranking

Cross-Encoder (ms-marco-MiniLM-L-6-v2)

MCP Server

FastMCP (Python SDK v1.x)

Chunking

SemanticChunker (LangChain Experimental)

Config

Pydantic Settings

Logging

structlog (JSON structured)

Prerequisites

  • Python 3.11+

  • Docker & Docker Compose

  • Ollama installed and running

  • At least 8GB RAM (16GB recommended)

Quick Start

1. Clone & Configure

git clone <repo-url>
cd local-rag-mcp
cp .env.example .env

2. Start Qdrant

docker compose up -d
# Verify: open http://localhost:6333/dashboard

3. Pull Ollama Models

# On Linux/macOS:
bash scripts/pull_models.sh

# On Windows (PowerShell):
ollama pull llama3
ollama pull nomic-embed-text

4. Create Python Environment

python -m venv .venv

# Linux/macOS:
source .venv/bin/activate

# Windows:
.venv\Scripts\activate

pip install -r requirements.txt

5. Seed the SQLite Database

python scripts/seed_database.py

6. Ingest Documents

python main.py
# Inside the REPL:
> /ingest data/documents

7. Run the Agent

python main.py
> What does our documentation say about authentication?
> List all employees in the Engineering department
> How many engineers earn above the department average?

Project Structure

local-rag-mcp/
ā”œā”€ā”€ docker-compose.yml       # Qdrant container
ā”œā”€ā”€ .env.example             # Config template
ā”œā”€ā”€ requirements.txt         # Pinned deps
ā”œā”€ā”€ pyproject.toml           # Project metadata & tool config
│
ā”œā”€ā”€ config/
│   └── settings.py          # Centralized Pydantic config
│
ā”œā”€ā”€ data/
│   ā”œā”€ā”€ documents/           # Drop PDFs/Markdown here
│   └── sqlite/
│       └── employees.db     # Auto-seeded SQLite DB
│
ā”œā”€ā”€ scripts/
│   ā”œā”€ā”€ seed_database.py     # Seeds employee DB
│   └── pull_models.sh       # Ollama model helper
│
ā”œā”€ā”€ src/
│   ā”œā”€ā”€ ingestion/           # PDF/MD loader, semantic chunker, embedder
│   ā”œā”€ā”€ mcp_server/          # FastMCP server + vector_search + local_db_query tools
│   ā”œā”€ā”€ agent/               # Agent loop + HyDE
│   └── utils/               # Structured logging
│
ā”œā”€ā”€ tests/                   # Pytest test suite
└── main.py                  # CLI REPL entry point

CLI Commands

Command

Description

/ingest <path>

Ingest all PDFs/Markdown from directory

/hyde on|off

Toggle HyDE query enhancement

/help

Show available commands

/quit

Exit the application

Any other text

Ask the agent a question

Configuration

All settings are controlled via .env. Key variables:

Variable

Default

Description

OLLAMA_LLM_MODEL

llama3

LLM for generation & tool calling

OLLAMA_EMBEDDING_MODEL

nomic-embed-text

Embedding model

QDRANT_COLLECTION_NAME

rag_documents

Qdrant collection name

HYDE_ENABLED

true

Enable HyDE query enhancement

RERANKER_TOP_K

5

Number of final results after re-ranking

Running Tests

pytest tests/ -v

License

MIT

A
license - permissive license
-
quality - not tested
C
maintenance

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Orax0001/rag-agent-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server