Ollama RAG + MCP Server: Query Your Docs Locally

Goal

Build a local RAG pipeline over a small document collection, then expose it through an MCP server so a connected AI client (Claude Desktop) can query it as a tool.

This mirrors the following architecture: licensed data prepared for AI use, served through a controlled, auditable interface.

Related MCP server: vectorise-mcp

Architecture Overview

Documents (PDF/text)
       |
  [Ingestion Script]
       |
  Chunk + Embed (Ollama embeddings)
       |
  ChromaDB (local vector store)
       |
  [MCP Server]  <-- exposes a `search_documents` tool
       |
  Claude Desktop (MCP client)

Part 1: RAG Pipeline

Stack

• Ollama — local LLM and embedding model runner

• ChromaDB — local vector store, runs in-process, no server needed

• LangChain — orchestration: document loading, chunking, embedding, retrieval

• Python 3.11+

Models to use

• Embeddings: nomic-embed-text via Ollama (fast, good quality, runs well on Mac)

• Generation: qwen2.5:7b via Ollama

Steps

1. Install dependencies

Libraries needed:

• langchain-text-splitters — chunking documents

• langchain-ollama — Ollama embeddings and LLM wrappers

• langchain-chroma — Chroma vector store integration

• langchain-core — shared Document type

• chromadb — local vector store engine

• pypdf — PDF text extraction

pip install langchain-text-splitters langchain-ollama langchain-chroma langchain-core chromadb pypdf
ollama pull nomic-embed-text
ollama pull qwen2.5:7b

2. Prepare documents

Pick 5 to 10 PDFs or Markdown/text files.

Place them in a raw/ folder.

3. Ingest and embed (ingest.py)

What the script does, in order:

1. Clear out any existing chroma_db/ directory so re-running the script doesn't duplicate chunks.

2. Load documents from raw/: read .md files as plain text, and extract text page-by-page from .pdf files, tagging each chunk with its source filename (and page number for PDFs).

3. Split the loaded documents into overlapping chunks (~500 characters, 50 character overlap) so retrieval can return focused excerpts instead of whole documents.

4. Create embeddings for each chunk using the nomic-embed-text model via Ollama.

5. Create a vector store: persist the chunks and their embeddings to disk in chroma_db/ using Chroma.

6. Print how many chunks were ingested.

Always resolve chroma_db/ to an absolute path (relative to the script's own location), since this script and mcp_server.py may be launched from different working directories and must both point at the same store.

4. Query the RAG pipeline (query.py)

What the script does, in order:

1. Create the same embeddings model (nomic-embed-text) used during ingestion — queries and documents must be embedded the same way to be comparable. The embeddings model is used to embed the query.

2. Open the existing vector store from chroma_db/.

3. Create a retriever from the vector store that, given a query, returns the top-k (e.g. 3) most similar chunks.

4. Create the generation model: qwen2.5:7b via Ollama.

5. Build a chain: retriever fetches context for the question → a prompt template combines the question and retrieved context → the LLM generates an answer → the output is parsed to plain text.

6. Invoke the chain with a question and print the answer.

Run ingest.py once to build the vector store, then query.py to test retrieval.

Part 2: MCP Server

Stack

• MCP Python SDK — pip install mcp

• ChromaDB — same vector store built in Part 1

• Transport — stdio (simplest for local development, works with Claude Desktop)

What the server exposes

One tool: search_documents

Input: a natural language query string. Output: the top 3 most relevant document chunks with their source filename and page number.

The MCP server does not call the LLM. It only handles retrieval. The connected AI client (Claude Desktop) does the reasoning over what is returned. This is the correct pattern for auditability and access control.

Steps

1. Install MCP SDK

pip install mcp

2. Build the server (mcp_server.py)

What the script does, in order:

1. Create a FastMCP server instance named rag-server.

2. Create the same embeddings model (nomic-embed-text) used during ingestion.

3. Open the existing vector store from chroma_db/, resolved to an absolute path relative to the script's own location (Claude Desktop may launch this script from an arbitrary working directory, so a relative path would fail or point at the wrong — possibly read-only — location).

4. Register one tool, search_documents, that takes a query string, retrieves the top 3 most similar chunks from the vector store, and returns them formatted with their source filename and page number.

5. Run the server over the stdio transport when executed directly, so Claude Desktop can spawn it as a subprocess and communicate over stdin/stdout.

Note: the server does not call the generation LLM (qwen2.5:7b). It only handles retrieval — the connected AI client (Claude Desktop) does the reasoning over what is returned.

3. Connect to Claude Desktop

Claude Desktop's own config file is at ~/Library/Application Support/Claude/claude_desktop_config.json. Don't hand-edit a copy elsewhere (e.g. under ~/.claude/) — that directory belongs to Claude Code, a separate tool, and is not read by Claude Desktop.

The easiest way to reach the correct file: open Claude Desktop → Settings → Developer → Local MCP servers → Edit Config. This opens the exact file the app reads, avoiding path mix-ups.

Add an mcpServers entry, using the Python interpreter from the project's virtualenv (not a bare python, which may not resolve correctly for a GUI-launched process) and an absolute path to mcp_server.py:

{
  "mcpServers": {
    "rag-server": {
      "command": "/absolute/path/to/venv/bin/python",
      "args": ["/absolute/path/to/mcp_server.py"]
    }
  }
}

Restart Claude Desktop, or restart just the server from Settings → Developer. The search_documents tool will appear and Claude can call it during a conversation.

Milestones

1. Ollama running on local network with nomic-embed-text and qwen2.5:7b pulled.

2. ingest.py runs cleanly and ChromaDB persists to disk.

3. query.py returns sensible answers from your documents.

4. mcp_server.py starts without errors.

5. Claude Desktop connects and can call search_documents interactively.

What to research next

• Chunking strategy: try different chunk sizes and overlaps, or a semantic/markdown-aware splitter instead of fixed character length. Does retrieval quality change?

• Embedding model choice: swap nomic-embed-text for another Ollama embedding model and compare retrieved chunks for the same queries.

• Retrieval evaluation: build a small set of test questions with known expected chunks, and measure how often the retriever actually returns them (precision/recall at k).

• Multiple MCP tools: add a second tool beyond search_documents — e.g. one that lists ingested sources, or one that filters by document/date.

• Transport options: mcp_server.py currently uses stdio. Look into the streamable-http transport and what changes to run the server as a standalone process reachable over the network.

• Re-ranking: after the initial similarity search, add a re-ranking step (cross-encoder or LLM-based) before returning the top chunks.

• Incremental ingestion: right now ingest.py wipes and rebuilds the whole store. Explore adding/updating only changed documents using stable chunk IDs.

• Generation model comparison: swap qwen2.5:7b for other local models and compare answer quality/latency on the same retrieved context.

• Trade-offs worth documenting: local inference latency vs. hosted API speed, chunk size vs. retrieval precision, retrieval-only MCP tool vs. one that also calls the LLM.