mcp-rag-demo

A small, read-only MCP server over a RAG index (Postgres + pgvector, local embeddings), with provenance on every result and a retrieval evaluation you can re-run.

The corpus is 8 sanitized Architecture Decision Records (ADRs, written in Spanish) from a WhatsApp appointment-booking assistant built on n8n. It is a deliberately small demo: the goal is to show the design decisions (fail-closed index, read-only access, untrusted-corpus handling, measurable retrieval quality), not scale.

Status: working demo. 63 tests pass; retrieval was evaluated on 18 questions (see Evaluation and its limits). Not production software.

Architecture

flowchart LR
    C["corpus/*.md<br/>8 ADRs"] -->|"chunk by headings"| I["ingest.py"]
    I -->|"local embeddings<br/>fastembed ONNX"| P[("Postgres 16<br/>+ pgvector HNSW")]
    P -->|"read-only session"| S["search.py"]
    S --> M["server.py<br/>MCP, stdio"]
    S --> A["ask.py CLI"]
    S --> E["evaluate.py<br/>hit-at-k, MRR, threshold sweep"]
    M -->|"search_docs / get_doc / list_docs"| K["MCP client<br/>Claude Desktop, Claude Code, ..."]

Related MCP server: LTM MCP Server

Design decisions

Quick start

Requirements: Python 3.10+, Docker. First run downloads the embedding model (~0.64 GB).

docker compose up -d                       # Postgres + pgvector on 127.0.0.1:5433
py -3 -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -e ".[mcp]"
python -m ragdemo.ingest --corpus corpus   # first run: add --reset if you changed model
python -m ragdemo.ask "¿Con qué algoritmo se cifran los teléfonos?" -k 3
python scripts\smoke_mcp.py                # optional: exercises the MCP server as a real client

Expected: the top hit is ADR-016-cifrado-de-campo-postgres with its heading and a score.

(On Linux/macOS use python3 -m venv .venv && source .venv/bin/activate.)

Use it from an MCP client

Add this to your client's MCP configuration (path to the venv's Python must be absolute):

{
  "mcpServers": {
    "ragdemo": {
      "command": "C:\\path\\to\\mcp-rag-demo\\.venv\\Scripts\\python.exe",
      "args": ["-m", "ragdemo.server"],
      "env": {
        "RAGDEMO_DSN": "postgresql://ragdemo:ragdemo@127.0.0.1:5433/ragdemo",
        "RAGDEMO_MIN_SCORE": "0.30"
      }
    }
  }
}

Tools: search_docs(query, k=5), get_doc(doc_id), list_docs(). The server is stdio-based: stdout is the protocol, all logs go to stderr. It exits with an error if there is no index or the model does not match.

Evaluation

python -m ragdemo.evaluate --k 3 --json eval\report-jina.json

18 questions in eval/questions.jsonl: 14 answerable (12 Spanish, 2 English) with the expected ADR, and 4 out of scope. Metrics are at document level (did an expected ADR appear in the top-k?). Result with jinaai/jina-embeddings-v2-base-es, run on 2026-11-12:

How to read this honestly

• Perfect scores on 14 questions show the pipeline works; they do not rank it against alternatives.

• The questions were written by the corpus author after reading the ADRs, so they share vocabulary with the documents. Real users phrase things worse. A short query ("dos agentes") scored 0.382 in a manual check, well below the long-form scores.

• The 4 out-of-scope questions are unrelated to the domain (recipes, geography, stock prices). There are no hard negatives (on-topic questions the corpus does not answer), so the 0.30 threshold is untested where it matters most.

• Only one embedding model was evaluated.

Tests

docker compose exec db createdb -U ragdemo ragdemo_test
$env:RAGDEMO_TEST_DSN = "postgresql://ragdemo:ragdemo@127.0.0.1:5433/ragdemo_test"
pip install pytest
pytest -q          # 63 tests

Without RAGDEMO_TEST_DSN, tests that need a database are skipped (on Linux/macOS with Python < 3.13, pip install -e ".[dev]" brings an embedded Postgres via pgserver). Tests use a bag-of-words hash embedder, so they exercise the pipeline, not semantic quality.

Known limits

• Tiny corpus (8 documents, ~70 chunks); no re-ranking, no hybrid (keyword) search.

• Chunking is by Markdown headings with a character cap; no token-aware splitting.

• The stdio server is tested end-to-end with the official MCP Python client: in the test suite (hash embedder) and with the real model via scripts/smoke_mcp.py (Windows, Python 3.14). It has not been verified with a desktop MCP client such as Claude Desktop.

• The corpus was sanitized by rules plus manual review (names, phone numbers, client names). Indirect identifiers can survive any automated pass.

• Spanish-first: the default model is Spanish-oriented; English queries work (see q13, q14) but were only lightly evaluated.

License

MIT.