Skip to main content
Glama
rag-mcp-agent-demo

rag-mcp-agent-demo

by khovala

RAG + MCP Agent Demo

A compact, dependency-free reference implementation of the core pieces behind a production LLM/agent stack:

  • a RAG pipeline (embed → store → retrieve → assemble context),

  • an MCP server exposing read-only tools over stdio (JSON-RPC 2.0),

  • a Text-to-SQL safety gate that validates generated SQL before execution,

  • an offline evaluation harness (retrieval recall@k + SQL-guard accuracy),

  • and a small agent loop that routes between tools.

This is a sanitized, self-contained demo. It contains no employer code or data.

Why this exists

Most "RAG demos" show a notebook that retrieves a chunk. Production systems need the boring parts: a swappable vector store, a hard safety boundary in front of the database, and an evaluation harness that fails CI when quality regresses. This repo isolates those parts so they can be read, tested and reused.

Related MCP server: shop-sql-mcp

Architecture

                ┌─────────────┐
   question ───▶│   Agent     │  (plan → tool → observe → answer)
                └──────┬──────┘
             ┌─────────┴─────────┐
             ▼                   ▼
      ┌────────────┐      ┌──────────────┐
      │ RAG        │      │ SQL tool     │
      │ embed→store│      │  guard()     │  ← blocks DML/DDL, enforces LIMIT
      │ →retrieve  │      └──────┬───────┘
      └─────┬──────┘             ▼
            │               ┌──────────┐
            ▼               │ SQLite   │
      ┌──────────┐          └──────────┘
      │ corpus   │
      └──────────┘

      MCP server (stdio, JSON-RPC 2.0)
      tools: search_docs · run_sql · list_tables

Quickstart

No third-party runtime dependencies (Python 3.10+).

# run the offline evaluation harness
python eval/run_eval.py

# ask the agent a question
PYTHONPATH=src python -m rag_mcp_demo.cli "how many customers are there"
PYTHONPATH=src python -m rag_mcp_demo.cli "how does the RAG pipeline retrieve chunks"

# run the MCP server over stdio
PYTHONPATH=src python -m rag_mcp_demo.mcp_server

Install as a package (optional):

pip install -e ".[dev]"
rag-agent "total paid revenue"
pytest

Example: MCP over stdio

echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | PYTHONPATH=src python -m rag_mcp_demo.mcp_server

Evaluation

eval/run_eval.py reports two signals and exits non-zero if either drops below 0.8:

Metric

Meaning

retrieval recall@3

fraction of golden questions whose expected document is in the top-3

sql guard accuracy

fraction of safe/unsafe SQL cases classified correctly

Design decisions

  1. Vector store behind an interface — in-memory for tests, Qdrant/pgvector in production.

  2. Safety before execution — the SQL gate rejects non-SELECT statements and enforces a row limit; in production this is built on a real parser (e.g. sqlglot).

  3. Tools are read-only by default — the MCP server exposes only read operations.

  4. Evals are first-class — a golden set and a harness that fails CI on regression.

  5. LLM-agnostic agent — the router is heuristic so the control flow is testable without a model; swap in an LLM to replace Agent.route().

Project layout

src/rag_mcp_demo/
  embeddings.py     # TF-IDF embedder + cosine (swap for a real model)
  vector_store.py   # in-memory vector store
  rag.py            # corpus loading + retrieval pipeline
  sql_guard.py      # pre-execution SQL safety gate
  db.py             # sample SQLite database
  agent.py          # minimal agent loop
  mcp_server.py     # MCP server (stdio, JSON-RPC 2.0)
  cli.py            # `rag-agent` CLI
eval/               # golden set + evaluation harness
tests/              # pytest suite
data/corpus.jsonl   # sample documents

License

MIT — see LICENSE.

Available Tools

3 tools
list_tablesA

List tables in the sample database (read-only).

Inputs: no named parameters.

run_sqlA

Run a read-only SQL SELECT against the sample database.

Inputs: sql (required).

search_docsC

Search the documentation corpus (read-only).

Inputs: query (required), top_k.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.7/5.0

    Scored across 3 tools

    Disambiguation5/5

    Each tool has a clearly distinct purpose: search_docs targets the documentation corpus, run_sql executes SQL queries, and list_tables lists database tables. There is no overlap or ambiguity between them.

    Naming Consistency5/5

    All tool names follow a consistent verb_noun snake_case pattern: search_docs, run_sql, list_tables. The naming convention is uniform and predictable.

    Tool Count5/5

    With 3 tools, the set is well-scoped for a demo RAG agent that combines documentation search and read-only database exploration. Each tool serves a distinct purpose and none feel redundant or excessive.

    Completeness4/5

    The core workflows of searching docs and querying the database are covered, but there is a minor gap: list_tables only lists table names without exposing schema details, and there is no tool to retrieve full document content beyond search results. Still, the surface is adequate for a demo.

    Maintenance

    ActivityStale
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      Exposes SQLite database query tools and markdown document resources over JSON-RPC 2.0 stdio transport, enabling AI assistants to read and search documents and execute read-only SQL queries.
      8
      1
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      Provides AI agents read-only analytical access to a SQLite database over stdio, with tools for listing tables, describing schemas, and running paginated SQL queries.
      -
    • F
      license
      A
      quality
      C
      maintenance
      Enables safe read-only analysis of a local SQLite e-commerce database over stdio, providing tools for schema inspection and parameterized SELECT queries.
      2
      -
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables LLM agents to run read-only SQL queries and inspect database schema over stdio by default, with optional opt-in DML/DDL and PL/SQL execution that auto-commits.
      MIT