Skip to main content
Glama
khovala

rag-mcp-agent-demo

by khovala
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

RAG + MCP Agent Demo

A compact, dependency-free reference implementation of the core pieces behind a production LLM/agent stack:

  • a RAG pipeline (embed → store → retrieve → assemble context),

  • an MCP server exposing read-only tools over stdio (JSON-RPC 2.0),

  • a Text-to-SQL safety gate that validates generated SQL before execution,

  • an offline evaluation harness (retrieval recall@k + SQL-guard accuracy),

  • and a small agent loop that routes between tools.

This is a sanitized, self-contained demo. It contains no employer code or data.

Why this exists

Most "RAG demos" show a notebook that retrieves a chunk. Production systems need the boring parts: a swappable vector store, a hard safety boundary in front of the database, and an evaluation harness that fails CI when quality regresses. This repo isolates those parts so they can be read, tested and reused.

Related MCP server: mcpserve-py

Architecture

                ┌─────────────┐
   question ───▶│   Agent     │  (plan → tool → observe → answer)
                └──────┬──────┘
             ┌─────────┴─────────┐
             ▼                   ▼
      ┌────────────┐      ┌──────────────┐
      │ RAG        │      │ SQL tool     │
      │ embed→store│      │  guard()     │  ← blocks DML/DDL, enforces LIMIT
      │ →retrieve  │      └──────┬───────┘
      └─────┬──────┘             ▼
            │               ┌──────────┐
            ▼               │ SQLite   │
      ┌──────────┐          └──────────┘
      │ corpus   │
      └──────────┘

      MCP server (stdio, JSON-RPC 2.0)
      tools: search_docs · run_sql · list_tables

Quickstart

No third-party runtime dependencies (Python 3.10+).

# run the offline evaluation harness
python eval/run_eval.py

# ask the agent a question
PYTHONPATH=src python -m rag_mcp_demo.cli "how many customers are there"
PYTHONPATH=src python -m rag_mcp_demo.cli "how does the RAG pipeline retrieve chunks"

# run the MCP server over stdio
PYTHONPATH=src python -m rag_mcp_demo.mcp_server

Install as a package (optional):

pip install -e ".[dev]"
rag-agent "total paid revenue"
pytest

Example: MCP over stdio

echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | PYTHONPATH=src python -m rag_mcp_demo.mcp_server

Evaluation

eval/run_eval.py reports two signals and exits non-zero if either drops below 0.8:

Metric

Meaning

retrieval recall@3

fraction of golden questions whose expected document is in the top-3

sql guard accuracy

fraction of safe/unsafe SQL cases classified correctly

Design decisions

  1. Vector store behind an interface — in-memory for tests, Qdrant/pgvector in production.

  2. Safety before execution — the SQL gate rejects non-SELECT statements and enforces a row limit; in production this is built on a real parser (e.g. sqlglot).

  3. Tools are read-only by default — the MCP server exposes only read operations.

  4. Evals are first-class — a golden set and a harness that fails CI on regression.

  5. LLM-agnostic agent — the router is heuristic so the control flow is testable without a model; swap in an LLM to replace Agent.route().

Project layout

src/rag_mcp_demo/
  embeddings.py     # TF-IDF embedder + cosine (swap for a real model)
  vector_store.py   # in-memory vector store
  rag.py            # corpus loading + retrieval pipeline
  sql_guard.py      # pre-execution SQL safety gate
  db.py             # sample SQLite database
  agent.py          # minimal agent loop
  mcp_server.py     # MCP server (stdio, JSON-RPC 2.0)
  cli.py            # `rag-agent` CLI
eval/               # golden set + evaluation harness
tests/              # pytest suite
data/corpus.jsonl   # sample documents

License

MIT — see LICENSE.

Available Tools

3 tools
list_tablesA

List tables in the sample database (read-only).

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden of behavioral disclosure. It usefully states 'read-only', ruling out side effects, but it does not describe what is returned (e.g., just names vs. full metadata) or any ordering/limit behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One front-loaded sentence with no filler. The parenthetical 'read-only' adds a valuable safety signal without bloating the description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter, read-only listing tool, the description covers purpose, scope, and safety. The absence of an output schema is mitigated because 'list tables' implies the return value, though a note on return format would make it fully complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the baseline of 4 applies. The description adds the scope 'sample database', which is the only contextual qualifier needed; there are no parameter semantics to clarify.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb 'List' and names the resource 'tables', scoped to 'the sample database'. This clearly distinguishes it from sibling tools like search_docs and run_sql, whose purposes are visibly different.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit guidance about when to use list_tables versus search_docs or run_sql, and no exclusions are stated. The intended use is only implied by the action itself, making this adequate but not instructive.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_sqlA

Run a read-only SQL SELECT against the sample database.

ParametersJSON Schema
NameRequiredDescriptionDefault
sqlYes

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden of behavioral disclosure. Stating 'read-only' is a meaningful safety signal, but the description does not mention result format, error behavior, query limits, or whether the database state is ever affected.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, clear sentence that front-loads the action and the read-only constraint. Every word contributes meaning and there is no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter tool, the description is minimally adequate for invocation, but with no output schema and no annotations it leaves the agent to infer the return shape and operational details. It could be stronger for a tool that executes arbitrary SQL.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides no description for the single 'sql' parameter, but the tool description specifies it should be a SQL SELECT query. This adds critical semantic meaning beyond the bare string type, though it does not cover syntax details or constraints.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('Run a read-only SQL SELECT') and a clear resource ('the sample database'). It does not explicitly differentiate from sibling tools like search_docs or list_tables, but the SQL-specific verb makes the purpose reasonably unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use when an agent needs to execute a SQL SELECT query against the sample database, but it provides no explicit guidance about when to prefer this over search_docs or list_tables, nor any exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_docsC

Search the documentation corpus (read-only).

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYes
top_kNo

TDQS

C2.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the burden of behavioral disclosure, and it does contribute the useful fact that the operation is 'read-only'. However, it does not describe what results look like, whether top_k limits results, or any other runtime behavior, so transparency is only partial.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence with no filler; the core purpose and read-only nature are stated efficiently. It is somewhat under-specified, but concisely structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and no annotations, the minimal description leaves several things an agent would need to know unstated: what results are returned, how top_k affects the search, and when to choose this tool over run_sql/list_tables. This is incomplete for a tool with two parameters and no structured schema support.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not explain the query parameter or the top_k parameter beyond the names in the schema. 'query' is mildly self-explanatory, but 'top_k' and its default behavior are not addressed, so the description fails to compensate for the missing schema docs.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states a specific verb ('Search') and resource ('documentation corpus'), and adds the safety qualifier 'read-only'. It distinguishes from the sibling tools run_sql and list_tables by the object being searched, though it does not explicitly name them or contrast their purposes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no guidance on when to use search_docs versus run_sql or list_tables, and it does not mention any exclusions or context. While 'documentation corpus' implies a docs search scenario, no explicit alternative or condition is provided, which is effectively no guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 3 tool updatesv0.1.0
    • First observedlist_tables
    • First observedrun_sql
    • First observedsearch_docs
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.7/5.0

Scored across 3 tools

Disambiguation5/5

Each tool has a clearly distinct purpose: search_docs targets the documentation corpus, run_sql executes SQL queries, and list_tables lists database tables. There is no overlap or ambiguity between them.

Naming Consistency5/5

All tool names follow a consistent verb_noun snake_case pattern: search_docs, run_sql, list_tables. The naming convention is uniform and predictable.

Tool Count5/5

With 3 tools, the set is well-scoped for a demo RAG agent that combines documentation search and read-only database exploration. Each tool serves a distinct purpose and none feel redundant or excessive.

Completeness4/5

The core workflows of searching docs and querying the database are covered, but there is a minor gap: list_tables only lists table names without exposing schema details, and there is no tool to retrieve full document content beyond search results. Still, the surface is adequate for a demo.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • mcpserve-py

    devaloi
    A
    license
    A
    quality
    D
    maintenance
    Exposes SQLite database query tools and markdown document resources over JSON-RPC 2.0 stdio transport, enabling AI assistants to read and search documents and execute read-only SQL queries.
    8
    1
    MIT

  • shop-sql-mcp

    lampmaster
    F
    license
    Not graded
    quality
    C
    maintenance
    Provides AI agents read-only analytical access to a SQLite database over stdio, with tools for listing tables, describing schemas, and running paginated SQL queries.
    -

  • sql-guard-mcp

    egoring
    A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI agents to safely query and explore SQLite databases through read-only, guard-protected tools that block writes, sensitive table access, and runaway queries.
    1
    MIT