RAG as MCP

一个面向生产的 Agentic Hybrid RAG 引擎与 MCP Knowledge Service：在可靠的 Dense + BM25 + RRF 基础上提供自适应查询规划、证据驱动的纠正检索、有界执行、评估与链路追踪。

中文 · English

中文

项目简介

RAG as MCP 将 Retrieval-Augmented Generation 的核心链路拆分为可替换、可测试的工程组件，并通过 MCP stdio server 提供标准化知识检索接口。

项目重点不只是“让 RAG 跑起来”，而是围绕以下工程问题建立清晰边界：

• 文档如何稳定摄取并按 collection 隔离

• Dense 与 Sparse retrieval 如何融合

• rerank 如何作为可选阶段插拔

• 查询链路如何端到端追踪

• 检索质量如何离线评估和回归

• MCP 服务如何被其他 Agent 系统稳定调用

• 容器如何通过 CI/CD 可重复构建、验证与发布

项目可独立运行，也可作为 Incident Lifecycle Copilot 的外部知识检索后端。

Related MCP server: mcp-rag-assistant

核心能力

• Adaptive Query Planning：auto 路由让简单查询走低延迟 classic path，复杂查询进入 agentic path。

• Multi-Query / Query Decomposition：复杂问题拆解后复用既有 HybridSearch，并用 RRF 去重融合。

• Evidence-Guided Corrective Retrieval：证据不足时改写并在预算内重试。

• Bounded Agent Execution：显式 iteration、subquery 和 candidate budget，终止后返回最佳证据。

• Agentic Trace & Offline Evaluation：记录计划、评分、改写、终止与降级，并比较 classic 和 agentic。

• Document Ingestion Pipeline：PDF 加载、切分、转换、Embedding 与持久化。

• Hybrid Retrieval：Dense retrieval + BM25 sparse retrieval。

• RRF Fusion：使用 Reciprocal Rank Fusion 合并不同召回通道。

• Pluggable Reranking：支持可选 Cross-Encoder / LLM rerank。

• Collection Isolation：Dense、BM25、ImageStorage 与 FileIntegrity 数据按 collection 隔离。

• MCP Serving：通过 stdio JSON-RPC 暴露知识库查询工具。

• Observability：单一 TraceContext 贯穿 retrieval、fusion、rerank、multimodal 与 response 阶段。

• Offline Evaluation：支持本地评估与可选 Ragas 评估依赖。

• Engineering Validation：Unit / Integration / E2E 分层测试，Python 3.10 / 3.12 CI。

• Container Delivery：Git tag 驱动 Docker build、MCP smoke test 与 GHCR 发布。

架构

查询服务统一通过 query_knowledge_hub 暴露三种模式：

MCP query_knowledge_hub(mode=classic | auto | agentic)
                         |
                         v
                  Adaptive Router
                  /             \
       classic/direct          agentic
             |                   |
             |             Query Analysis
             |                   |
             |             Retrieval Plan
             |                   |
             |        Multi-query / Decomposition
             |                   |
             +----------> HybridSearch
                         Dense + BM25 + RRF
                                  |
                         Evidence Grader
                         /              \
                    sufficient       rewrite + retry
                         \              /
                          Final Reranker
                                  |
                       Citation / MCP Response

auto 对简单查询选择 classic path，对复杂查询选择 agentic path。agentic 是带明确预算、终止原因和 fallback 的 bounded adaptive retrieval loop。

摄取链路独立于查询路由：

                         Ingestion
                            |
                            v
PDF -> Loader -> Chunker -> Transform -> Embedding -> ChromaDB
                               |
                               +------------------> BM25 Index
                               |
                               +------------------> File / Image Metadata

核心模块：

src/ingestion/            文档摄取与存储协调
src/core/query_engine/    查询、混合检索、融合与重排序
src/core/agentic/         自适应分析、计划、评分、改写与有界编排
src/mcp_server/           MCP 协议与 tools
src/observability/        Trace、Dashboard 与评估

查询链路

三种模式共享同一个 MCP contract 和 HybridSearch primitive：

• classic：单次 HybridSearch，随后执行可选 final rerank。

• agentic：分析与规划查询，必要时拆分多路检索；证据不足时在预算内 rewrite/retry，最后只 rerank 一次。

• auto：使用 deterministic analyzer 路由，避免简单查询承担 agent loop 成本。

混合检索的目标不是简单增加召回通道，而是把语义召回与关键词精确匹配放在同一可评估管线中：

• Dense retrieval 负责语义相似内容

• BM25 对术语、错误码、服务名等 lexical signal 更敏感

• RRF 在不依赖两个检索器分数可比性的情况下融合排序

• reranker 作为独立阶段进一步优化候选顺序

Agentic 模式不是无界 autonomous agent，而是一个 bounded adaptive retrieval loop：它有明确的 iteration budget、fallback、termination reason 和 trace。任何智能模块失败都降级到 deterministic 或 classic retrieval，不牺牲服务可用性。

MCP 工具

当前 MCP Server 暴露以下主要工具：

query_knowledge_hub 的 mode 是可选参数：

{
  "query": "比较 BM25 与 Dense Retrieval 的失败模式",
  "collection": "default",
  "top_k": 10,
  "mode": "auto"
}

允许值为 classic、auto、agentic；省略时使用 agentic.default_mode。

Server 使用 stdout 传输 JSON-RPC，运行日志写入 stderr，避免污染 MCP transport。

启动：

python main.py

支持 stdio MCP 的客户端可使用类似配置：

{
  "mcpServers": {
    "rag-as-mcp": {
      "command": "/absolute/path/to/rag-as-mcp/.venv/bin/python",
      "args": ["/absolute/path/to/rag-as-mcp/main.py"],
      "cwd": "/absolute/path/to/rag-as-mcp",
      "env": {
        "MCP_SETTINGS_PATH": "/absolute/path/to/rag-as-mcp/config/settings.yaml"
      }
    }
  }
}

Correctness 设计

Collection 隔离

Dense index、BM25 index、ImageStorage 和 FileIntegrity 都按 collection 隔离，避免不同知识域之间发生隐式数据污染。

可重试文档更新

文档更新采用“先写新版本，再清理 stale 数据”的收敛策略，使中途失败的更新可以安全重试，而不要求跨多个存储实现分布式事务。

显式 best-effort 删除

DocumentManager.delete_document() 协调多个存储执行删除，但不会伪装成跨存储 ACID transaction。返回结果会明确暴露各 store 的成功与失败状态。

端到端 TraceContext

一次 query 创建单一 TraceContext，贯穿：

adaptive_route
  -> query_analysis / retrieval_plan       (agentic)
  -> retrieval_attempt_N
     -> Dense -> BM25 -> RRF
  -> evidence_grade_N / query_rewrite_N    (agentic)
  -> final_rerank
  -> multimodal / response

Trace metadata 记录 retrieval mode、strategy、iteration/rewrite/retrieval-call 计数、termination reason 与 degraded 状态。失败路径同样记录，便于区分智能模块降级、召回失败、fusion 异常或 rerank fallback。

环境要求

• Python 3.10+

• 在线 LLM / Embedding provider 的 API Key（按需）

• Ollama（仅当选择本地 Ollama provider 时需要）

安装

git clone https://github.com/Whalefallg/rag-as-mcp.git
cd rag-as-mcp

python -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

cp config/settings.example.yaml config/settings.yaml

Windows PowerShell：

.venv\Scripts\Activate.ps1

可选依赖：

python -m pip install -e ".[dashboard]"   # Streamlit Dashboard
python -m pip install -e ".[rerank]"      # Cross-Encoder reranking
python -m pip install -e ".[evaluation]"  # Ragas evaluation
python -m pip install -e ".[all]"         # All optional features

配置

config/settings.example.yaml 是可提交模板。

本地配置：

config/settings.yaml

已被 Git 忽略，凭证建议通过环境变量注入：

export RAG_LLM_API_KEY="your-llm-key"
export RAG_EMBEDDING_API_KEY="your-embedding-key"
export RAG_VISION_API_KEY="your-vision-key"

可选：

export RAG_LLM_AZURE_ENDPOINT="https://example.openai.azure.com/"
export RAG_EMBEDDING_BASE_URL="https://api.example.com/v1"
export MCP_SETTINGS_PATH="/absolute/path/to/settings.yaml"

不要将真实密钥写入示例配置或提交到 Git。

Agentic retrieval 的默认配置：

agentic:
  enabled: true
  default_mode: auto
  max_iterations: 2
  max_subqueries: 3
  max_candidate_results: 30
  planner:
    use_llm: false
  grader:
    use_llm: false
    min_results: 2
    min_confidence: 0.45
  rewriter:
    use_llm: false

LLM 辅助默认关闭；deterministic analyzer、grader 和 rewriter 可在无 API key 的环境运行。

运行完整流程

生成演示 PDF：

python scripts/gen_sample_pdfs.py

摄取文档：

python scripts/ingest.py \
  --path data/documents/default \
  --collection default

命令行查询：

python scripts/query.py \
  --query "RRF 如何融合检索结果？" \
  --collection default

启动 MCP Server：

python main.py

启动 Dashboard：

python scripts/start_dashboard.py

运行本地检索评估或并排比较：

python scripts/evaluate.py --mode classic
python scripts/evaluate.py --mode agentic
python scripts/evaluate.py --mode compare

测试与 CI

项目将测试分为 Unit、Integration 和 E2E 三层：

python -m pytest tests/unit -q
python -m pytest tests/integration -q
python -m pytest tests/e2e -q
python -m pytest -q

CI 策略：

Unit          Python 3.10 + 3.12
Integration   Python 3.12
E2E           Python 3.12

PytestCollectionWarning 被升级为错误，避免测试 helper 因命名或收集问题被静默忽略。

核心算法与协议测试可以在不调用付费外部模型服务的情况下运行。

Docker 与持续交付

本地构建：

docker build -t rag-as-mcp:local .

MCP 容器 smoke test：

python scripts/smoke_mcp_container.py rag-as-mcp:local

发布由 Git tag 驱动：

git tag v0.1.0
git push origin v0.1.0

Release Container workflow 执行：

Full pytest regression
        |
        v
Build Docker image
        |
        v
MCP stdio smoke test
        |
        v
Push version tag + latest to GHCR

发布镜像使用：

ghcr.io/whalefallg/rag-as-mcp:<version>
ghcr.io/whalefallg/rag-as-mcp:latest

容器默认可以使用示例配置启动 MCP 协议；实际环境可通过 MCP_SETTINGS_PATH 与 volume mount 注入配置和数据。

当前限制

• Loader 当前主要面向 PDF，其他文档格式尚未形成完整实现。

• 当前已注册 VectorStore backend 主要为 Chroma。

• 多进程并发写入与分布式部署不是当前设计范围。

• DocumentManager.delete_document() 是 best-effort coordination，而不是跨存储 ACID transaction。

• 外部模型的稳定性、限流、价格和输出质量不由本项目控制。

• 测试与评估结果用于验证实现正确性和回归，不等价于生产容量或 SLA 结论。

仓库结构

config/          配置模板与 prompt 配置
data/            运行时生成的示例文档、索引与持久化数据
scripts/         摄取、查询、评估、示例数据生成与 smoke test
src/             Agentic RAG、检索、摄取、MCP 与可观测性实现
tests/           unit、integration 与真正的 MCP/dashboard E2E tests

Roadmap

• 建立基于固定已摄取语料的可重复检索 benchmark

• 扩展更多文档 Loader

• 增加新的 VectorStore 与 Fusion backend

• 完善 MCP client 端到端集成示例

• 持续增强 retrieval evaluation 与 observability

License

MIT License。详见 LICENSE。

English

Overview

RAG as MCP is a production-oriented Agentic Hybrid RAG engine exposed through MCP. It adds adaptive planning, evidence-guided corrective retrieval, bounded execution, evaluation, and tracing on top of dense + BM25 + RRF retrieval.

The project focuses on the engineering boundaries required to make a RAG pipeline understandable and testable:

• deterministic document ingestion and collection isolation

• hybrid semantic and lexical retrieval

• pluggable fusion and reranking stages

• end-to-end query tracing

• offline retrieval evaluation

• a stable MCP interface for downstream agents

• repeatable testing, container validation, and release automation

It can run as a standalone retrieval service or as the external knowledge backend for Incident Lifecycle Copilot.

Highlights

• Adaptive query planning with classic, agentic, and low-latency auto routing.

• Multi-query and query decomposition over the existing HybridSearch primitive.

• Evidence-guided corrective retrieval with deterministic offline grading and optional LLM assistance.

• Bounded agent execution with iteration, subquery, and candidate budgets.

• Agentic tracing and offline evaluation including classic-vs-agentic comparison.

• Document ingestion pipeline — PDF loading, chunking, transformation, embeddings, and persistence.

• Hybrid retrieval — dense retrieval combined with BM25 sparse retrieval.

• RRF fusion — Reciprocal Rank Fusion combines independent rankers without requiring directly comparable scores.

• Pluggable reranking — optional Cross-Encoder or LLM-based reranking.

• Collection isolation — dense, BM25, image, and file-integrity data are isolated by collection.

• MCP serving — knowledge tools are exposed through stdio JSON-RPC.

• Observability — one TraceContext spans retrieval, fusion, reranking, multimodal processing, and response generation.

• Offline evaluation — local evaluation with optional Ragas support.

• Layered testing — unit, integration, and E2E suites across Python 3.10 and 3.12 CI.

• Container delivery — tag-driven Docker build, MCP smoke testing, and GHCR publishing.

Architecture

The query service exposes three modes through one query_knowledge_hub contract:

MCP query_knowledge_hub(mode=classic | auto | agentic)
                         |
                         v
                  Adaptive Router
                  /             \
       classic/direct          agentic
             |                   |
             |             Query Analysis
             |                   |
             |             Retrieval Plan
             |                   |
             |        Multi-query / Decomposition
             |                   |
             +----------> HybridSearch
                         Dense + BM25 + RRF
                                  |
                         Evidence Grader
                         /              \
                    sufficient       rewrite + retry
                         \              /
                          Final Reranker
                                  |
                       Citation / MCP Response

auto selects the classic path for simple queries and the agentic path for complex queries. Agentic execution is bounded by explicit budgets, termination reasons, and fallbacks.

Ingestion remains independent from query routing:

                         Ingestion
                            |
                            v
PDF -> Loader -> Chunker -> Transform -> Embedding -> ChromaDB
                               |
                               +------------------> BM25 Index
                               |
                               +------------------> File / Image Metadata

Core modules:

src/ingestion/            ingestion and storage coordination
src/core/query_engine/    query processing, hybrid retrieval, fusion, reranking
src/core/agentic/         adaptive analysis, planning, grading, rewriting, orchestration
src/mcp_server/           MCP protocol and tools
src/observability/        traces, dashboard, and evaluation

Query Pipeline

All modes share the same MCP contract and HybridSearch primitive:

• classic: one HybridSearch pass followed by optional final reranking.

• agentic: analyze and plan the query, optionally retrieve multiple subqueries, grade evidence, and rewrite/retry within the configured budget; rerank only the final pool.

• auto: route with the deterministic analyzer so simple queries avoid agent-loop overhead.

Each retrieval stage addresses a different signal:

• dense retrieval captures semantic similarity

• BM25 is effective for lexical signals such as service names, identifiers, and error codes

• RRF merges rankings without assuming the underlying scores are calibrated

• the optional reranker refines the final candidate order

Agentic mode is a bounded adaptive retrieval loop, not an unbounded autonomous agent. Intelligent components degrade to deterministic or classic retrieval when they fail.

MCP Tools

The server currently exposes the following primary tools:

query_knowledge_hub accepts an optional mode argument:

{
  "query": "Compare failure modes of BM25 and dense retrieval",
  "collection": "default",
  "top_k": 10,
  "mode": "auto"
}

Valid values are classic, auto, and agentic. When omitted, the server uses agentic.default_mode.

The server writes JSON-RPC messages to stdout and operational logs to stderr, keeping the MCP transport clean.

Start the server with:

python main.py

Example stdio MCP client configuration:

{
  "mcpServers": {
    "rag-as-mcp": {
      "command": "/absolute/path/to/rag-as-mcp/.venv/bin/python",
      "args": ["/absolute/path/to/rag-as-mcp/main.py"],
      "cwd": "/absolute/path/to/rag-as-mcp",
      "env": {
        "MCP_SETTINGS_PATH": "/absolute/path/to/rag-as-mcp/config/settings.yaml"
      }
    }
  }
}

Correctness Model

Collection Isolation

Dense indexes, BM25 indexes, image storage, and file-integrity state are separated by collection to avoid implicit cross-domain contamination.

Retryable Document Updates

Document replacement follows a convergence-oriented strategy: write the new version first, then remove stale state. This keeps updates retryable without pretending that all storage layers participate in one distributed transaction.

Explicit Best-Effort Deletion

DocumentManager.delete_document() coordinates deletion across multiple stores, but it is intentionally not presented as an ACID transaction. The result reports which stores succeeded and which failed.

End-to-End TraceContext

A query creates one TraceContext that spans:

adaptive_route
  -> query_analysis / retrieval_plan       (agentic)
  -> retrieval_attempt_N
     -> Dense -> BM25 -> RRF
  -> evidence_grade_N / query_rewrite_N    (agentic)
  -> final_rerank
  -> multimodal / response

Trace metadata includes retrieval mode, strategy, iteration/rewrite/retrieval-call counts, termination reason, and degradation state. Failure paths distinguish intelligent-component degradation, retrieval failures, fusion errors, and reranker fallback.

Requirements

• Python 3.10+

• API credentials for online LLM or embedding providers when enabled

• Ollama only when using a local Ollama-backed provider

Installation

git clone https://github.com/Whalefallg/rag-as-mcp.git
cd rag-as-mcp

python -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

cp config/settings.example.yaml config/settings.yaml

Windows PowerShell:

.venv\Scripts\Activate.ps1

Optional extras:

python -m pip install -e ".[dashboard]"   # Streamlit dashboard
python -m pip install -e ".[rerank]"      # Cross-Encoder reranking
python -m pip install -e ".[evaluation]"  # Ragas evaluation
python -m pip install -e ".[all]"         # All optional features

Configuration

config/settings.example.yaml is the committed configuration template.

Create a local configuration at:

config/settings.yaml

Credentials should be injected through environment variables:

export RAG_LLM_API_KEY="your-llm-key"
export RAG_EMBEDDING_API_KEY="your-embedding-key"
export RAG_VISION_API_KEY="your-vision-key"

Optional settings:

export RAG_LLM_AZURE_ENDPOINT="https://example.openai.azure.com/"
export RAG_EMBEDDING_BASE_URL="https://api.example.com/v1"
export MCP_SETTINGS_PATH="/absolute/path/to/settings.yaml"

Do not commit real credentials.

Default Agentic retrieval configuration:

agentic:
  enabled: true
  default_mode: auto
  max_iterations: 2
  max_subqueries: 3
  max_candidate_results: 30
  planner:
    use_llm: false
  grader:
    use_llm: false
    min_results: 2
    min_confidence: 0.45
  rewriter:
    use_llm: false

LLM assistance is disabled by default. Deterministic analysis, grading, and rewriting work without API credentials.

End-to-End Usage

Generate the bundled sample PDFs:

python scripts/gen_sample_pdfs.py

Ingest documents:

python scripts/ingest.py \
  --path data/documents/default \
  --collection default

Run a CLI query:

python scripts/query.py \
  --query "How does RRF combine retrieval results?" \
  --collection default

Start the MCP server:

python main.py

Start the dashboard:

python scripts/start_dashboard.py

Run retrieval evaluation or a side-by-side comparison:

python scripts/evaluate.py --mode classic
python scripts/evaluate.py --mode agentic
python scripts/evaluate.py --mode compare

Testing and CI

Tests are separated into unit, integration, and end-to-end layers:

python -m pytest tests/unit -q
python -m pytest tests/integration -q
python -m pytest tests/e2e -q
python -m pytest -q

Current CI matrix:

Unit          Python 3.10 + 3.12
Integration   Python 3.12
E2E           Python 3.12

PytestCollectionWarning is promoted to an error so accidentally uncollected tests do not fail silently.

Core algorithm and protocol tests are designed to run without paid external model calls.

Docker and Continuous Delivery

Build locally:

docker build -t rag-as-mcp:local .

Run the MCP container smoke test:

python scripts/smoke_mcp_container.py rag-as-mcp:local

Releases are driven by Git tags:

git tag v0.1.0
git push origin v0.1.0

The Release Container workflow performs:

Full pytest regression
        |
        v
Build Docker image
        |
        v
MCP stdio smoke test
        |
        v
Push version tag + latest to GHCR

Published images use:

ghcr.io/whalefallg/rag-as-mcp:<version>
ghcr.io/whalefallg/rag-as-mcp:latest

The container can boot the MCP protocol with the example configuration; production usage can inject local settings and data through MCP_SETTINGS_PATH and volume mounts.

Current Limitations

• The loader is currently focused on PDF documents.

• Chroma is the primary registered VectorStore backend.

• Multi-process write coordination and distributed deployment are outside the current scope.

• DocumentManager.delete_document() provides best-effort coordination rather than cross-store ACID semantics.

• External model availability, rate limits, cost, and output quality are outside this project's control.

• Test and evaluation results validate implementation behavior and regression; they should not be interpreted as production capacity or SLA claims.

Repository Structure

config/          configuration templates and prompt configuration
data/            generated sample documents, indexes, and runtime persistence
scripts/         ingestion, query, evaluation, sample generation, and smoke tests
src/             Agentic RAG, retrieval, ingestion, MCP, and observability
tests/           unit, integration, and genuine MCP/dashboard E2E coverage

Roadmap

• Add reproducible retrieval benchmarks against a fixed ingested corpus

• Support additional document loaders

• Add new VectorStore and fusion backends

• Expand end-to-end MCP client examples

• Continue improving retrieval evaluation and observability

License

Licensed under the MIT License.