RAG-MCP-SERVER

A modular RAG retrieval service that is pluggable and fully observable. It exposes retrieval capabilities as MCP (Model Context Protocol) tools and can be called directly by MCP clients such as Claude Desktop and GitHub Copilot.

The core design goals are to solve two specific pain points in RAG engineering:

1. Difficult-to-trace pipelines — when retrieval results are wrong, is the problem in recall, fusion, or rerank? The two pipelines (indexing and query) have 10 stages in total; each stage records latency, candidate count, scores, and rank changes, which can be visualized and traced back in the Dashboard.

2. Tuning by intuition — is switching to a different Embedding model actually better or worse? Hit Rate@K / MRR is combined with Ragas Faithfulness / Context Precision; regression is run on a fixed test set, and metrics are used to calibrate instead of subjective judgment.

Table of Contents

• Architecture Overview

• Core Capabilities

• Quick Start

• Configuration

• Observability

• Evaluation

• Tests

• Project Structure

• License

Related MCP server: mcp-rag-assistant

Architecture Overview

                    ┌──────────────────────────────────────────┐
  文档 (PDF/DOCX/    │           Ingestion Pipeline             │
  MD/TXT)      ───▶ │  load → split → transform → embed →      │
                    │  upsert                                  │
                    └────────────────┬─────────────────────────┘
                                     │  SHA256 指纹 + SQLite 摄取历史
                                     │  （文档级增量索引 / 幂等）
                                     ▼
                    ┌──────────────────────────────────────────┐
                    │   ChromaDB (Dense)  +  BM25 (Sparse)     │
                    └────────────────┬─────────────────────────┘
                                     ▼
                    ┌──────────────────────────────────────────┐
  查询          ───▶│            Query Engine                  │
                    │  query_processing → dense ┐              │
                    │                            ├→ RRF fusion │
                    │                    sparse ┘      │       │
                    │                                  ▼       │
                    │                              rerank      │
                    │                    （失败回退至 RRF 顺序） │
                    └────────────────┬─────────────────────────┘
                                     ▼
             ┌───────────────┬───────────────┬──────────────────┐
             │  MCP Server   │  CLI Scripts  │  Dashboard       │
             │  (3 tools)    │  (5 scripts)  │  (Streamlit 6页) │
             └───────────────┴───────────────┴──────────────────┘

  贯穿全程：TraceContext（trace → stage）写入 logs/traces.jsonl

Pluggable Foundation

Every core stage defines a unified Base interface. Components are swapped via Factory + YAML configuration with zero code changes:

Any OpenAI-compatible endpoint can be connected through provider: "openai" and a custom base_url, with no new code required.

Core Capabilities

Hybrid Retrieval: BM25 sparse retrieval handles exact matching for proper nouns, while dense vector retrieval handles semantic matching. After dual-path recall, RRF fusion merges the results, and a Reranker refines the final ranking. If the reranking backend fails, the system automatically falls back to the RRF fusion order, so a single timeout will not break the entire pipeline.

Incremental Indexing with Idempotency: SHA256 content fingerprints plus a SQLite ingestion_history table provide document-level incremental indexing. Repeated ingestion is skipped outright, and only content changes trigger a rebuild; duplicate ingestion never produces dirty data.

Multimodality: PyMuPDF extracts images embedded in PDFs and preserves their original positions; a Vision LLM generates image descriptions that are stitched into chunks. This reuses the plain-text RAG pipeline so you can “search text and get images”. The MCP response returns images as ImageContent.

MCP Tools:

Dashboard (Streamlit, six pages): system overview / data browsing / ingestion management / ingestion tracing / query tracing / evaluation panel.

Quick Start

Prerequisites

Python ≥ 3.10.

Installation

git clone <your-repo-url>
cd RAG-MCP-SERVER

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate

pip install -e ".[dev]"

All dependencies have version upper bounds. mcp is pinned below <2.0 (2.x renamed fields such as CallToolResult.isError), and langchain-community is pinned below <0.4 (0.4 removed chat_models.vertexai, which breaks ragas imports).

Configuration

cp config/settings.yaml.example config/settings.yaml

Edit config/settings.yaml and fill in your own API key. This file is ignored by .gitignore; do not commit it.

Ingest Documents

python scripts/ingest.py --path ./your_docs --collection my_kb
python scripts/ingest.py --path ./your_docs --collection my_kb --force   # 强制重建
python scripts/ingest.py --path ./your_docs --dry-run                    # 只看会处理哪些文件

Query

python scripts/query.py -q "你的问题" -c my_kb --top-k 5 --verbose

--verbose prints the intermediate results for each step: dense / sparse / fusion / rerank.

Start the Dashboard

python scripts/start_dashboard.py

Connect to an MCP Client

Using Claude Desktop as an example, add the following to claude_desktop_config.json:

{
  "mcpServers": {
    "rag-mcp-server": {
      "command": "<绝对路径>/.venv/Scripts/python.exe",
      "args": ["<绝对路径>/main.py"]
    }
  }
}

Configuration

Key configuration sections (full comments are in config/settings.yaml.example):

retrieval:
  dense_top_k: 20
  sparse_top_k: 20
  fusion_top_k: 10
  rrf_k: 60
  # 路由开关，用于 A/B 基线：只测 dense 则 enable_sparse: false，反之亦然
  enable_dense: true
  enable_sparse: true

rerank:
  enabled: true
  provider: "llm"           # 走已配置的 LLM，零额外依赖
  # provider: "cross_encoder"  # 本地 BGE cross-encoder，需 pip install sentence-transformers
  top_k: 5

evaluation:
  enabled: true
  provider: "composite"     # 同时跑检索指标与生成指标
  backends: ["custom", "ragas"]
  metrics: ["hit_rate", "mrr", "faithfulness", "context_precision"]

embedding.dimensions cannot be changed after the initial ingestion is complete — existing Chroma collections are bound to the vector dimension.

Observability

Both ingestion and query generate a trace written to logs/traces.jsonl, structured as trace → stages[]; each stage records elapsed_ms and the data of that stage.

Rank Change Tracking

Recording only the score list after each stage cannot answer the question: “Did this stage actually improve the ranking, and which chunk did it improve?” Therefore, the fusion and rerank stages additionally record rank changes (src/core/query_engine/rank_tracking.py):

• 1-based ranking is used, with rank_delta = rank_before - rank_after; a positive value means the rank moved up.

• For fusion, rank_before is the chunk’s best rank across the two paths, answering “Did RRF raise it above the single-path recall?” It also records dense_rank / sparse_rank to show which path recalled it.

• For rerank, rank_before is the position in the fused list passed to the reranker, accurately showing which chunk the reranker moved up or down.

• A newly appearing chunk reports None instead of a fabricated rank improvement.

• Stage-level summary: moved_up / moved_down / unchanged / new / max_gain / max_drop / dropped

Actual trace snippet:

stage=fusion   elapsed=0.2ms
  rank_changes: {moved_up: 3, moved_down: 1, unchanged: 1, max_gain: 2, dropped: 18}
  rank=2  before=4  delta=+2   dense_rank=4  sparse_rank=4

stage=rerank   elapsed=12231ms
  rank_changes: {moved_up: 1, moved_down: 1, unchanged: 3, max_gain: 1}
  rank=1  before=2  delta=+1

The Dashboard’s “query trace” page renders these as a stage waterfall chart and a rank-change table.

Evaluation System

python scripts/evaluate.py --collection my_kb
python scripts/experiment.py --variants dense,sparse,hybrid,hybrid_rerank

• Retrieval metrics (CustomEvaluator): Hit Rate@K, MRR — this expects the test set to provide expected_chunk_ids as ground truth.

• Generation metrics (RagasEvaluator): Faithfulness, Answer Relevance, Context Precision

• CompositeEvaluator runs both backends and combines their results; each backend selects its own subset from the shared metrics list, and one backend failing does not affect the others.

scripts/experiment.py is used for A/B comparison across different retrieval variants, and it outputs metrics and latency per variant to answer the question: “Is adding the reranker really worth those 12 seconds?”

Tests

Layered tests, 1456 cases in total:

pytest tests/unit                      # 1298 passed, 1 skipped
pytest tests/integration -m "not llm"  #   94 passed, 10 skipped
pytest tests/e2e -m "not llm"          #   30 passed,  2 skipped

-m "not llm" excludes test cases that require real LLM API calls. When credentials for a provider are missing, the affected tests log the reason and skip instead of failing.

Key branches have dedicated coverage:

Project Structure

src/
├── core/
│   ├── query_engine/       # 混合检索：dense / sparse / RRF fusion / rerank
│   │   └── rank_tracking.py  # 排名变化计算（融合与重排共用）
│   ├── response/           # 响应组装、引用生成、多模态拼装
│   ├── trace/              # TraceContext：trace → stage
│   ├── tokenization.py     # BM25 分词器（索引端与查询端唯一实现）
│   └── settings.py         # YAML 配置加载与校验
├── ingestion/
│   ├── chunking/ embedding/ storage/ transform/
│   ├── pipeline.py         # 五阶段摄取流水线
│   └── document_manager.py # 文档删除（跨 Chroma / BM25 / 图片 / 摄取历史）
├── libs/                   # 可插拔底座：base_*.py + *_factory.py
│   ├── llm/ embedding/ loader/ reranker/ splitter/ vector_store/ evaluator/
├── mcp_server/             # MCP 协议与 3 个 Tool
└── observability/
    ├── dashboard/          # Streamlit 六页
    └── evaluation/         # ragas / composite / eval_runner

scripts/   ingest / query / evaluate / experiment / start_dashboard
config/    settings.yaml.example + prompts/
tests/     unit / integration / e2e

data/ (Chroma, BM25 index, extracted images, ingestion history) and logs/ (traces) are all runtime-generated local artifacts. They are ignored by .gitignore, are not distributed with the repository, and are created automatically on the first run.

License

MIT