Modular RAG MCP Server

Retrieval-augmented knowledge search and observable RAG infrastructure for study-abroad applications, built for the internal advisors of a DIY study-abroad service team

Modular RAG MCP Server is a local-first, pluggable, observable retrieval-augmented generation (RAG) service. The system provides knowledge retrieval to AI clients through the Model Context Protocol (MCP) and manages documents, ingestion jobs, query traces, and evaluation results through a Streamlit Dashboard.

This project grew out of a real collaboration scenario during university: a DIY study-abroad service team within the college helped students with overseas university applications. This system was built to solve the problem of advisors repeatedly searching across university requirements, application materials, process guidelines, and historical experience, with no way to trace sources. The system is currently deployed within the team; the public repository only contains anonymized synthetic samples, with no real student data, internal documents, or runtime data.

All sample materials in the repository must use anonymized synthetic data; system output is retrieval evidence for advisors to verify, not a substitute for advisor judgment, and does not constitute university, visa, or legal advice.

Table of Contents

• Business Background

• System Boundaries

• Core Capabilities

• System Architecture

• Data and Storage Consistency

• MCP Tools

• Dashboard

• Quick Start

• Quality Assurance

• Security and Operational Constraints

• Current Status and Roadmap

Related MCP server: mcp-rag-assistant

Business Background

When processing applications, DIY study-abroad advisors need to consult university official pages, program handbooks, material templates, internal checklists, and historical cases at the same time. Source materials are usually scattered as PDFs, and the following problems exist:

• The same requirement may appear in multiple documents with different wording, so pure keyword search tends to miss recalls.

• Proper nouns such as institutions, programs, degrees, and intake terms need exact matching; pure vector search tends to produce false recalls.

• Tables, flowcharts, and screenshots in PDFs contain important information; plain-text parsing loses context.

• Advisors need to know which document and which segment an answer came from, and whether the document is still valid.

• After a document is updated, the vector store, BM25 index, image index, and ingestion records must stay consistent.

• Retrieval quality needs to be validated through a stable test set regression, not subjective experience.

The system serves internal team advisors. A typical workflow is:

1. Ingest institution program materials, internal checklists, and anonymized cases into a designated Collection.

2. Submit natural-language questions through an MCP Client or the command line.

3. The system runs Dense + BM25 dual-path retrieval, RRF fusion, and optional Rerank.

4. Return text segments with source citations, and multimodal content blocks when images are hit.

5. Inspect ingestion progress, recall results, latency, and evaluation metrics through the Dashboard.

System Boundaries

This project is responsible for knowledge ingestion, retrieval, citation, evaluation, and pipeline observation. It is not responsible for:

• Making school-selection, admission-probability, or visa conclusions on behalf of advisors.

• Automatically submitting applications, sending emails, or modifying student materials.

• Providing student-facing accounts, CRM, payments, or application progress management.

• Automatically scraping and claiming to know the latest institution policies.

• Generating deterministic business conclusions without source support.

Core Capabilities

System Architecture

flowchart LR
    A["PDF 业务资料"] --> B["Ingestion Pipeline"]
    B --> C["Chroma 向量库"]
    B --> D["BM25 索引"]
    B --> E["SQLite 摄取历史"]
    B --> F["图片文件与索引"]
    G["顾问 / MCP Client"] --> H["MCP Server"]
    H --> I["Query Processor"]
    I --> J["Dense Retrieval"]
    I --> K["Sparse Retrieval"]
    J --> L["RRF Fusion"]
    K --> L
    L --> M["Optional Rerank"]
    M --> N["Response + Citations + Images"]
    B --> O["Ingestion Trace"]
    I --> P["Query Trace"]
    O --> Q["Streamlit Dashboard"]
    P --> Q

Core directories:

src/
├── core/            # 数据契约、查询编排、响应构建、Trace、配置
├── ingestion/       # Chunk、Transform、Embedding、Storage 与 Pipeline
├── libs/            # LLM/Embedding/Loader/Reranker/Splitter/VectorStore 抽象
├── mcp_server/      # MCP 协议处理、Server 与 Tools
└── observability/   # Dashboard、评估与结构化日志

scripts/             # ingest、query、evaluate、Dashboard 启动入口
config/              # Provider、检索、重排、评估与摄取配置
tests/               # Unit、Integration、E2E 测试与固定样例

Detailed interfaces, data flows, and module constraints are in DEV_SPEC.md.

Data and Storage Consistency

A single ingestion coordinates multiple storage backends:

File integrity checks use SHA256 to skip files that were already successfully processed and have not changed. Chunk IDs are stably generated from source, location, and content; repeated ingestion uses idempotent Upsert. DocumentManager coordinates deletion across Chroma, BM25, ingestion history, and the image index, and returns partial failure information.

MCP Tools

The current Server exposes four Tools. The first three general-purpose Tools are kept as-is; the fourth is a study-abroad business adaptation layer:

MCP uses Stdio Transport. stdout is reserved exclusively for JSON-RPC; runtime logs go to stderr to avoid corrupting protocol frames.

search_admissions_knowledge queries the admissions_knowledge collection by default and always restricts to business_domain=study_abroad_admissions. It supports exact filtering by country, institution, program, degree level, intake, application round, and source type; by default it excludes materials whose valid_until is earlier than the query business date. Materials with missing or unparseable validity dates are marked as needs_review and are not silently treated as current rules. The business Tool is only a parameter and response adaptation layer; underneath it still runs Dense + BM25, RRF, Cross-Encoder/LLM Rerank, citations, and multimodal responses.

Dashboard

The Dashboard keeps a six-page structure:

1. Overview: component configuration, data assets, runtime status, and current, needs-review, and expired statistics for study-abroad materials.

2. Data Browser: documents, Chunks, Metadata, and associated images; supports combined filtering by country, institution, program, degree, intake, application round, source type, and validity status.

3. Ingestion Manager: trigger ingestion, view progress, and coordinate document deletion.

4. Ingestion Traces: ingestion stages, processing methods, latency, and exceptions.

5. Query Traces: Dense/Sparse recall, fusion, reranking, and final results.

6. Evaluation Panel: run evaluations and view metrics and historical results.

Quick Start

1. Environment Setup

Requires Python 3.10–3.12.

git clone https://github.com/xiaoxinbuxingyeyuan/MODULAR-RAG-MCP-SERVER.git
cd MODULAR-RAG-MCP-SERVER
python -m venv .venv

Windows PowerShell:

.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

macOS / Linux:

source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

pyproject.toml pins the direct dependency versions currently validated for this project. When upgrading MCP, Ragas, LangChain, or storage components, upgrade them separately and re-run both offline and online regressions.

2. Configure Providers

Edit config/settings.yaml to configure the LLM, Embedding, Vision LLM, VectorStore, Reranker, and evaluation backends. API Keys should be injected through secure configuration and must not be committed to the repository.

When no local model service is available, you can disable non-essential LLM enhancements and Rerank to validate the base pipeline that does not depend on external services.

3. Ingest Documents

python scripts/ingest.py \
  --path tests/fixtures/sample_documents/simple.pdf \
  --collection admissions_knowledge

Directory ingestion:

python scripts/ingest.py \
  --path tests/fixtures/sample_documents/ \
  --collection admissions_knowledge

Ingestion with a study-abroad business Manifest:

python scripts/ingest.py \
  --path examples/documents/synthetic/ \
  --collection admissions_knowledge \
  --manifest examples/admissions_manifest.example.jsonl

The Manifest uses UTF-8 JSONL, with one PDF per line. Relative document_path values are resolved against the directory containing the Manifest; required fields are document_path, title, country, and source_type. Optional fields include institution, program, degree_level, intake, application_round, published_at, valid_until, language, tags, and access_scope. A full example is in examples/admissions_manifest.example.jsonl.

When a Manifest is provided, every PDF to be ingested must have a unique match. Unknown fields, duplicate paths, invalid enums, and inverted dates fail before anything is written to storage. Manifest metadata propagates from the Document to Chunk and Chroma records; Chunk-level LLM titles and tags do not override document_title and business_tags.

Incremental detection compares both the PDF SHA256 and the normalized business metadata SHA256: ingestion is skipped only when both are unchanged. Modifying only the Manifest automatically re-ingests and overwrites the metadata for the corresponding stable Chunk IDs. When PDF content changes, the system first writes the new version, then cleans up Chroma Chunks, images, and old ingestion records by the old doc_hash; BM25 replaces postings by the stable source path prefix. Legacy SQLite ingestion history automatically gains the metadata_hash field without manual migration. --force can still be used for explicit rebuilds, but is no longer required to apply Manifest updates.

The repository provides three fully fictional, personal-information-free business samples covering current materials, materials with missing validity dates, and expired materials; the course guide includes a flowchart used to validate the multimodal pipeline. To regenerate the sample PDFs, run:

python examples/generate_synthetic_admissions_pdfs.py

4. Command-Line Query

python scripts/query.py \
  --query "申请材料需要包含哪些证明？" \
  --collection admissions_knowledge \
  --verbose

5. Start the Dashboard

python scripts/start_dashboard.py

The default address is http://localhost:8501.

6. Start the MCP Server

python -m src.mcp_server.server

Configuration formats differ slightly between MCP Clients; the core process configuration is:

{
  "command": "<project>/.venv/Scripts/python.exe",
  "args": ["-m", "src.mcp_server.server"],
  "cwd": "<project>"
}

On macOS / Linux, replace the Python path with <project>/.venv/bin/python.

7. Run Evaluations

python scripts/evaluate.py \
  --test-set examples/admissions_golden_test_set.json \
  --collection admissions_knowledge

When no external retrieval environment is available, run:

python scripts/evaluate.py --no-search

Quality Assurance

The project uses a three-layer test structure:

• Unit: data contracts, algorithms, Factory, Tool Handlers, and storage adapters.

• Integration: combined behavior of ingestion, hybrid retrieval, MCP, Providers, and Traces.

• E2E: CLI ingestion, MCP Client, Dashboard smoke, and Recall regression.

python -m pytest tests/unit
python -m pytest tests/integration
python -m pytest tests/e2e
python -m pytest

The commands above skip all cases marked online by default and do not actively call real Providers. To use real Azure, OpenAI, or Ollama services, explicitly run in an environment with the corresponding credentials and services available:

python -m pytest --run-online -m online

An OpenAI-compatible gateway can be injected via OPENAI_API_KEY, OPENAI_BASE_URL, and OPENAI_MODEL without modifying repository configuration or committing credentials. Provider cases that are not configured should remain skipped.

To verify only that online cases are correctly categorized without making any calls, run python -m pytest --collect-only -m online. Offline and online results should be recorded separately; --run-online only lifts the skip restriction and does not replace Provider configuration.

The evaluation layer continues to support Custom Evaluator and Ragas. The business Golden Test Set additionally records combined filters, business dates, expiry policy, expected sources, and reference answers; offline business acceptance checks both these fields and the validity behavior of the MCP business adapter. The general evaluation interface and the original Golden Test Set are unchanged.

Security and Operational Constraints

• Local Stdio and local storage are used by default; no network ports are opened.

• API Keys and student personal information are never stored in logs, Traces, test fixtures, or Git history.

• Business materials must complete authorization confirmation and privacy redaction before entering the knowledge base.

• Retrieval results must retain source citations; when no reliable evidence can be found, return an empty result or prompt for manual verification.

• Institution requirements are time-sensitive; the business Tool excludes materials past valid_until by default and explicitly flags items with missing validity dates, but advisors must still verify against official sources.

• The current architecture is a single-user local service and provides no authentication, permission isolation, or multi-tenancy guarantees.

Current Status and Roadmap

The existing main branch already has a complete skeleton for general RAG, MCP, Dashboard, Trace, and evaluation. The study-abroad domain adaptation proceeds incrementally and must not remove or simplify existing technical capabilities.

Every stage's implementation must preserve full-pipeline PDF ingestion, Dense + BM25, RRF, Rerank, multimodal, multi-storage coordination, incremental and deletion support, the original three MCP Tools, the six-page Dashboard, dual-pipeline Traces, Custom + Ragas, three-layer testing, and all pluggable interfaces.