# Enterprise Knowledge MCP Server

> Unstructured Data Pipeline & Remote MCP Server — a production-ready enterprise
> document knowledge base. Ingests DOCX / PDF / PPTX, parses with Docling, cleans
> and chunks with metadata, indexes into a hybrid search store, and exposes a
> Remote MCP Server for Claude Desktop and other agents.

## Status

Built incrementally, one step at a time (see `CLAUDE.md` for the full plan).

- [x] **Step 1 — Project Bootstrap**: FastAPI + FastMCP + Docker + Pytest
- [x] **Step 2 — Document Upload**: `POST/GET /documents` + persistent catalogue (DOCX/PDF/PPTX); upload now auto-runs the full pipeline and indexes immediately (no restart)
- [x] **Step 3 — Docling Parser**: Docling -> structured `ParsedDocument` (headings/text/tables/figures, page & slide provenance)
- [x] **Step 4 — Cleaning Pipeline**: strip repeated headers/footers, page numbers, empty/symbol-only noise (structure preserved)
- [x] **Step 5 — Metadata-aware Chunking**: semantic chunks (section/table/figure) with full metadata; no fixed-width cuts
- [x] **Step 6 — Chroma Indexing**: BGE dense embeddings into embedded persistent Chroma (`index`/`search`/`get`/`delete`)
- [x] **Step 7 — Hybrid Retrieval**: dense (BGE) + sparse (BM25) fused with RRF; mixed CN/EN tokenizer
- [x] **Step 8 — MCP Tools**: `search_documents` / `list_documents` / `get_document` / `get_chunk` on FastMCP
- [x] **Step 9 — MCP Resources**: `documents://all` and `documents://{document_id}`
- [x] **Step 10 — Integration Test**: end-to-end MCP protocol test (`Client` -> server -> tool/resource) + runnable client demo

## Architecture (target)

```
DOCX / PDF / PPTX
    -> Docling Parser
    -> Cleaning Pipeline
    -> Metadata-aware Chunking
    -> Hybrid Search Index (BGE dense + BM25 sparse, Chroma)
    -> Remote MCP Server (FastMCP)
    -> Claude Desktop
```

## Tech Stack

| Area             | Choice           |
| ---------------- | ---------------- |
| Language         | Python 3.11      |
| API              | FastAPI          |
| Parsing          | Docling          |
| Search           | Hybrid Retrieval |
| Dense Retrieval  | BGE Embedding    |
| Sparse Retrieval | BM25             |
| Vector DB        | Chroma (embedded) |
| MCP Framework    | FastMCP          |
| Deployment       | Docker           |
| Testing          | Pytest           |

## Quick Start (local)

This repo ships a pre-created virtual environment (`kb_mcp_env/`, Windows).

```powershell
# Install dependencies (incl. dev/test extras)
kb_mcp_env\Scripts\python.exe -m pip install -e ".[dev]"

# Run the tests
kb_mcp_env\Scripts\python.exe -m pytest -q

# Run the server
kb_mcp_env\Scripts\python.exe -m uvicorn app.main:app --reload
```

- Health check: <http://localhost:8000/health>
- Remote MCP endpoint: <http://localhost:8000/mcp>

## Run with Docker

```bash
cp .env.example .env   # optional
docker compose up --build
```

Brings up the app on port 8000. Chroma runs embedded in-process (no separate
service); its data persists in the `chroma_storage` Docker volume.

## Deploy to Zeabur (public URL)

Zeabur auto-detects the `Dockerfile`. This is a heavy ML service (BGE-M3 ~2.2GB
embedding model + Docling/Torch + Chroma), so it needs a plan with enough RAM
(recommended **≥ 4 GB**) and a persistent **Volume**; the first request is slow
while models download.

1. Push this repo to GitHub.
2. Zeabur → **Create Project** → **Add Service → Deploy from GitHub** → pick this repo
   (it builds from the `Dockerfile`).
3. **Volumes** → add a volume mounted at `/data` (keeps the index, uploaded files,
   and downloaded models across redeploys).
4. **Variables** → set:
   ```
   UPLOAD_DIR=/data/uploads
   CHROMA_PERSIST_DIR=/data/chroma_storage
   HF_HOME=/data/hf_cache
   ```
   (The platform injects `$PORT`; the Dockerfile already honours it.)
5. **Networking** → Generate Domain → `https://<your>.zeabur.app`.

After deploy:

- Health check: `https://<your>.zeabur.app/health`
- Remote MCP endpoint: `https://<your>.zeabur.app/mcp`
- API docs (Swagger UI): `https://<your>.zeabur.app/docs`

Point Claude Desktop / Claude Code at the `/mcp` URL to use the deployed server.

## Parsing & OCR

> 提醒：OCR 是在解析時對每張圖跑，26 張圖會增加數十秒解析時間。若某類文件不需要，可在 `.env` 設 `OCR_IMAGES=false` 關閉。

> 提醒：預設 embedding 模型為 `BAAI/bge-m3`（多語，適合中英混雜，1024 維、約 2.2GB，首次會下載）。若只需英文且要更輕量，可在 `.env` 設 `EMBEDDING_MODEL=BAAI/bge-small-en-v1.5`。切換模型若維度不同，需先清空 `chroma_storage` 重新索引。

## Example Queries (target MCP tools)

```
What is the yield improvement plan?
Show me the KPI table from Q4 report.
Summarize slide 5.
```

## Add a document (auto-indexed, no restart)

`POST /documents` saves the file **and** runs the full pipeline (Docling parse ->
clean -> metadata-aware chunk -> BGE embed -> Chroma index) in the same process,
then refreshes BM25. Because it shares the MCP server's vector-store/retriever
singletons, the document is searchable over MCP immediately — no restart needed.

```powershell
# server running on :8000
curl.exe -X POST http://127.0.0.1:8000/documents -F "file=@E:\path\to\report.pdf"
# -> 201 {"document_id": "...", "status": "indexed", "num_chunks": 42, ...}
```

The response carries `status` (`indexed`, or `failed` with HTTP 500 if parsing
errors — the file is still recorded) and `num_chunks`. The call blocks until
indexing finishes (Docling/OCR/embedding can take tens of seconds for large or
image-heavy files). `scripts/ingest_file.py` shares the same pipeline for
command-line ingestion.

## Verify the MCP Server (client demo + server log)

Start the server, then drive it over the real MCP protocol with the bundled
client demo:

```powershell
# Terminal 1 — run the server
kb_mcp_env\Scripts\python.exe -m uvicorn app.main:app --host 127.0.0.1 --port 8000

# Terminal 2 — connect a remote MCP client and run the example queries
kb_mcp_env\Scripts\python.exe tests\mcp_client_demo.py
# ...or pass your own query:
kb_mcp_env\Scripts\python.exe tests\mcp_client_demo.py "yield improvement plan"
```

The demo connects (`Client -> MCP Server -> search_documents -> Result`), lists
the server's tools/resources, reads `documents://all`, and prints the retrieved
chunks. Meanwhile the server console logs each invocation:

```
INFO:app.mcp_server:MCP tool invoked: search_documents | query='...' top_k=3
INFO:app.mcp_server:search_documents retrieved 3 chunk(s): [...]
```

The hermetic equivalent (no running server, isolated temp index) is the pytest
integration test:

```powershell
kb_mcp_env\Scripts\python.exe -m pytest tests\test_mcp_integration.py -q
```

## AI Workflow

This project is developed with an AI-only workflow (Claude Code + MCP). Each
development step follows: plan -> implement -> review -> test, with a dedicated
commit per step. See `CLAUDE.md` for the step-by-step record.