MCP-Knowledge-Toolbox

MCP-Knowledge-Toolbox is a local knowledge-base MCP toolbox built on top of the Project 1 DocuPilot-RAG baseline. Project 2 does not modify Project 1 core code. It packages local document ingest, retrieval, context reading, citation checking, and evaluation-report reading as MCP-callable tools.

This repository is currently an engineering MVP, not a production multi-tenant RAG platform.

Architecture

flowchart LR
    A[Local Documents] --> B[Parser]
    B --> C[Chunker]
    C --> D[SQLite Metadata Store]
    C --> E[Vector Index]
    C --> F[BM25 Index]
    E --> G[Hybrid Retriever]
    F --> G
    G --> H[Lightweight Reranker]
    H --> I[MCP Tools]
    I --> J[MCP stdio Client]
    I --> K[Citation Verifier]
    I --> L[Eval Report Reader]

Related MCP server: KnowFlow MCP

Tech Stack

• Python 3.10/3.11 compatible code path

• SQLite metadata store

• MCP stdio JSON-RPC compatible MVP transport

• Optional official MCP Python SDK when installed

• sentence-transformers with BAAI/bge-small-zh-v1.5 as the default embedding model

• hashing vector fallback when the embedding model is unavailable

• PyMuPDF for PDF, python-docx for docx, native readers for Markdown/txt

• pytest integration tests

Tools

The server exposes 11 tools:

ingest_file, ingest_folder, search_knowledge, read_chunk_neighbors, summarize_document, query_table, verify_citation, get_eval_report, list_documents, delete_document, server_status.

MCP Compatibility

Current implementation is an MCP stdio JSON-RPC compatible MVP. It can use the official MCP Python SDK if installed; otherwise it uses the built-in stdio JSON-RPC transport.

Reproduce From Scratch

From a fresh clone:

pip install -r requirements.txt
python scripts/ingest_demo_docs.py --input data/raw --collection demo
python scripts/build_index.py --collection demo
python scripts/run_mcp_stdio_client_demo.py
pytest tests

Expected scale after ingest:

ingested files: 20
success: 20
failed: 0
chunks: 1201
documents: 20
collections: demo
embedding_provider: sentence-transformers

End-to-End Demo

Generate the full E2E MCP log:

python scripts/run_e2e_demo.py --collection e2e --input data/raw --output docs/e2e_demo_log.md

The log records:

• MCP server startup through stdio subprocess

• stdio client initialize

• tools/list

• tools/call ingest_folder

• tools/call list_documents

• tools/call search_knowledge

• tools/call read_chunk_neighbors

• tools/call verify_citation

• final answer with citations

See docs/e2e_demo_log.md.

Retrieval Evaluation

Generate 50 QA samples and evaluate four retrieval strategies:

python scripts/run_retrieval_eval.py --collection demo

Outputs:

• data/eval/demo_qa.jsonl

• docs/retrieval_eval_report.md

Current measured metrics:

Hybrid improved over individual retrieval modes on this demo set. Hybrid + rerank did not improve over hybrid; the report explains that the corpus is synthetic and repetitive, so first-stage retrieval already ranks many expected documents at the top.

Final Acceptance Artifacts

• docs/e2e_demo_log.md

• docs/retrieval_eval_report.md

• docs/final_acceptance.md

• data/eval/demo_qa.jsonl

Limitations

• hashing vector is only a fallback when the sentence-transformers model is unavailable.

• verify_citation is a lightweight keyword/similarity check, not an LLM judge.

• query_table is Markdown table caption/content matching, not complex table reasoning.

• rerank is lightweight token-overlap reranking, not a cross-encoder reranker.

• summarize_document uses extractive summarization when no LLM is configured.

• current storage is local SQLite and local JSON indexes, not a distributed vector database.

• current MCP support covers tools over stdio, not resources/prompts/sampling.

• this is not a production-grade multi-tenant platform.

Resume Wording

MCP-Knowledge-Toolbox: a local knowledge-base MCP toolbox for Agent workflows. Built an MCP stdio JSON-RPC compatible server exposing 11 tools for document ingest, SQLite metadata management, sentence-transformers vector retrieval, BM25, hybrid retrieval, context reading, citation verification, document deletion sync, and evaluation report reading. Added an end-to-end stdio client demo, 50-sample retrieval evaluation, and 37 pytest tests. Demo acceptance reached 20 documents and 1201 chunks across Markdown, txt, docx, and PDF.