rag-docs-assistant MCP server
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
rag-docs-assistant
Retrieval-augmented question answering over a document collection, built the way I'd put RAG into production: hybrid retrieval, answers that cite their sources (and are checked for it), and evals that gate every pull request.
flowchart LR
subgraph Ingest
D[Markdown / text docs] --> S[LangChain splitter<br/>heading-aware chunks]
S --> E[Embeddings<br/>local fastembed by default]
E --> PG[(PostgreSQL<br/>pgvector + tsvector)]
end
subgraph Query
Q[Question] --> V[Vector search<br/>cosine, HNSW]
Q --> K[Keyword search<br/>Postgres full-text]
V --> F[Reciprocal Rank Fusion]
K --> F
F --> G{Enough evidence?}
G -- no --> N[“I don't know” + closest sources]
G -- yes --> L[LLM answer with [n] citations]
L --> C[Citation check:<br/>every claim cites a retrieved chunk]
end
PG --- V & K
MCP[MCP server<br/>search_docs / ask] --> QAPI | FastAPI: |
Store | PostgreSQL 17 with |
Retrieval | vector + keyword, fused with Reciprocal Rank Fusion (no score normalisation needed) |
LLM | pluggable: Claude (default), OpenAI, Ollama, and a deterministic fixture provider for CI |
Evals | golden question set, with recall@k and MRR computed in CI as a gate. LLM-judged faithfulness runs when a real model is configured |
MCP |
|
Key decisions are recorded as ADRs:
ADR-0001: Hybrid retrieval fused with Reciprocal Rank Fusion
ADR-0002: Local embeddings by default, behind an interface
ADR-0003: Citations are verified, not trusted, and the system says "I don't know"
ADR-0004: Retrieval evals gate CI. Answer evals are separate and opt-in
Ingestion
Heading-aware chunking (LangChain
MarkdownHeaderTextSplitter, thenRecursiveCharacterTextSplitteronly inside sections that are too long). A chunk never straddles two sections, and each carries a breadcrumb (On-call rotation > Paging policy) that is embedded with it, so chunks that read alike out of context stay distinguishable.Idempotent and incremental: documents are keyed by
sourceand a SHA-256 of their content. Unchanged documents are skipped, changed ones are re-chunked in place, and--prunemirrors a directory exactly. Switching embedding model forces re-embedding, because the model is stored per chunk.One store:
chunks.embedding vector(384)with an HNSW index, plus a generated, weightedtsvector(heading > body) with a GIN index, ready for hybrid retrieval.
make up && make ingest # PostgreSQL + API, then index corpus/ (a sample engineering handbook)
make test # unit + integration tests against pgvectorRelated MCP server: Databricks Docs MCP Server
Retrieval and evals
POST /search {"query": "...", "top_k": 5, "mode": "hybrid|vector|keyword"} returns chunks with their source,
heading breadcrumb, fused score and each retriever's rank (so you can see why a chunk ranked where it did).
Vector and keyword candidates (20 each) come from one PostgreSQL round-trip each and are fused with RRF (k = 60).
The retrieval eval runs in CI with the real model (bge-small-en-v1.5) on 35 labelled questions over corpus/.
Many questions are paraphrased away from the document wording ("how long can I stay home sick without a doctor's note?").
A hit counts only if it comes from the right document and the right section.
mode | recall@5 | MRR | misses |
vector | 0.971 | 0.911 | q04 |
keyword | 0.943 | 0.830 | q15, q19 |
hybrid (RRF) | 0.971 | 0.887 | q19 |
Honest reading: on this small, clean corpus, dense retrieval alone is already strong. Hybrid matches its recall but has a slightly lower MRR, because keyword ranks occasionally pull a relevant chunk down one place. Hybrid stays the default because it rescues exact-token queries (codes, numbers, acronyms) that dense search misses on larger corpora (q04 is rescued here). The CI gate fails below recall@5 0.94 / MRR 0.85, set just under the measured baseline. Tuning RRF weights against this set without a held-out split would overfit, so that waits for a larger golden set.
python -m evals.retrieval_eval --check # needs DATABASE_URL; ingests corpus/ and prints the tableThe answer eval (opt-in, needs a real model) runs every golden question through the full /ask pipeline and reports
coverage (answered rate on answerable questions), faithfulness and correctness (LLM-as-judge, judged only
against the passages the answer cited), whether the cited passage was the right section, and abstention accuracy
on the unanswerable questions. Reports go to evals/reports/. It isn't a CI gate because it costs money and varies
between runs. No live report exists yet: no API key was available while building this, and the harness is unit-tested.
LLM_PROVIDER=anthropic ANTHROPIC_API_KEY=... python -m evals.answer_evalAnswers with verified citations
POST /ask {"question": "..."} → {answer, abstained, reason, citations: [{n, chunk_id, source, heading}], sources}
Retrieve the top 5 passages (hybrid).
Stage 1 abstain: if the best cosine similarity is below 0.62 (calibrated on the golden set, see ADR-0003), answer "I couldn't find this" without calling the model, and list the closest sources.
Otherwise the model (Claude by default, OpenAI, Ollama, or fixtures) returns structured output
{answerable, answer, citations}, with inline[n]markers.Verification: citations must point at retrieved passages, inline markers must match the declared citations, and an answerable reply must cite something. A failing reply gets one corrective retry with the exact problems, then abstains (
unverifiable_citations).Stage 2 abstain: the model may say the passages don't contain the answer (
not_in_context).
The prompt treats documents as untrusted input (prompt-injection guard).
MCP server
The knowledge base is available to any MCP client (Claude Code, Claude Desktop, IDEs) as two tools with typed input and output schemas:
Tool | Returns |
| passages: |
|
|
# Claude Code (stdio transport; the server reads DATABASE_URL and LLM_PROVIDER from the environment)
claude mcp add ragdocs --env DATABASE_URL=postgresql://rag:rag@localhost:5432/rag -- python -m ragdocs.mcp_serverBuilt on the MCP Python SDK 2.x (MCPServer). Tests call the tools through the server's own list_tools/call_tool.
Roadmap
# | Milestone | Status |
1 | Ingestion: heading-aware chunking, local embeddings, pgvector schema, | ✅ done (#1) |
2 | Hybrid retrieval (vector + full-text + RRF), | ✅ done (#2) |
3 | Cited answers: providers, two-stage abstain, citation verification, | ✅ done (#3) |
4 | MCP server ( | ✅ done (#4) |
5 | Answer eval harness: LLM-judged faithfulness/correctness, coverage, abstention accuracy, reports | ✅ done (#5) (no live run yet) |
Licence
MIT
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
Ingest, manage, and retrieve documents for RAG-powered AI applications
Convert PDFs, Office docs, websites and audio into AI-ready Markdown, then search and ask your docs.
Private, multimodal knowledge store for agents: hybrid search with cited, provenance-backed results.
Read-only search and Markdown pages from the OpenPlanr docs, for coding agents.
Related MCP Servers
- AlicenseAqualityDmaintenanceProvides tools for retrieving and processing documentation through vector search, enabling AI assistants to augment their responses with relevant documentation context.720 npm1MIT
- FlicenseNot gradedqualityDmaintenanceEnables semantic search over Databricks docs, API reference, Terraform provider docs, and knowledge base via tools like search_databricks_docs and research.-
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to query and manage a document knowledge base via MCP, with RAG-powered search and grounded answers with citations.MIT
- FlicenseNot gradedqualityDmaintenanceEnables document indexing and question answering with Retrieval-Augmented Generation, providing cited answers from user-supplied documents.-