techdocs-agent

Ask questions against a pile of technical documents — vendor manuals, standards, past reports — and get answers with page-level citations. Built as three pieces that other agents can reuse:

1. Retrieval as an MCP server — read-only tools (search_docs, get_passage, get_page, …) over a local SQLite index with hybrid BM25 + embedding search. Works with Claude Desktop, Claude Code, or any MCP client.

2. A grounded Claude agent (techdocs ask) that answers only from retrieved passages, cites [chunk_id]s it actually received, and replies NOT_IN_DOCUMENTS when the documents do not contain the answer.

3. An evaluation harness (techdocs eval) that turns "is it right?" into numbers: retrieval recall@k, citation precision, LLM-judged correctness, refusal accuracy, tokens and latency.

Status: work in progress (started 2026-09-21). Day 1–2 of a one-week plan are done — ingest, hybrid retrieval, MCP server, agent loop and the eval harness exist and are tested; the full answer-quality eval has not been run yet. See docs/SPEC.md for the plan and docs/LESSONS.md for what has already gone wrong.

Why

Engineering teams sit on thousands of PDFs nobody can search properly, so the same question gets answered from scratch every time. I had already shipped a tool-using Claude agent inside an internal CRM (18 tools, audit log, permission checks) — what it lacked was retrieval over unstructured documents, a way to measure answer quality, and a boundary that lets other agents reuse the retrieval. This repo is those three things. First real users: colleagues asking questions about robot and sensor manuals (Dobot CR-A, iRayple SS5000) during commissioning.

Related MCP server: insight-mcp

Quick start

uv sync --all-groups

# 1. index a folder of PDFs / Markdown (first run downloads the ~130 MB embedding model)
uv run techdocs ingest ./collections/manuals -c manuals

# 2. retrieval only — no API key needed
uv run techdocs search "rated payload of CR5A" --mode hybrid      # bm25 | dense | hybrid

# 3. grounded answers — needs ANTHROPIC_API_KEY in .env or the environment
uv run techdocs ask "What is the maximum payload of the CR5A?" -c manuals

# 4. evaluation
uv run techdocs eval eval/manuals.jsonl -c manuals --retrieval-only   # no key needed
uv run techdocs eval eval/manuals.jsonl -c manuals --out eval/results/$(date +%F).json

# 5. MCP server
uv run techdocs serve            # stdio — for Claude Desktop / Claude Code
uv run techdocs serve --http     # streamable HTTP on :8765

Claude Code: claude mcp add techdocs -- uv --directory /path/to/techdocs-agent run techdocs serve

How it works

PDF / Markdown ──ingest──▶ page- & heading-aware chunks ──▶ SQLite (FTS5 BM25 + float32 vectors)
                                                                 │
                                                  search(): BM25 ⊕ dense (bge-small) → RRF
                                                                 │
                                    ┌────────────────────────────┴───────────────────┐
                              MCP server (FastMCP)                      Claude agent (tool loop)
                              read-only tools                           cite-or-refuse, citation check
                                    │                                               │
                     Claude Desktop / Claude Code / other agents        CLI `ask` · eval harness

Design decisions (ADRs are being written up in docs/adr/):

• SQLite + FTS5 + local embeddings, no vector DB — zero infrastructure, one file, good enough for tens of thousands of chunks.

• Hybrid retrieval — manuals are full of part numbers and error codes where lexical match wins; prose questions need dense. Reciprocal-rank fusion combines them.

• Grounding is checked in code, not trusted from the model — every [chunk_id] in an answer must be one the tools returned in that conversation (Answer.invalid_citations).

• Document text is untrusted input — passages are data; the system prompt tells the model to ignore instructions found inside documents. All tools are read-only.

Evaluation

eval/manuals.jsonl — 25 questions (21 answerable with expected document + page, 4 deliberately unanswerable) over the private manuals collection.

Read these numbers with care. The first golden set was written by reading chunks that BM25 had already surfaced, so recall is optimistic; real questions from users replace it next (see LESSONS.md).

Repository

src/techdocs_agent/   ingest · store · embed · search · server (MCP) · agent · evaluate · cli
eval/                 golden sets (committed) · results/ (git-ignored)
collections/          corpora (git-ignored — vendor PDFs are copyrighted)
docs/                 SPEC.md · LESSONS.md · adr/
tests/                pytest

CI runs ruff, pytest and a gitleaks secret scan on every push. API keys come only from .env (git-ignored) or the environment.

Author

Nemin Suksen — linkedin.com/in/nemin-suksen-a4a05139 · also see factory-ops-console.

License: MIT