# techdocs-agent

Ask questions against a pile of technical documents — vendor manuals, standards, past reports — and
get answers with page-level citations. Built as three pieces that other agents can reuse:

1. **Retrieval as an MCP server** — read-only tools (`search_docs`, `get_passage`, `get_page`, …) over
   a local SQLite index with hybrid BM25 + embedding search. Works with Claude Desktop, Claude Code,
   or any MCP client.
2. **A grounded Claude agent** (`techdocs ask`) that answers *only* from retrieved passages, cites
   `[chunk_id]`s it actually received, and replies `NOT_IN_DOCUMENTS` when the documents do not
   contain the answer.
3. **An evaluation harness** (`techdocs eval`) that turns "is it right?" into numbers: retrieval
   recall@k, citation precision, LLM-judged correctness, refusal accuracy, tokens and latency.

> **Status: work in progress (started 2026-09-21).** Day 1–2 of a one-week plan are done — ingest,
> hybrid retrieval, MCP server, agent loop and the eval harness exist and are tested; the full
> answer-quality eval has not been run yet. See [docs/SPEC.md](docs/SPEC.md) for the plan and
> [docs/LESSONS.md](docs/LESSONS.md) for what has already gone wrong.

## Why

Engineering teams sit on thousands of PDFs nobody can search properly, so the same question gets
answered from scratch every time. I had already shipped a tool-using Claude agent inside an internal
CRM (18 tools, audit log, permission checks) — what it lacked was retrieval over unstructured
documents, a way to *measure* answer quality, and a boundary that lets other agents reuse the
retrieval. This repo is those three things. First real users: colleagues asking questions about
robot and sensor manuals (Dobot CR-A, iRayple SS5000) during commissioning.

## Quick start

```bash
uv sync --all-groups

# 1. index a folder of PDFs / Markdown (first run downloads the ~130 MB embedding model)
uv run techdocs ingest ./collections/manuals -c manuals

# 2. retrieval only — no API key needed
uv run techdocs search "rated payload of CR5A" --mode hybrid      # bm25 | dense | hybrid

# 3. grounded answers — needs ANTHROPIC_API_KEY in .env or the environment
uv run techdocs ask "What is the maximum payload of the CR5A?" -c manuals

# 4. evaluation
uv run techdocs eval eval/manuals.jsonl -c manuals --retrieval-only   # no key needed
uv run techdocs eval eval/manuals.jsonl -c manuals --out eval/results/$(date +%F).json

# 5. MCP server
uv run techdocs serve            # stdio — for Claude Desktop / Claude Code
uv run techdocs serve --http     # streamable HTTP on :8765
```

Claude Code: `claude mcp add techdocs -- uv --directory /path/to/techdocs-agent run techdocs serve`

## How it works

```
PDF / Markdown ──ingest──▶ page- & heading-aware chunks ──▶ SQLite (FTS5 BM25 + float32 vectors)
                                                                 │
                                                  search(): BM25 ⊕ dense (bge-small) → RRF
                                                                 │
                                    ┌────────────────────────────┴───────────────────┐
                              MCP server (FastMCP)                      Claude agent (tool loop)
                              read-only tools                           cite-or-refuse, citation check
                                    │                                               │
                     Claude Desktop / Claude Code / other agents        CLI `ask` · eval harness
```

Design decisions (ADRs are being written up in `docs/adr/`):

- **SQLite + FTS5 + local embeddings, no vector DB** — zero infrastructure, one file, good enough for
  tens of thousands of chunks.
- **Hybrid retrieval** — manuals are full of part numbers and error codes where lexical match wins;
  prose questions need dense. Reciprocal-rank fusion combines them.
- **Grounding is checked in code, not trusted from the model** — every `[chunk_id]` in an answer
  must be one the tools returned in that conversation (`Answer.invalid_citations`).
- **Document text is untrusted input** — passages are data; the system prompt tells the model to
  ignore instructions found inside documents. All tools are read-only.

## Evaluation

`eval/manuals.jsonl` — 25 questions (21 answerable with expected document + page, 4 deliberately
unanswerable) over the private manuals collection.

| metric | value | notes |
|---|---|---|
| retrieval recall@8 | 1.00 | hybrid |
| retrieval recall@3 | 0.86 | hybrid |
| citation precision / correctness / refusal | – | not yet run |

**Read these numbers with care.** The first golden set was written by reading chunks that BM25 had
already surfaced, so recall is optimistic; real questions from users replace it next
(see [LESSONS.md](docs/LESSONS.md)).

## Repository

```
src/techdocs_agent/   ingest · store · embed · search · server (MCP) · agent · evaluate · cli
eval/                 golden sets (committed) · results/ (git-ignored)
collections/          corpora (git-ignored — vendor PDFs are copyrighted)
docs/                 SPEC.md · LESSONS.md · adr/
tests/                pytest
```

CI runs ruff, pytest and a gitleaks secret scan on every push. API keys come only from `.env`
(git-ignored) or the environment.

## Author

Nemin Suksen — [linkedin.com/in/nemin-suksen-a4a05139](https://www.linkedin.com/in/nemin-suksen-a4a05139) ·
also see [factory-ops-console](https://github.com/neminsuksen/factory-ops-console).

License: MIT