# pdf-guide-agent

A chatbot that becomes an expert on **any folder of PDF guides**. Point it at the PDFs and ask
questions. An agent decides when, and how often, to search the guides, answers only from what it
finds, cites the guide and page for every fact, and says so when the guides don't cover a question.
The same search is served over **MCP**, so Claude Code or any MCP client can use the guides as a tool.

About 220 lines of Python: retrieval-augmented generation (RAG), a tool-calling agent and an MCP
server, with no agent framework and no vector database.

![Chat window: a follow-up question answered from the guides, with page citations](docs/screenshot.png)

## Sample conversation

The demo guides are three PDFs about mini computers for running AI models locally (see
[Demo data](#demo-data)). These answers come from the [measured run](eval/results.md):

> **What can I buy between $1,000 and $2,000?**
>
> Between $1,000 and $2,000, the guide lists three options (Buyers Guide, p. 3):
> - **Minisforum M2 — $1,249**: the easiest to upgrade, but has the lowest memory bandwidth in the guide.
> - **Framework Desktop — $1,269 (32 GB) or $1,959 (64 GB)**: 256 GB/s bandwidth; the 64 GB version fits larger models.
> - **Mac mini (M5 Pro) — $1,699**: the highest bandwidth in the guide at 307 GB/s, but only 24 GB in its base configuration.

> **Which product has the highest memory bandwidth?**
>
> The **Apple Mac mini (M5 Pro)**, with **307 GB/s** of peak memory bandwidth (Buyers Guide, p. 2). […]

> **What is the cheapest product in the guides?** → the Jetson Orin Nano Super Developer Kit, $399.
> **How much did it cost when it launched?**
>
> It launched in December 2024 at $249 (Product Profiles, p. 2). Its price is now $399, after NVIDIA raised it in July 2026 (Product Profiles, p. 2).

> **How do I bake sourdough bread?**
>
> The guides don't cover that.

## Architecture

```mermaid
flowchart LR
    PDF[PDF guides] -->|pypdf| P[Passages<br/>file + page]
    P -->|embeddings| IDX[(index.json)]
    subgraph Agent
        LLM[Chat model] -->|tool call| T[search_guides<br/>top-3 cosine + threshold]
        T -->|passages + scores| LLM
    end
    IDX --> T
    UI[Gradio chat] <--> Agent
    IDX --> MCP[MCP server<br/>search_guides] <--> CC[Claude Code /<br/>any MCP client]
```

| Step | File | What happens |
|---|---|---|
| Ingest | [`index.py`](pdf_guide_agent/index.py) | `pypdf` reads each page as one passage (long pages are split on paragraphs). All passages are embedded in one API call and stored with file name and page in `index.json`. |
| Search | [`index.py`](pdf_guide_agent/index.py) | Cosine similarity against every passage; the top 3 above a confidence threshold are returned with their scores. |
| Agent | [`agent.py`](pdf_guide_agent/agent.py) | A plain chat-completions tool loop: the model calls `search_guides` as often as it needs (capped at 5 rounds), rewrites follow-ups into its own queries, and cites `(Guide, p. N)`. Conversation history comes from the chat window. |
| Refusal | [`agent.py`](pdf_guide_agent/agent.py) | Enforced in code, not only in the prompt: below the threshold the tool returns "No passage in the guides covers this.", and the model answers "The guides don't cover that." |
| Chat | [`app.py`](pdf_guide_agent/app.py) | Gradio `ChatInterface`. |
| MCP | [`mcp_server.py`](pdf_guide_agent/mcp_server.py) | The same search as an MCP tool over stdio. It returns raw passages with scores and citations; the MCP client does the reasoning. |

## Stack

Python 3.12+, [uv](https://docs.astral.sh/uv/), OpenAI Python SDK (chat completions with tool
calling, embeddings), pypdf, NumPy, Gradio, the official MCP Python SDK, pytest. fpdf2 renders the
demo PDFs.

The model endpoints are configurable. By default the code talks to OpenAI (`gpt-4o-mini`,
`text-embedding-3-small`). Any OpenAI-compatible server works too, set separately for chat and for
embeddings (see [`.env.example`](.env.example)), so the embeddings can run locally on Ollama.

## Quick start

```bash
uv sync
export OPENAI_API_KEY=sk-...
uv run python -m pdf_guide_agent.index guides        # build index.json from a folder of PDFs
uv run python -m pdf_guide_agent.app                 # chat at http://localhost:7860
```

Use your own PDFs by pointing the first command at another folder. Nothing in the code is tied to
the demo subject.

The search threshold (0.46) was measured for the `bge-m3` embedding model. With another model, such as
the default `text-embedding-3-small`, measure it once with `uv run python -m eval.run_eval --scores`
and set `SEARCH_THRESHOLD` (see [Measured results](#measured-results)); otherwise covered questions
may be refused.

## Use it from Claude Code (MCP)

```bash
claude mcp add pdf-guides -e GUIDE_INDEX=$PWD/index.json -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -- uv run --directory $PWD python -m pdf_guide_agent.mcp_server
```

Then ask Claude Code something like *"Using the pdf-guides tool, which machines have 128 GB?"*.

## Measured results

[`eval/questions.json`](eval/questions.json) holds 12 fixed questions with known answers, in four
groups: price and memory ranges, highest/lowest, follow-ups that need the conversation history, and
questions the guides don't cover. [`eval/run_eval.py`](eval/run_eval.py) grades the answers
deterministically: an answer passes when it names every required product or price, and an uncovered
question passes only with the refusal.

| | |
|---|---|
| Score | **12/12** ([full answers](eval/results.md)) |
| Threshold | **0.46**, measured: the lowest top score of a covered question was 0.486, the highest of an uncovered one 0.440 (`python -m eval.run_eval --scores`) |
| Models in this run | chat `mimo-v2.5` via an OpenAI-compatible gateway, embeddings `bge-m3` on local Ollama |

The threshold depends on the embedding model. With another model, re-run `--scores` and set
`SEARCH_THRESHOLD`.

Range questions were the main risk of plain RAG. They work because the buyer's guide keeps the
comparison table on its own page, which `pypdf` extracts one row per line, so a single passage
holds every product with its price and memory. No catalogue-specific filter tool was needed.

## Demo data

[`guides/`](guides) holds three PDFs written for this project and rendered from
[`guides/source`](guides/source) by [`scripts/make_guides.py`](scripts/make_guides.py):

- **Buyers Guide**: the comparison table (10 configurations from Apple, Framework, Minisforum and NVIDIA) and advice by budget
- **Product Profiles**: one page per product: chip, memory, bandwidth, AI compute, power, current and launch price
- **Memory and Bandwidth**: how much memory a model needs and why bandwidth sets the speed

The product data is real, as of 26 October 2026. Each guide lists its sources on its last page.
Prices change often, and memory prices rose sharply in 2025–2026, so treat them as a snapshot.

## Tests

```bash
uv run pytest
```

The tests cover passage splitting, ranking and threshold, the agent loop (with a scripted model: tool
calls, refusal, history, round cap), the MCP tool registration, the eval grader, and that the
comparison table survives `pypdf` extraction. No test calls a real API.

## License

MIT