# textbook-agent-mcp

OCR an Italian textbook PDF with Mistral, structure it with an LLM, store it
in SQLite, and expose it via MCP so external chatbot agents (e.g. Mistral
Vibe Work) can work through the book with you.

Source book: **"Italiano facile"** (Diffusione Scolastica, 2009), 118 pages,
CEFR A2–B1 — grammar for non-native speakers.

```
data/texbook/textbook.pdf   (gitignored)
 └─▶ 01_ocr.py        mistral-ocr-latest  → data/ocr/            (+ pdftotext cross-check)
       └─▶ 02_structure.py  mistral-medium → data/structured/    (toc+sections, page classes,
                                                                  exercises, retag/topics)
             └─▶ 03_load.py               → data/textbook.db    (SQLite, FTS5)
                   └─▶ mcp_server/  FastMCP (stdio | streamable-http + bearer)  → your chatbot
```

## Requirements

- Python ≥ 3.11, [uv](https://docs.astral.sh/uv/), `poppler-utils`
  (`pdftotext`/`pdfinfo`) — Docker only needed for deployment
- A Mistral API key

## Quickstart (local)

```bash
uv sync
cp .env.example .env          # put your MISTRAL_API_KEY in .env

uv run python pipeline/01_ocr.py                    # OCR the PDF (~$0.12)
uv run python pipeline/02_structure.py toc          # -> units tree incl. sections
uv run python pipeline/02_structure.py classify     # -> per-page metadata
uv run python pipeline/02_structure.py extract      # -> exercises (pilot: --scheda 1)
uv run python pipeline/02_structure.py retag        # -> precise topics + section assignment
uv run python pipeline/03_load.py                   # -> data/textbook.db
```

Every step is **idempotent** (cached artifacts; `--force` to redo). Re-running
`03_load.py` preserves exercise ids, so recorded solutions survive reloads.

### Sanity check

```bash
uv run python scripts/smoke_test.py     # exercises every tool, both transports
uv run python scripts/quality_report.py # audit of the extracted exercises
```

The quality audit currently reports **0 severe / 0 warnings** (3 INFO:
instruction-only exercises that intentionally reference a comic/table on
another page).

## Database

SQLite at `data/textbook.db` (WAL mode). **Full reference: [`db/SCHEMA.md`](db/SCHEMA.md).**

| table | content |
|---|---|
| `units` | TOC levels 1–2: 9 topic areas → 12 schede, with book/PDF page ranges |
| `sections` | TOC level 3: 93 printed index entries ("L'imperfetto", p.54) — the anchor for what a page/exercise is about |
| `pages` | all 118 pages: OCR markdown, plain text, header/footer, kind, `section_id` |
| `exercises` | 150 exercises: label, instruction, markdown + structured items, type, **precise topics** (oriented on the index, e.g. "passato prossimo"), CEFR, `section_id` |
| `solutions` | submitted answers; grading fields (`is_correct`, `score`, `feedback`, `graded_by`) are written **by the chatbot**, never by this repo |
| `exercise_progress` | view: attempts, last result, status per exercise |
| `unit_progress` | view: per-scheda rollup + `is_complete` **checkmark** |
| `pages_fts` / `exercises_fts` | FTS5 (trigram) full-text index |

**Checkmarks:** an exercise is `completed` when its latest graded solution is
correct; a scheda's `is_complete` flips when all its exercises are completed
(see `get_progress`, and `is_complete` in `get_toc`).

## MCP server

```bash
# stdio (local agents)
uv run python mcp_server/server.py

# streamable-http on :8000 with bearer auth (server agents)
MCP_TRANSPORT=streamable-http MCP_API_KEY=<token> uv run python mcp_server/server.py
```

Tools: `get_toc` (3 levels incl. sections + checkmarks) · `get_page` ·
`list_exercises` (scheda/topic/type/section/status filters) · `get_exercise`
(incl. solution history) · `search` (Italian FTS) · `submit_solution` (record
answer ± grading) · `get_progress` (per scheda + total, with `is_complete`)

Local client config (stdio), e.g. `opencode.json` / Claude Desktop:

```json
{
  "mcpServers": {
    "textbook": {
      "command": "uv",
      "args": ["run", "python", "mcp_server/server.py"],
      "cwd": "/path/to/textbook-agent-mcp"
    }
  }
}
```

## Deploy to Hetzner ("one package")

Prereq: a domain with an **A record → VM IP** (Caddy needs it for HTTPS).

```bash
# on the VM
git clone <repo> && cd textbook-agent-mcp
scp data/textbook.db vm:.../textbook-agent-mcp/deploy/textbook.db   # ship the db
cp deploy/.env.example deploy/.env      # set DOMAIN + MCP_API_KEY (openssl rand -hex 32)
docker compose -f deploy/docker-compose.yml up -d --build
```

That's it: one compose file, two services (`app` = MCP server,
`caddy` = automatic Let's Encrypt HTTPS). The db is a bind-mounted file —
update content later by `scp`-ing a new `textbook.db` and restarting `app`.
The PDF, pipeline and OCR artifacts never leave your machine.

### Register the connector in Mistral (Vibe Work)

In Mistral Studio → Connectors (or via API), with the static-header auth:

```python
await client.beta.connectors.create_async(
    name="textbook",
    server="https://<your-domain>/mcp",
    visibility="private",
    headers={"Authorization": "Bearer <MCP_API_KEY>"},
)
```

Validate with the Studio **Connectors Debugger**, then use the connector in
Vibe Work conversations.

## Data notes & limitations

- **Exercise labels** mirror the printed numbers. The book's numbering has
  quirks (a few duplicates/unnumbered); the loader resolves them
  deterministically: duplicates become `3`, `3a`, `3b`, unnumbered ones
  `x<page>-<n>`. Missing numbers are recovered from the text layer where
  possible; 2 labels were hand-corrected in the artifacts (see git history
  of this section / loader warnings).
- **Topics are controlled**: they derive from the printed index sections
  (e.g. "scelta dell'ausiliare"), not free-form guesses. 4 exercises are
  intentionally section-less (cross-scheda vocabulary reviews).
- Printed book page = PDF page + 2 (verified against OCR footers on ~100
  pages). Vocabulary boards (*LA SCUOLA*, …) are attached to their scheda.
- The book has **no answer key** — `solutions` only stores what agents/users
  submit.
- Pipeline artifacts in `data/structured/` are human-inspectable JSON; fix
  anything by editing them and re-running `03_load.py`.
- FTS uses trigram matching (no Italian stemming): queries need ≥ 3 chars.
- Total pipeline cost: **< $1** (OCR ≈ $0.12 + structuring tokens).

### If it ever outgrows SQLite

Schema is small and standard SQL; migrate with a short script or `pgloader`.
The only SQLite-isms: FTS5 trigram (→ Postgres `pg_trgm` or `tsvector
'italian'`) and JSON-as-TEXT columns (→ `JSONB`).