# esp32-docs-mcp

Semantic search over ESP32 documentation, exposed as an
[MCP](https://modelcontextprotocol.io) server — so a coding agent can look things
up in the ESP-IDF guides, the API reference, and the chip Technical Reference
Manuals while it works.

Everything runs locally. Embeddings are computed on-device with MLX (Apple
Silicon) and stored in LanceDB; no documentation or query text leaves the machine.

The current corpus is **25,362 chunks** — 11,157 from ESP-IDF across nine build
targets, 10,515 from ten Technical Reference Manuals, and 3,690 from the SoC
headers that define the registers those manuals describe.

## Why this exists

Pointing a retrieval system at ESP-IDF's `.rst` sources produces confidently
wrong answers, because **ESP-IDF documentation is built, not written.** The build
is where the facts come from:

- `{IDF_TARGET_CONFIG_BOOTLOADER_OFFSET_IN_FLASH}` and its siblings are
  substituted at build time from the chip's `soc_caps` headers and Kconfig. Parse
  the source and you embed the placeholder instead of the address.
- `only::` blocks select per-chip branches. Parse the source and mutually
  exclusive branches for nine different chips collapse into one self-contradictory
  passage.
- The entire API reference is generated by doxygen from C headers, and does not
  exist in the source tree at all.

So this project builds the docs once per chip and indexes the build output.
Constants are real values, chip-specific claims are correct per chip, and
function signatures, parameters, structs and enums are all present.

Chunks byte-identical across every target are stored once and tagged with every
chip they appeared under, so filtering to a chip narrows results to what is true
for it without hiding general content. That collapses 43,274 per-target chunks to
11,157 unique (74.2%) while keeping every genuine per-chip difference — a chunk
that differs by one substituted constant stays a separate, correctly narrower row.

The Technical Reference Manuals come from Espressif's
[public LaTeX sources](https://github.com/espressif/esp-technical-reference-manual-latex)
rather than the compiled PDFs, at 99.9% register capture. Registers are indivisible:
name, address and every bitfield stay in one chunk. Where Espressif publishes more
than one manual for a chip — ESP32-P4 has both a mainline TRM and a "Chip Revision
v1.3" one, whose register sets diverge in both directions — both are ingested and
each result states which silicon it applies to.

## Requirements

- macOS on Apple Silicon (MLX)
- Python 3.12+ and [`uv`](https://docs.astral.sh/uv/)
- To build the ESP-IDF half of the corpus: an ESP-IDF checkout installed **with
  the docs feature**, plus `doxygen`
- To build the TRM half: a clone of Espressif's TRM LaTeX repo

There is no prebuilt index to download. See
[docs/building-the-corpus.md](docs/building-the-corpus.md) — budget roughly 20
minutes of docs build per chip and a single embedding pass measured in hours.

## Install

```bash
git clone https://github.com/ozonejunkieau/esp32-docs-mcp.git
cd esp32-docs-mcp
uv sync
```

## Quickstart

Build the corpus once ([full instructions](docs/building-the-corpus.md)), confirm
it, then register the server:

```bash
uv run validate_store.py     # row count, doc_type breakdown, search round-trip
uv run mcp_server.py         # speaks stdio; Ctrl-C to stop
```

If you have [`just`](https://just.systems) (>= 1.43), every command in the
documentation has a recipe — `just` on its own lists them, grouped, and
`just --list` records what a healthy result from each check looks like.

Register with Claude Code:

```bash
claude mcp add esp32-docs -- uv run --directory /path/to/esp32-docs-mcp mcp_server.py
```

Or in `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "esp32-docs": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/esp32-docs-mcp", "mcp_server.py"]
    }
  }
}
```

The embedding model and database connection load once at startup and are reused,
so the first request pays the model load and subsequent queries are fast.
[docs/usage.md](docs/usage.md) covers what good queries look like and how to read
the results.

## Tools

### `esp32_docs_search`

| Parameter | Description |
|---|---|
| `query` | Natural-language query, e.g. "how does I2S clock configuration work" |
| `doc_type` | `trm` (manuals), `idf` (guides and API reference), `src` (SoC headers), or omit for all three |
| `chip` | e.g. `esp32p4`. Narrows to what's true for that chip; content common to all chips still matches |
| `revision` | Silicon revision, e.g. `v1.3` or `mainline`. Narrows any content that has a revision — the manuals, and ESP32-P4's SoC register headers. Content without one is unaffected. Omit to see every revision |
| `k` | Results to return (1–20, default 5) |

Returns JSON. Each result carries its text plus `source_doc`, `section_path`,
`file_path` and `chunk_index` for citation, `relevance_distance` (lower is
closer), and three reference lists for follow-up lookups: `file_refs` (ESP-IDF
source paths), `doc_refs` (other doc pages), `symbol_refs` (C/C++ symbols).

Two fields matter when accuracy does:

- **`revision_scope`** — `null` for ESP-IDF content, `"all published revisions
  (…)"` for TRM content true of every stepping, or `"ONLY revision X …"` where it
  isn't. Applying a revision-specific register definition to the wrong silicon is
  a hardware bug, and this field says which case you have without you having to
  reason about a list.
- **`source_version`** — the upstream revision the chunk was built from. Both
  corpora track moving upstreams, so this is what makes an answer reproducible.

### `esp32_docs_find_symbol`

Exact lookup of a C/C++ identifier — a register, a bitfield, a function, a type —
across every corpus that names it. Use it instead of `esp32_docs_search` whenever
you already know the identifier: symbol lookup is exact, not semantic, and it
skips the query embedding entirely, so it answers in well under a tenth of a
second.

| Parameter | Description |
|---|---|
| `symbol` | Exact identifier, e.g. `I2C_SCL_LOW_PERIOD_REG`. `foo` and `foo()` are treated the same |
| `doc_type`, `chip` | Optional, same meaning as in search |
| `k` | Results to return (default 5) |

This is the cross-corpus join: one call returns the SoC header defining a
register's address and bitmasks *and* the manual chapter explaining what it does.
Results are interleaved across corpora rather than truncated arbitrarily —
without that, one corpus vanished from the top `k` for 37% of the ~20,000 symbols
that appear in more than one, which defeats the point of the lookup.

The response is an object rather than a bare array, so truncation is visible:

```json
{
  "symbol": "LEDC_CH0_CONF0_REG",
  "total_matches": {"all": 18, "src": 10, "trm": 8},
  "returned": 5,
  "results": [ ... ]
}
```

Raise `k` when `total_matches` shows more than you were given.

### `esp32_docs_list_chips`

Valid `chip` values with their coverage: `has_idf_docs`, `has_trm`, and
`revisions`. Both coverage flags can be false independently — a chip may have a
Technical Reference Manual but no per-chip docs build, or the reverse.

## Status and limitations

All three pipelines work end to end and the store is populated. Known limits,
stated plainly:

- **Apple Silicon only.** Embedding goes through MLX. Nothing else is
  platform-specific, but there is no fallback backend.
- **You must build the corpus yourself.** The LanceDB store is derived data and
  is not committed; the ESP-IDF half additionally needs a full toolchain install
  per target, because the docs build runs `idf.py set-target` to extract the
  constants.
- **The corpus is a snapshot.** ESP-IDF docs change weekly and the TRM sources
  run ahead of the published PDFs. Every row records `source_version` and
  `source_commit`; refreshing is a manual re-run.
- **Coverage is uneven by design.** Nine of the thirteen known chips have an
  ESP-IDF docs build; ten have a published TRM. `esp32_docs_list_chips` reports
  which.
- **Diagrams are not recoverable.** Block, timing and `bytefield` diagrams
  survive only as their captions. No register bit layout was found to exist
  *only* as a figure, but the diagrams themselves are absent.
- **TRM chunks have no `doc_refs`.** Cross-reference extraction is implemented
  for the ESP-IDF corpus only; it is deferred, not broken.
- **A register census shortfall is not automatically content loss.** Some source
  registers are disabled upstream or tagged for a different chip, and the parser
  is right to drop them while an independent checker may still count them. Read
  the named registers before treating a shortfall as a regression — see
  [docs/trm-latex.md](docs/trm-latex.md#verification).

## Documentation

| Document | For |
|---|---|
| [docs/usage.md](docs/usage.md) | Wiring the server in, querying it, reading results, troubleshooting |
| [docs/building-the-corpus.md](docs/building-the-corpus.md) | Building and refreshing the index from source |
| [docs/architecture.md](docs/architecture.md) | Why the pipelines are shaped this way; what was tried and deleted |
| [docs/trm-latex.md](docs/trm-latex.md) | TRM LaTeX ingest — design record and verification |
| [docs/development.md](docs/development.md) | The `justfile`, the tests, the fixtures |
| [CLAUDE.md](CLAUDE.md) | Operating guide: invariants, environment traps, commands, current state |

## Layout

| File | Purpose |
|---|---|
| `mcp_server.py` | The MCP server and its two tools |
| `build_idf_docs.sh` | Builds ESP-IDF docs to XML for one or more chips |
| `sphinx_xml.py` | Turns a built page's XML into chunks |
| `chunking.py` | Heading-aware chunk assembly, shared by both corpora |
| `ingest_sphinx_xml.py` | Chunks one target's build into JSONL |
| `dedup_chunks.py` | Collapses chunks identical across chips |
| `embed_and_store.py` | Embeds chunks into LanceDB |
| `embedder.py` | Qwen3-Embedding-4B via MLX (2560 dims) |
| `schema.py` | The stored chunk schema |
| `chips.yaml`, `chip_vocab.py` | Verified chip vocabulary and doc coverage |
| `latex_parser.py`, `latex_coverage_check.py` | TRM LaTeX parsing and macro-coverage reporting — see [docs/trm-latex.md](docs/trm-latex.md) |
| `ingest_trm.py` | Chunks the TRM manuals, deduplicated across silicon revisions |
| `ingest_source.py` | Chunks the ESP-IDF SoC headers (`doc_type = "src"`) |
| `provenance.py`, `backfill_provenance.py` | Record/repair the upstream revision each chunk came from |
| `check_thin_files.py` | Thin-file and capture-rate checks (`xml` / `latex`) |
| `register_census.py`, `trm_verify.py` | Check TRM registers survive into chunks |
| `make_trm_fixture.py` | Synthetic corpora that prove the checkers work |
| `validate_store.py` | LanceDB row count, samples, search round-trip |
| `justfile` | Every documented command as a recipe; `just --list` |
| `tests/` | `uv run pytest`; `slow` tests need a local corpus and skip without one |

## License

The code in this repository is MIT — see [LICENSE](LICENSE).

### Upstream licensing of the corpus

**The licence above covers this software, not the documentation it indexes.**
Building the corpus produces a store containing verbatim documentation text, so
redistributing that store means redistributing Espressif's documentation under
its own terms — not this project's.

| Source | Licence |
|---|---|
| [ESP-IDF](https://github.com/espressif/esp-idf) documentation | Apache-2.0 |
| [TRM LaTeX sources](https://github.com/espressif/esp-technical-reference-manual-latex) | **CC-BY-SA 4.0** (the repo's scripts are Apache-2.0) |

The practical consequence is **CC-BY-SA is copyleft**. Roughly half the corpus —
10,515 of 25,362 chunks — comes from the Technical Reference Manuals, so a built
store carries a ShareAlike obligation and cannot simply be republished under a
permissive licence. If you publish a store or a derived dataset:

- Keep the two corpora as separate artifacts. A combined store is more readily
  argued to be adapted material, where ShareAlike propagates; separable datasets
  are more clearly a collection.
- Licence the TRM artifact **CC-BY-SA 4.0** and the ESP-IDF artifact
  **Apache-2.0**, each attributing Espressif and naming the upstream commit. The
  store already records `source_version` and `source_commit` on every row, so the
  exact provenance is available per chunk.
- Preserve attribution and copyright notices in the indexed text. ESP-IDF's
  `COPYRIGHT` document carries third-party attributions, including contributor
  email addresses, that the Apache-2.0 and BSD terms require be retained — they
  are not incidental content to strip.

Nothing here is legal advice, and whether a mixed store counts as a collection or
an adaptation turns on specifics. Get a real opinion before relying on it
commercially.

This project distributes no corpus. Everything is built locally from sources you
obtain yourself, which is why there is no prebuilt index to download.