# regcite

Machine-checked register data from chip datasheet PDFs, served to AI coding
agents over MCP.

- LLMs hallucinate register values when they write firmware, regcite makes
  them look the values up instead.
- Deterministic parsers (no AI) extract register tables from vendor PDFs;
  five validator families check every register.
- Six MCP tools do the lookup and the bit math: `get_register`, `decode`,
  `encode`, `check_errata`, `search_doc`, `init_sequence`.
- Every answer cites its PDF page and document hash; anything unverified is
  refused, never guessed.
- Graded against vendor ground truth (RP2040: 949/949 registers), enforced
  in CI. Five chips ingested; Python + SQLite; MIT.

What that buys a model, measured — 30 register questions (lookups, bit
math, and traps that deserve a refusal) against local Qwen2.5 models, with
and without the server attached:

|                | no tools | with regcite |
|----------------|---------:|-------------:|
| Qwen2.5-3B     |     0/30 |        22/30 |
| Qwen2.5-7B     |     2/30 |        29/30 |

The bare models answered every trap with a confident fabrication; with the
server attached, no arm ever asserted a wrong hex value. `abtest/`
reproduces this against your own local models.

## Problem

Language models write firmware from memory. A recalled register address or
bit value is often close and wrong, and the resulting code compiles and
returns bad measurements. The true values exist in datasheet PDFs, in
tables that language models and plain PDF search both read poorly.

regcite extracts those tables with deterministic parsers, checks the result
mechanically, stores it with provenance, and exposes lookup and arithmetic
tools through MCP. The model writing firmware calls the tools. The tools
answer from checked data or refuse.

## Pipeline

1. **Fetch.** `python -m regcite.fetch <PART>` downloads the PDF named in
   `manifest/parts.json` and verifies its SHA-256. A download that is short
   or lacks a PDF header is discarded before it can poison the cache.
2. **Extract.** Deterministic parsers read the register tables. Each
   manufacturer lays tables out differently, so each has its own parser
   (see Grammars below). Rows are recovered from cell ruling lines and
   columns from header words. Cell text comes from raw positioned words,
   because the PDF library's own table extractor mangles underscored names.
3. **Validate.** Five check families run on every register. Fields must tile
   the register without overlap, reset values must fit and reconstruct,
   access modes must be legal, addresses must sit in the declared range
   without collisions, and enum values must fit their fields. A failing
   register is stored as `unverified` and the tools refuse to compute with
   it. Each register also records which checks actually ran, because some
   formats omit the data a check needs.
4. **Store.** One SQLite file holds registers, fields, enums, init
   sequences, errata, and prose chunks. Every row keeps the part, document,
   page, and hash it came from.
5. **Serve.** An MCP server exposes the tools below. Registered with Claude
   Code through the repo's `.mcp.json`.

A vision-model pass exists for registers the parsers could not verify. It
renders the PDF page, asks a local Ollama model for schema-checked JSON,
rejects any answer naming a different register, and runs the same
validators on what remains. Output that passes is stored as `vlm_verified`.
Output that fails is discarded and the deterministic data stays in place.

## Tools

| tool | returns |
|------|---------|
| `get_register(part, name or address)` | fields, enums, reset, access, extraction status, page |
| `decode(part, register, value)` | the value split into named fields with enum names resolved |
| `encode(part, register, {field: value})` | the byte to write, with unset fields kept at reset |
| `check_errata(part, revision?, peripheral?)` | known silicon bugs scoped to a revision |
| `search_doc(part, query)` | ranked prose passages with page numbers |
| `init_sequence(part, mode)` | ordered startup steps with timing, each step cited |

A wrong field name returns the valid field list. An unknown enum returns
the valid values. An unverified register returns its status and the page to
check by hand. `check_errata` with no data loaded says so and states that
absence of data is absent data.

## Verification

Extraction is graded against ground truth the vendors publish. MCU makers
ship SVD files listing every register, so the parsers run on two MCU
reference manuals and the output is diffed against the SVD. The thresholds
run as a regression gate that fails the build when a change lowers them.

| metric | RP2040 (642 pages, table format) | STM32F4 (1,749 pages, prose format) |
|---|---:|---:|
| register recall | 949/949 | 1,334/1,459 (91.4%) |
| register precision | 100% | 81.1% |
| offset accuracy | 100% | 98.1% |
| field precision | 100% | 97.6% |
| field recall | 4,585/4,585 | 97.0% |
| field name accuracy | 100% | 95.7% |
| reset accuracy | 4,367/4,368 | 93.1% |

The one RP2040 reset mismatch is a disagreement between the datasheet and
the vendor's own SVD. The datasheet gives `XOSC.STARTUP.DELAY` a reset of
`0xC4` and the SVD gives `0x0`.

Two caveats on the STM32 column. Duplicate register acronyms in that manual
are resolved by picking the section that agrees best with the SVD, which
makes the field metrics upper bounds, and register precision counts every
unused section against the parser. Most unused sections describe F42x-only
peripherals absent from the F405 SVD. A family rule matches one documented
field spanning a run of single-bit SVD fields, and it never counts toward
name accuracy.

Two more test tiers run above the parser level. Tier 2 is 44 hand-written
questions answered from the datasheets and executed against the tools,
including cases where the correct answer is a refusal. Tier 3 drives a mock
I2C bus with a driver that computes every byte through `encode` and diffs
the write log against hand-derived golden traces, plus a check that a
nonsense setting raises instead of writing. All 44 questions and all 8
writes pass.

## Parts

| part | grammar | registers | checks that ran |
|------|---------|-----------|-----------------|
| BMI270 | bosch_toc | 85/86 verified | all five families (25 of 85 registers have enums to check) |
| BME688 | bosch_content | 46/52 verified | tiling and address (the format carries no reset or access data and no enums) |
| TPS65219 | ti | 57/57 verified | tiling, access, address, and resets on the 9 registers that state one |
| ICM-42688-P | tdk | 108/110 verified | tiling, access, address, enums, register-level reset width |
| SGP41 | prose only | none | command-based part with no register map. `search_doc` covers it |

Grammar means the parser for one vendor's table format. `bosch_toc` reads
field-per-row tables joined against the PDF's own table of contents.
`bosch_content` reads text runs shaped like `ctrl_meas 0x74 mode<1:0>`.
`ti` reads offset anchors plus Bit/Field/Type/Reset tables cross-checked
against a summary table. `tdk` reads `Name:`/`Address:` text blocks across
four register banks, with enum lines attributed to the nearest field label.

BME688 has separate SPI and I2C address maps. The store models one space
per part, so its three SPI-page registers collide with three I2C registers
and all six are marked unverified, the SPI ones with a note saying why. The
two unverified ICM registers are parser defects, queued for the
vision-model pass along with them.

Six more parts wait on PDFs their vendors do not serve to plain HTTP
clients. The manifest and parsers are ready for LSM6DSV16X, MAX17048,
TMC5160, WM8960, SX1262, and nRF52840 once the files are downloaded by
hand.

## Benchmarks

Measured on an RTX 4060 Laptop GPU (8 GB) against the same machine's CPU.

| metric | GPU | CPU |
|--------|-----|-----|
| embedding, 385 prose chunks | 41.4 chunks/s | 1.9 chunks/s |
| `search_doc`, hybrid with rerank, warm median | 527 ms | 11,115 ms |
| `decode` / `encode` | 0.6 ms | 0.8 ms |
| full BMI270 extraction | 5.6 s | 5.8 s |

Prose search runs on FTS5 alone when the embedding stack is absent, and
the response names the mode that produced it. `python -m eval.bench`
reproduces the table.

The model comparison in the introduction comes from `abtest/`: a
four-arm harness (two local model sizes, with and without the MCP server)
over 30 questions whose ground truth is read from the database, with the
full answers and tool traces committed under `abtest/results/`. It needs a
local folder of model checkpoints — set `REGCITE_MODELS_DIR` or
`abtest/models_dir.txt` — and its own torch environment; see
`abtest/README.md`.

## Quickstart

```
python -m venv .venv && .venv/Scripts/pip install -r requirements.txt
python -m regcite.fetch BMI270
python -m regcite.store ingest BMI270
python -m regcite.prose ingest cache/pdfs/bmi270.pdf BMI270
python -m regcite.store seed BMI270 manifest/curated/bmi270.json
python -m regcite.store get BMI270 ACC_CONF
python -m pytest tests/ -q
```

Paths above use the Windows venv layout (`.venv/Scripts/`); on Linux and
macOS substitute `.venv/bin/`, and point `.mcp.json`'s `command` at
`.venv/bin/python`. The unit tests need no PDFs. GPU embeddings need
`pip install sentence-transformers` and a CUDA build of torch from the
index matching your driver.

The MCP server starts from `.mcp.json` inside Claude Code, or by hand with
`python -m regcite.server`.

## Data and redistribution

The repo ships the parsers and a manifest of source URLs with SHA-256
hashes. A fetch script builds your local cache from the manifest. Vendor
PDFs and the built database stay on your machine, because republishing a
corpus derived from someone's documents differs legally from reading
documents you downloaded.

## CI

`.github/workflows/ci.yml` runs lint and the unit tests on every push. A
second job builds the BMI270 database from a fetched PDF and runs Tiers 2
and 3. Both jobs block the build. An encoding test scans every tracked
text file for byte-order marks and mojibake.