Skip to main content
Glama
README.md
# pivd — personal information vector database

Your documents are read **once by an agent**, not by a parser. The agent looks at the pages,
writes down what matters as structured entries (claim + evidence, method, result, reference…)
with the page they came from. pivd embeds each entry on write, and later finds it by meaning.

```console
$ pivd search "why does pump noise leak into the Stokes wave"
1. /home/hiti/Research/sim/velpu_sim/literature/R04_ma_pump_to_stokes.pdf
      p1 claim  claim: Low-frequency pump-to-Stokes RIN transfer can be suppressed to
               below 1 dB by pumping a low-loss waveguide at high intensity | evidence: …
      p1 claim  claim: A conversion efficiency of 70% is attainable in mid-IR silicon
               Raman lasers | evidence: Abstract (Optics Express 20(16), p. 17962)
```

No OCR, no PDF text extraction, no chunking heuristics, no torch, no long-running server (the MCP
server runs on stdio when an agent starts it). SQLite + `sqlite-vec` + FTS5 + EmbeddingGemma-2
(768d, int8 ONNX, 314 MB, CPU).

## Quickstart

```console
uv venv --python 3.13 .venv && uv pip install -p .venv -e .     # sqlite-vec needs loadable sqlite
export PATH=$PWD/.venv/bin:$PATH

pivd db create papers && pivd use papers
pivd profile --set profiles/research_article.toml     # decide what to record
pivd pages ~/Research/Pdfs/shg/paper.pdf --pages 1-3  # PNGs for an agent (or your own eyes)

pivd put --kind claim --path ~/Research/Pdfs/shg/paper.pdf --page 2 \
         --field claim="soliton fission sets the blue edge" \
         --field evidence="Fig. 3 and the dispersion profile" --tag supercontinuum

pivd search "what sets the blue edge of a supercontinuum"
pivd entries ~/Research/Pdfs/shg/paper.pdf     # what is already recorded
pivd open ~/Research/Pdfs/shg/paper.pdf --page 2
```

Batch writes (what an agent does after reading a paper):

```console
pivd put --json < entries.json     # [{"kind","fields","path","page","tags"}]
```

`put` refuses entries whose document does not exist, whose `page` is not a positive integer, or
whose field values are not strings (≤ 2000 chars each, ≤ 4000 per entry). Re-putting the same
entries is a no-op. To redo a document:
`pivd rm --path <pdf>` then `pivd put …` again.

## Figures are stored, not just described

A schematic is only useful if you can look at it again. Crop it, attach it, describe what it
shows:

```console
pivd pages paper.pdf --pages 3 --clip 150,418,472,545 --out /tmp/figs   # x0,y0,x1,y1 in points
pivd put --kind result --path paper.pdf --page 3 \
         --field result="TE modes at 2.88 and 3.39 um overlap in the SOS rib" \
         --field numbers="Gamma = 99.75%" --image /tmp/figs/paper_p3_crop.png

pivd search "rib cross-section mode overlap" --images   # prints the entry AND the PNG path
pivd image 42 --out /tmp/look.png                       # or --base64
```

The crop is stored inside the collection (PNG/JPEG/WebP, ≤ 8 MB), so it survives the PDF being
moved, and the entry is found by its description. Search marks such entries `[img]`.

## As an MCP server

```console
pivd serve --all
```

```json
{"mcpServers": {"pivd": {"command": "/home/hiti/Hiti/projects/collections/.venv/bin/pivd",
                          "args": ["serve", "--all"]}}}
```

Tools: `profile` (what to record — read first), `pages` (PDF → PNG paths, optional `pages`/
`clip`/`dpi`), `put` (validated, embedded, idempotent, accepts `image`/`image_base64`),
`search` (`with_images`), `image`, `entries`, `list_databases`, `stats`. When the server exposes
several collections, every tool requires `db=<name>` — it never picks one silently.

## Teaching an agent to use it

```console
pivd skill              # print the skill (indexing loop, entry rules, failure modes)
pivd skill --install    # write it to ~/.agents/skills/pivd/SKILL.md
```

The skill tells an agent the loop (`profile` → `pages` → read the PNGs → `put` → `search`), how
to write entries a future query will match, and what to do when `put` rejects one.

## The profile decides the schema

`profiles/research_article.toml` names the entry kinds and their required/optional fields;
`pivd put` rejects entries that do not match. Write a different profile for datasheets,
contracts or notes — same store, different vocabulary.

## How it behaves

| | |
|---|---|
| write | entry → validate → embed (768d) → row + vector + FTS, one transaction |
| find | dense KNN + BM25 fused (RRF), grouped per file, page + fields shown |
| space | ≈ 4 KB per text entry; an attached figure adds its PNG size (10,000 entries ≈ 40 MB) |
| speed | ~30 ms per entry warm, search ≈ 10 ms + model load |
| truth | every result is a path + page; the entry is the indexer's words, the page is ground truth |
| reads | commands that only read open the file read-only: no table is created in a foreign `.db` |
| cache | rendered pages and extracted crops are pruned to the newest 300 files per directory |

## License

MIT — see `LICENSE`.

## Tests

```console
.venv/bin/python tests/test_smoke.py
```