pivd
by hitaishi2222
README.md
# pivd — personal information vector database
Your documents are read **once by an agent**, not by a parser. The agent looks at the pages,
writes down what matters as structured entries (claim + evidence, method, result, reference…)
with the page they came from. pivd embeds each entry on write, and later finds it by meaning.
```console
$ pivd search "why does pump noise leak into the Stokes wave"
1. /home/hiti/Research/sim/velpu_sim/literature/R04_ma_pump_to_stokes.pdf
p1 claim claim: Low-frequency pump-to-Stokes RIN transfer can be suppressed to
below 1 dB by pumping a low-loss waveguide at high intensity | evidence: …
p1 claim claim: A conversion efficiency of 70% is attainable in mid-IR silicon
Raman lasers | evidence: Abstract (Optics Express 20(16), p. 17962)
```
No OCR, no PDF text extraction, no chunking heuristics, no torch, no long-running server (the MCP
server runs on stdio when an agent starts it). SQLite + `sqlite-vec` + FTS5 + EmbeddingGemma-2
(768d, int8 ONNX, 314 MB, CPU).
## Quickstart
```console
uv venv --python 3.13 .venv && uv pip install -p .venv -e . # sqlite-vec needs loadable sqlite
export PATH=$PWD/.venv/bin:$PATH
pivd db create papers && pivd use papers
pivd profile --set profiles/research_article.toml # decide what to record
pivd pages ~/Research/Pdfs/shg/paper.pdf --pages 1-3 # PNGs for an agent (or your own eyes)
pivd put --kind claim --path ~/Research/Pdfs/shg/paper.pdf --page 2 \
--field claim="soliton fission sets the blue edge" \
--field evidence="Fig. 3 and the dispersion profile" --tag supercontinuum
pivd search "what sets the blue edge of a supercontinuum"
pivd entries ~/Research/Pdfs/shg/paper.pdf # what is already recorded
pivd open ~/Research/Pdfs/shg/paper.pdf --page 2
```
Batch writes (what an agent does after reading a paper):
```console
pivd put --json < entries.json # [{"kind","fields","path","page","tags"}]
```
`put` refuses entries whose document does not exist, whose `page` is not a positive integer, or
whose field values are not strings (≤ 2000 chars each, ≤ 4000 per entry). Re-putting the same
entries is a no-op. To redo a document:
`pivd rm --path <pdf>` then `pivd put …` again.
## Figures are stored, not just described
A schematic is only useful if you can look at it again. Crop it, attach it, describe what it
shows:
```console
pivd pages paper.pdf --pages 3 --clip 150,418,472,545 --out /tmp/figs # x0,y0,x1,y1 in points
pivd put --kind result --path paper.pdf --page 3 \
--field result="TE modes at 2.88 and 3.39 um overlap in the SOS rib" \
--field numbers="Gamma = 99.75%" --image /tmp/figs/paper_p3_crop.png
pivd search "rib cross-section mode overlap" --images # prints the entry AND the PNG path
pivd image 42 --out /tmp/look.png # or --base64
```
The crop is stored inside the collection (PNG/JPEG/WebP, ≤ 8 MB), so it survives the PDF being
moved, and the entry is found by its description. Search marks such entries `[img]`.
## As an MCP server
```console
pivd serve --all
```
```json
{"mcpServers": {"pivd": {"command": "/home/hiti/Hiti/projects/collections/.venv/bin/pivd",
"args": ["serve", "--all"]}}}
```
Tools: `profile` (what to record — read first), `pages` (PDF → PNG paths, optional `pages`/
`clip`/`dpi`), `put` (validated, embedded, idempotent, accepts `image`/`image_base64`),
`search` (`with_images`), `image`, `entries`, `list_databases`, `stats`. When the server exposes
several collections, every tool requires `db=<name>` — it never picks one silently.
## Teaching an agent to use it
```console
pivd skill # print the skill (indexing loop, entry rules, failure modes)
pivd skill --install # write it to ~/.agents/skills/pivd/SKILL.md
```
The skill tells an agent the loop (`profile` → `pages` → read the PNGs → `put` → `search`), how
to write entries a future query will match, and what to do when `put` rejects one.
## The profile decides the schema
`profiles/research_article.toml` names the entry kinds and their required/optional fields;
`pivd put` rejects entries that do not match. Write a different profile for datasheets,
contracts or notes — same store, different vocabulary.
## How it behaves
| | |
|---|---|
| write | entry → validate → embed (768d) → row + vector + FTS, one transaction |
| find | dense KNN + BM25 fused (RRF), grouped per file, page + fields shown |
| space | ≈ 4 KB per text entry; an attached figure adds its PNG size (10,000 entries ≈ 40 MB) |
| speed | ~30 ms per entry warm, search ≈ 10 ms + model load |
| truth | every result is a path + page; the entry is the indexer's words, the page is ground truth |
| reads | commands that only read open the file read-only: no table is created in a foreign `.db` |
| cache | rendered pages and extracted crops are pruned to the newest 300 files per directory |
## License
MIT — see `LICENSE`.
## Tests
```console
.venv/bin/python tests/test_smoke.py
```
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues