# corpus-crm

A personal message-archive pipeline. It ingests your own exports from several
platforms, resolves the same human across them, builds a local
[DuckDB](https://duckdb.org) corpus, and serves ~30 read-only views over it —
plus a small CRM layer for annotating people and an
MCP (Model Context Protocol) server so an LLM can query the corpus.

Everything runs locally. Nothing is uploaded anywhere.

## Try it without any data

A fully synthetic demo dataset ships in `pipeline/output-demo/`:

```bash
npm install
DATA_DIR=./pipeline/output-demo node pipeline/serve.js
# http://127.0.0.1:8765
```

Or:

```bash
docker build -t corpus-crm . && docker run -p 8080:8080 corpus-crm
```

**The demo is complete.** A synthetic corpus ships too
(`pipeline/output-demo/raw/messages.duckdb`, ~4.6 MB — 994 messages across 13
invented people, with photos, places, events, calls and birthdays), so every
view renders, including the ones that query the database. Regenerate it with
`npm run demo:build-db`.

None of it is real: the cast, the messages, the phone numbers and the cities
are all generated by `scripts/generate-demo-data.py` and
`scripts/generate-demo-db.mjs` from a fixed seed.

## Building a real corpus

You supply the exports; nothing is scraped on your behalf:

| Source | What you need |
|---|---|
| iMessage | `chat.db` from a Mac |
| WhatsApp | an iOS backup, or the live bridge (see caveat) |
| Instagram / Messenger | Meta "Download Your Information", JSON format |
| Gmail | Google Takeout `.mbox` |
| Photos | Apple Photos or Google Takeout |

Then:

```bash
npm run build-db     # ingest -> identity resolution -> DuckDB corpus
npm run release      # publish it as an immutable, activated release
npm run serve
```

`npm run release` stages the build into `pipeline/releases/<timestamp>-<sha>`,
verifies the database checksum, refuses to publish a corpus that went
*backwards* (fewer messages than the live one), then swaps the
`pipeline/output` symlink atomically so a running server never sees a
half-written corpus. If the new release fails its health check it rolls back
to the previous one by itself. Set `RESTART_CMD` if you run the viewer as a
service; leave it unset and restart by hand.

Old releases accumulate — `npm run release:prune` shows what it would delete,
`npm run release:prune:apply` does it.

Set `SELF_NAME` to your own display name — identity resolution uses it to
recognise you in your own threads.

## Automating the exports

Two of the sources can be kept topped up without you clicking through export
flows every week. Both are macOS-oriented and both are optional — the manual
export path in the table above always works.

### Meta (Facebook + Instagram)

`pipeline/meta-dyi.mjs` drives Meta's "Download Your Information" flow in a
real browser. Three subcommands:

```bash
node pipeline/meta-dyi.mjs setup     # once: log in, by hand, in a headed window
node pipeline/meta-dyi.mjs request   # ask Meta to build an export
node pipeline/meta-dyi.mjs poll      # download anything that is ready, and unzip it
```

`setup` opens Google Chrome, waits while you sign in to **both**
facebook.com and instagram.com, and keeps the session in a persistent profile
at `~/.meta-dyi-profile`. Nothing else works until that exists.

`request` asks only for what you are missing: it reads the newest message
already in your corpus per source and picks the smallest date-range preset
that covers the gap, so a weekly run fetches a week rather than your whole
history. It skips any profile requested in the last seven days, so running it
by hand is harmless.

`poll` downloads every export Meta has finished into `META_DYI_DROPS_DIR`,
unzips it into `META_DYI_INPUTS_DIR`, and links it where the ingesters expect
it. Meta takes hours to days to build an export, so `poll` finding nothing is
the normal case.

Schedule both (Mondays 04:00 request, daily 04:30 poll):

```bash
node pipeline/install-meta-launchd.js install     # macOS launchd
node pipeline/install-meta-launchd.js uninstall
```

| Variable | Meaning |
|---|---|
| `META_DYI_DROPS_DIR` | where downloaded `.zip` exports land |
| `META_DYI_INPUTS_DIR` | where they are unzipped for ingest |
| `META_DYI_PASSWORD` | Meta re-asks for your password mid-flow; supply it here, or store it in the macOS Keychain as service `meta-dyi` |
| `META_DYI_PROFILE` | `facebook` or `instagram` — run just one |
| `META_DYI_DRYRUN=1` | configure the export but do not submit it |
| `META_DYI_CHANNEL` | `chromium` instead of system Chrome |

Needs `playwright` and Google Chrome, so install dev dependencies
(`npm install`, not `npm ci --omit=dev`).

**It will break.** This automates a website Meta redesigns without warning;
when a selector moves, the run fails and writes a screenshot next to the logs
so you can see which step lost its footing. Treat it as a convenience, not
infrastructure — and if a run fails, the manual export flow still works.

### iMessage

There is no automation here, and deliberately so: reading the message
database is a Full Disk Access grant, not something to hide inside a cron
job. `pipeline/ingest/imessage.js` parses the output of
[imessage-exporter](https://github.com/ReagentX/imessage-exporter):

```bash
brew install imessage-exporter
# System Settings → Privacy & Security → Full Disk Access → add your terminal
imessage-exporter -f txt -o inputs/imessage
npm run build-db
```

One file per conversation in `inputs/imessage/`, named by phone number or
email. Re-running the exporter and rebuilding is the whole update loop.


## MCP server

`pipeline/mcp/` exposes the corpus to an LLM over MCP (Model Context
Protocol) — nine tools: `resolve_person`, `search_messages`,
`get_conversation_window`, `person_summary`, `on_this_day`, `query` (read-only
SQL), `schema`, `log_interaction`, `set_follow_up`.

It speaks stdio by default. Point a client at it:

```json
{ "command": "node",
  "args": ["pipeline/mcp-server.js"],
  "env": { "MCP_DUCKDB_PATH": "pipeline/output/raw/messages.duckdb" } }
```

`npm run mcp:demo` runs it against the synthetic corpus, so you can try the
tools before pointing it at anything of your own. Set `MCP_HTTP_PORT` and
`MCP_TOKEN` for Streamable HTTP instead of stdio; it binds `127.0.0.1` only
unless you also set `MCP_BIND_TAILNET=1`.

The `query` tool accepts a single read-only `SELECT`/`WITH`. Everything the
server carries out of the database passes through `pipeline/lib/redactions.js`
— add patterns there for topics you never want leaving the archive.

## Layout

```
pipeline/ingest/      one module per source
pipeline/normalize/   identity resolution, merging, folding
pipeline/lib/         the view modules (atlas, weeks, braid, almanac, field…)
pipeline/serve.js     the viewer — every route in one file
pipeline/crm-server/  annotation store (SQLite)
pipeline/mcp/         MCP server exposing the corpus to an LLM
pipeline/output-demo/ synthetic demo data (JSON artifacts + a DuckDB corpus)
scripts/release.sh    publish a build as an activated release
scripts/generate-demo-*  regenerate the synthetic demo dataset
```

## Single machine

This runs entirely on one machine. Earlier versions pushed releases to a
second always-on host over SSH; that has been removed — `release.sh` does the
staging, verification and atomic swap locally.

## Caveats

- **macOS-leaning.** The core ingest → build → serve path is portable;
  thumbnailing, notifications and service installation are macOS-only.
- **The WhatsApp live bridge** (`pipeline/whatsapp-baileys/`) pairs as a
  companion device through [Baileys](https://github.com/WhiskeySockets/Baileys),
  an unofficial WhatsApp Web client. This is against WhatsApp's terms of
  service and accounts have been banned for it. It ships because archiving
  your own conversations is a legitimate thing to want and there is no
  official export worth the name — but the risk is yours, and it is real. The
  other ingesters all read official exports and carry no such risk.
- **Requires Node 22+.** Native modules (`better-sqlite3`, `@duckdb/node-api`)
  build on install.
- This is a personal instrument published as a reference, not a supported
  product. Expect rough edges.

## Privacy

An archive like this is the most sensitive data you own, and most of it is
about people who never opted in. `.gitignore` is deliberately aggressive:
raw exports, the database, portraits, face labels and annotations are all
excluded. Keep it that way. Nothing in this repository contains real
personal data — every name in the tests and demo data is fictional.

## License

[PolyForm Noncommercial 1.0.0](./LICENSE.md) — any noncommercial use is
permitted, including personal, hobby, research and study use. Commercial use
is not granted.