corpus-crm

A personal message-archive pipeline. It ingests your own exports from several platforms, resolves the same human across them, builds a local DuckDB corpus, and serves ~30 read-only views over it — plus a small CRM layer for annotating people and an MCP (Model Context Protocol) server so an LLM can query the corpus.

Everything runs locally. Nothing is uploaded anywhere.

Try it without any data

A fully synthetic demo dataset ships in pipeline/output-demo/:

npm install
DATA_DIR=./pipeline/output-demo node pipeline/serve.js
# http://127.0.0.1:8765

Or:

docker build -t corpus-crm . && docker run -p 8080:8080 corpus-crm

The demo is complete. A synthetic corpus ships too (pipeline/output-demo/raw/messages.duckdb, ~4.6 MB — 994 messages across 13 invented people, with photos, places, events, calls and birthdays), so every view renders, including the ones that query the database. Regenerate it with npm run demo:build-db.

None of it is real: the cast, the messages, the phone numbers and the cities are all generated by scripts/generate-demo-data.py and scripts/generate-demo-db.mjs from a fixed seed.

Related MCP server: Outlook MCP Server

Building a real corpus

You supply the exports; nothing is scraped on your behalf:

Then:

npm run build-db     # ingest -> identity resolution -> DuckDB corpus
npm run release      # publish it as an immutable, activated release
npm run serve

npm run release stages the build into pipeline/releases/<timestamp>-<sha>, verifies the database checksum, refuses to publish a corpus that went backwards (fewer messages than the live one), then swaps the pipeline/output symlink atomically so a running server never sees a half-written corpus. If the new release fails its health check it rolls back to the previous one by itself. Set RESTART_CMD if you run the viewer as a service; leave it unset and restart by hand.

Old releases accumulate — npm run release:prune shows what it would delete, npm run release:prune:apply does it.

Set SELF_NAME to your own display name — identity resolution uses it to recognise you in your own threads.

Automating the exports

Two of the sources can be kept topped up without you clicking through export flows every week. Both are macOS-oriented and both are optional — the manual export path in the table above always works.

Meta (Facebook + Instagram)

pipeline/meta-dyi.mjs drives Meta's "Download Your Information" flow in a real browser. Three subcommands:

node pipeline/meta-dyi.mjs setup     # once: log in, by hand, in a headed window
node pipeline/meta-dyi.mjs request   # ask Meta to build an export
node pipeline/meta-dyi.mjs poll      # download anything that is ready, and unzip it

setup opens Google Chrome, waits while you sign in to both facebook.com and instagram.com, and keeps the session in a persistent profile at ~/.meta-dyi-profile. Nothing else works until that exists.

request asks only for what you are missing: it reads the newest message already in your corpus per source and picks the smallest date-range preset that covers the gap, so a weekly run fetches a week rather than your whole history. It skips any profile requested in the last seven days, so running it by hand is harmless.

poll downloads every export Meta has finished into META_DYI_DROPS_DIR, unzips it into META_DYI_INPUTS_DIR, and links it where the ingesters expect it. Meta takes hours to days to build an export, so poll finding nothing is the normal case.

Schedule both (Mondays 04:00 request, daily 04:30 poll):

node pipeline/install-meta-launchd.js install     # macOS launchd
node pipeline/install-meta-launchd.js uninstall

Needs playwright and Google Chrome, so install dev dependencies (npm install, not npm ci --omit=dev).

It will break. This automates a website Meta redesigns without warning; when a selector moves, the run fails and writes a screenshot next to the logs so you can see which step lost its footing. Treat it as a convenience, not infrastructure — and if a run fails, the manual export flow still works.

iMessage

There is no automation here, and deliberately so: reading the message database is a Full Disk Access grant, not something to hide inside a cron job. pipeline/ingest/imessage.js parses the output of imessage-exporter:

brew install imessage-exporter
# System Settings → Privacy & Security → Full Disk Access → add your terminal
imessage-exporter -f txt -o inputs/imessage
npm run build-db

One file per conversation in inputs/imessage/, named by phone number or email. Re-running the exporter and rebuilding is the whole update loop.

MCP server

pipeline/mcp/ exposes the corpus to an LLM over MCP (Model Context Protocol) — nine tools: resolve_person, search_messages, get_conversation_window, person_summary, on_this_day, query (read-only SQL), schema, log_interaction, set_follow_up.

It speaks stdio by default. Point a client at it:

{ "command": "node",
  "args": ["pipeline/mcp-server.js"],
  "env": { "MCP_DUCKDB_PATH": "pipeline/output/raw/messages.duckdb" } }

npm run mcp:demo runs it against the synthetic corpus, so you can try the tools before pointing it at anything of your own. Set MCP_HTTP_PORT and MCP_TOKEN for Streamable HTTP instead of stdio; it binds 127.0.0.1 only unless you also set MCP_BIND_TAILNET=1.

The query tool accepts a single read-only SELECT/WITH. Everything the server carries out of the database passes through pipeline/lib/redactions.js — add patterns there for topics you never want leaving the archive.

Layout

pipeline/ingest/      one module per source
pipeline/normalize/   identity resolution, merging, folding
pipeline/lib/         the view modules (atlas, weeks, braid, almanac, field…)
pipeline/serve.js     the viewer — every route in one file
pipeline/crm-server/  annotation store (SQLite)
pipeline/mcp/         MCP server exposing the corpus to an LLM
pipeline/output-demo/ synthetic demo data (JSON artifacts + a DuckDB corpus)
scripts/release.sh    publish a build as an activated release
scripts/generate-demo-*  regenerate the synthetic demo dataset

Single machine

This runs entirely on one machine. Earlier versions pushed releases to a second always-on host over SSH; that has been removed — release.sh does the staging, verification and atomic swap locally.

Caveats

• macOS-leaning. The core ingest → build → serve path is portable; thumbnailing, notifications and service installation are macOS-only.

• The WhatsApp live bridge (pipeline/whatsapp-baileys/) pairs as a companion device through Baileys, an unofficial WhatsApp Web client. This is against WhatsApp's terms of service and accounts have been banned for it. It ships because archiving your own conversations is a legitimate thing to want and there is no official export worth the name — but the risk is yours, and it is real. The other ingesters all read official exports and carry no such risk.

• Requires Node 22+. Native modules (better-sqlite3, @duckdb/node-api) build on install.

• This is a personal instrument published as a reference, not a supported product. Expect rough edges.

Privacy

An archive like this is the most sensitive data you own, and most of it is about people who never opted in. .gitignore is deliberately aggressive: raw exports, the database, portraits, face labels and annotations are all excluded. Keep it that way. Nothing in this repository contains real personal data — every name in the tests and demo data is fictional.

License

PolyForm Noncommercial 1.0.0 — any noncommercial use is permitted, including personal, hobby, research and study use. Commercial use is not granted.