# iMessage RAG MCP

An MCP server that makes your local macOS iMessage history searchable by AI assistants.

It syncs `chat.db` into a local SQLite database, splits conversations into
context-aware chunks, and serves hybrid retrieval (dense + lexical, fused and
reranked) over an MCP endpoint. Everything runs locally — no message data leaves
your machine.

## Features

- **Hybrid retrieval** — FAISS dense vector search fused with TF-IDF lexical
  search via reciprocal rank fusion, then reranked with a cross-encoder.
- **Conversation-aware chunking** — messages are grouped into sessions by time
  gap, then chunked with overlap so retrieved passages stay coherent.
- **Context expansion** — results include surrounding messages, not just the
  matching chunk.
- **Contact name resolution** — phone numbers and emails are mapped to real
  names from your macOS Address Book.
- **Incremental sync** — a fingerprint of the source database avoids redundant
  work when nothing has changed.
- **Local only** — reads Apple's databases read-only; all indexes stay on disk.

## Requirements

- macOS (reads `~/Library/Messages/chat.db`)
- Python 3.10+
- **Full Disk Access** for whichever program runs the server (Terminal, iTerm,
  PyCharm, etc.) — grant it in System Settings → Privacy & Security → Full Disk
  Access, then restart that program.

## Installation

```bash
git clone git@github.com:jaredtkatz/imessage-rag-mcp.git
cd imessage-rag-mcp
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
```

The first run downloads the embedding and reranker models from Hugging Face
(a few hundred MB).

## Usage

Build the index and start the server:

```bash
SYNC_ON_STARTUP=true ./run.sh
```

The initial sync and index build can take several minutes depending on the size
of your message history. On later runs you can omit the flag to skip syncing and
start immediately against the existing index:

```bash
./run.sh
```

`run.sh` is a thin wrapper around:

```bash
python -m uvicorn mcp_server:app --host 0.0.0.0 --port 8000 --reload
```

### Connecting an MCP client

Point your MCP client at:

```text
http://localhost:8000/mcp
```

### HTTP endpoints

Both endpoints are also usable directly over HTTP:

- `GET /search?query=...&limit=8` — full hybrid pipeline (dense + lexical →
  fusion → rerank → context expansion). This is the tool exposed over MCP.
- `GET /lexical?query=...&limit=20` — TF-IDF results only, useful for debugging
  retrieval.

## Configuration

All settings are environment variables with sensible defaults. They can be set
in the shell or in a `.env` file in the project root:

```bash
cp .env.example .env
```

Shell variables take precedence over `.env`, so you can override a file value
for a single run:

```bash
SYNC_ON_STARTUP=true ./run.sh
```

`.env` is gitignored.

| Variable | Default | Description |
| --- | --- | --- |
| `SYNC_ON_STARTUP` | `false` | Sync messages and rebuild indexes on startup |
| `IMESSAGE_DB` | `~/Library/Messages/chat.db` | Source iMessage database |
| `IMESSAGE_SELF_SENDER_NAME` | `Me` | Name used for your own outgoing messages |
| `IMESSAGE_EMBEDDING_MODEL` | `BAAI/bge-small-en-v1.5` | Sentence-transformer embedding model |
| `IMESSAGE_RERANK_MODEL` | `cross-encoder/ms-marco-MiniLM-L-6-v2` | Cross-encoder reranking model |
| `IMESSAGE_SESSION_GAP_HOURS` | `8` | Idle gap that starts a new conversation session |
| `IMESSAGE_TARGET_CHUNK_CHARS` | `1800` | Target chunk size in characters |
| `IMESSAGE_MAX_CHUNK_MESSAGES` | `16` | Maximum messages per chunk |
| `IMESSAGE_CHUNK_OVERLAP_MESSAGES` | `3` | Messages repeated between adjacent chunks |
| `IMESSAGE_DENSE_CANDIDATES` | `40` | Candidates retrieved from FAISS |
| `IMESSAGE_LEXICAL_CANDIDATES` | `40` | Candidates retrieved from TF-IDF |
| `IMESSAGE_RERANK_CANDIDATES` | `40` | Fused candidates passed to the reranker |
| `IMESSAGE_RECENT_ROW_LOOKBACK` | `5000` | Rows re-examined behind the last synced row |

## How it works

1. **Ingest** (`ingest.py`) — reads new and recently changed rows from `chat.db`,
   recovers text from `attributedBody` when the plain `text` column is empty,
   resolves sender names against the Address Book, and upserts into the local
   canonical database.
2. **Index** (`indexer.py`) — groups messages per chat, splits them into sessions
   on time gaps, chunks each session with overlap, then writes a FAISS index and
   a TF-IDF matrix.
3. **Retrieve** (`rag.py`) — runs dense and lexical search, fuses the rankings
   with RRF, reranks with a cross-encoder, drops overlapping chunks, and expands
   each result with surrounding messages.
4. **Serve** (`mcp_server.py`) — exposes the pipeline as a FastAPI app mounted as
   an MCP server.

### Project layout

```text
config.py       Environment-driven settings and file paths
db.py           SQLAlchemy models for chat.db, Address Book, and local storage
ingest.py       Sync from chat.db into the canonical database
indexer.py      Session splitting, chunking, and index construction
rag.py          Hybrid retrieval, fusion, reranking, context expansion
mcp_server.py   FastAPI application and MCP mount
run.sh          Development server launcher
.env.example    Template for local configuration
```

### Data storage

Generated artifacts live in `imessage_rag_data/` (gitignored):

```text
messages.sqlite   Canonical messages and chunks
messages.faiss    Dense vector index
lexical.joblib    TF-IDF vectorizer and matrix
state.json        Sync watermark and source fingerprint
```

Delete the directory to force a clean rebuild.

## Notes and limitations

- Syncing only happens at startup, and only when `SYNC_ON_STARTUP=true`. There is
  no background or on-demand sync yet, so restart the server to pick up new
  messages.
- Attachments, reactions, and edited-message history are not indexed — text only.
- Running uvicorn with multiple workers currently causes 404s on the MCP mount,
  so the server runs single-worker.
- The whole index is rebuilt from scratch whenever the corpus changes; there is
  no incremental reindexing.