# MailOKF MCP

MailOKF is a local-first Python MCP server that turns Gmail, Microsoft Outlook/Microsoft 365, iCloud Mail, and generic IMAP/SMTP mailboxes into a synchronized, searchable knowledge layer—without making the local copy authoritative.

It exposes **38 tools and four resources** for mailbox actions, synchronization, OKF retrieval, attachments, and optional semantic search. The provider mailbox remains authoritative; SQLite and OKF are local projections.

MailOKF combines full email operations with durable Open Knowledge Format (OKF) projections in one local-first MCP:

- full initial mailbox import;
- provider-specific incremental synchronization;
- normalized SQLite storage and FTS5 search;
- one Open Knowledge Format (OKF) source concept per email;
- local, content-addressed attachment retention;
- raw MIME deletion after successful conversion;
- optional CocoIndex + sqlite-vec semantic indexing;
- draft, send, reply, forward, move, delete, label/category, and batch tools;
- explicit confirmation and idempotency controls for outbound email.
- tamper-evident managed outbound attachments for new sends and drafts.

> **Implementation status:** MailOKF 0.3 provides 25 stable core email operations with camelCase calling shapes plus thirteen MailOKF extensions. Live Gmail, Microsoft, and iCloud interoperability still requires testing with your own provider apps, credentials, tenants, mailbox sizes, throttling conditions, and compliance requirements before production use.

> **Production notice:** This project is local-first software, not a live-provider certification. Test it with isolated accounts and validate provider, tenant, scale, privacy, and compliance requirements before production use.

> **Keep private:** Never commit OAuth JSON, tokens, `MAILOKF_ROOT_DIR`, runtime mailbox data, attachments, quarantine, or backups.

## Documentation

| Guide | Use it for |
|---|---|
| [Documentation index](docs/README.md) | Choose the recommended path for users, operators, or contributors. |
| [Local usage](docs/LOCAL_USAGE.md) | Install, configure, sync, run MCP, back up, and troubleshoot. |
| [Google Cloud setup](docs/GOOGLE_CLOUD_SETUP.md) | Create a Gmail OAuth desktop client. |
| [Microsoft Entra setup](docs/MICROSOFT_ENTRA_SETUP.md) | Register an Outlook public client. |
| [Architecture](docs/ARCHITECTURE.md) | Understand authority, storage, synchronization, and failures. |
| [Security](docs/SECURITY.md) | Review supported versions, reporting, secrets, and trust boundaries. |
| [Contributing](docs/CONTRIBUTING.md) | Set up development and prepare a pull request. |

## Outbound attachments

`email_send` and `email_draft_create` accept an optional strict `attachments` array. Both require a nonblank `idempotencyKey` before provider access; terminal send/draft retries replay durable results and ambiguous provider outcomes remain blocked. Each item must
select exactly one source: `{stagedId, filename?}`, `{messageId, attachmentId, filename?}`, or
`{localPath, filename?, confirmed: true}`. Reply and forward do not accept attachment inputs.

Use `mail_attachment_list(account_id, message_id)` to inspect retained message attachments without
paths or bytes. Its typed results include `messageId` and `attachmentId`, so they can be copied into
the retained selector above. `mail_attachment_stage(local_path, filename?, confirmed?)` returns a
typed result whose `stagedId` can be copied into the staged selector. `mail_attachment_create`
accepts `filename`, optional `content_type`, and exactly one tagged `content` shape:
`{kind: "text", contentText: "..."}` or `{kind: "base64", contentBase64: "..."}`. Its typed result
also exposes `stagedId`. Configured workspace/import roots are allowed; arbitrary paths require both
`MAILOKF_DANGEROUSLY_ALLOW_ARBITRARY_ATTACHMENT_PATHS=true` and `confirmed=true`; the former
`MAILOKF_ALLOW_ARBITRARY_ATTACHMENT_PATHS` name is ignored. `MAILOKF_ATTACHMENT_IMPORT_ROOTS` uses
the platform `PATH` separator.

Outbound requests accept up to 20 files and 157,286,400 bytes combined; the per-file staging limit
is `Settings.max_attachment_bytes`, configured by `MAILOKF_MAX_ATTACHMENT_BYTES`. Inline UTF-8 text
is size-preflighted in bounded chunks before encoding, and strict base64 is size-preflighted then
decoded incrementally without exceeding that per-file cap. Bound filenames and parameter-free MIME
types are limited to 255 UTF-8 bytes each; managed metadata sidecars are limited to 4,096 bytes.
Count, per-file, total, and metadata limit failures use fixed public messages rather than embedding
configuration values.

Staging writes one physical `outbound/sha256/<digest>` blob per SHA-256 and an opaque,
restart-stable sidecar reference that persists no source path or source-path hash. Its validated safe
filename persists and defaults to the source basename when no filename override is supplied. An
unbound staged or created reference atomically binds to the first account that uses it for
`email_send` or `email_draft_create`. Content and metadata are reverified before each use. Reusing an
already-bound batch creates no empty transaction marker; stale markers with no referencing binding
are reclaimed under the store lock. Duplicate output names receive deterministic numeric suffixes.
Gmail and IMAP preflight a conservative complete-MIME bound before reading managed descriptors, then build MIME with the preserved filename and type. Gmail rejects a fully serialized RFC
2822 MIME message above its provider-specific 36,700,160-byte cap before creating a send/draft
operation; the global attachment contract is unchanged. Outlook uses Graph `fileAttachment` below
3,000,000 bytes and sequential upload sessions from 3,000,000 through 150,000,000 bytes.

Managed publication is atomic and cross-process serialized. Every configured root ancestor is
opened from the filesystem anchor without following symlinks. Platforms without the required
descriptor-relative filesystem operations or cross-process file locking fail closed rather than
falling back to process-local attachment safety.

Outbound attachments specifically require secure POSIX descriptor operations and `fcntl` locks.
They are intentionally unsupported on Windows; ordinary attachment-free mail, synchronization,
search, and the rest of MailOKF's Windows surface remain supported. Windows CI treats this feature
as an expected fail-closed capability rather than advertising attachment delivery. Tests that
exercise POSIX-only no-follow OAuth configuration reads or descriptor-secure storage migration are
also skipped on Windows instead of weakening those filesystem guarantees.

## Why this is an MCP server rather than only a converter

The provider mailbox remains authoritative. MailOKF keeps a synchronized local projection and exposes both:

- **resources and retrieval tools** for local OKF knowledge;
- **action tools** for provider operations such as draft, send, reply, move, and label.

This keeps read-heavy agent queries local and fast, while provider APIs remain available for fresh synchronization and controlled writes.

## Architecture

```mermaid
flowchart LR
    G[Gmail API] --> P[Provider adapters]
    O[Microsoft Graph] --> P
    I[iCloud / IMAP + SMTP] --> P
    P --> S[Full + incremental sync engine]
    S --> T[Private raw MIME staging]
    T --> M[MIME parser]
    M --> A[SHA-256 attachment blob store]
    M --> D[SQLite operational DB + FTS5]
    D --> K[Deterministic OKF renderer]
    K --> V[CocoIndex incremental vectors]
    D --> X[MCP tools/resources]
    K --> X
    P --> X
```

### Per-message commit boundary

```text
stage raw MIME privately
  -> parse text and metadata
  -> write/verify attachment blobs
  -> upsert SQLite + FTS
  -> atomically write OKF message and thread files
  -> delete temporary raw MIME unless retention is enabled
```

If parsing, attachment persistence, database commit, or OKF rendering fails, MailOKF retains the raw message rather than discarding it. It normally moves the file to quarantine; if that move fails, it leaves the bytes in staging and records the fallback path and error.

## Storage layout

```text
${MAILOKF_ROOT_DIR:-./lcl_data}/
├── mailokf.json                   # non-secret local layout manifest
├── mailokf.sqlite                 # sync ledger, normalized records, FTS5
├── bundles/
│   └── <account-id>/
│       ├── index.md              # OKF-reserved navigation file
│       ├── log.md                # OKF-reserved date-grouped change log
│       ├── bundle.md             # normal email_knowledge_bundle concept
│       ├── messages/YYYY/MM/*.md # one OKF source concept per message
│       └── threads/*.md          # deterministic thread manifests
├── attachments/
│   └── sha256/ab/cd/<digest>     # deduplicated attachment bytes
├── staging/                       # temporary raw MIME only
├── quarantine/                    # failed raw MIME conversions
├── vectors/
│   ├── okf-vectors.sqlite         # sqlite-vec index
│   └── cocoindex-state/           # CocoIndex incremental state
├── workspace/                     # only authorized outbound attachment import source
├── outbound/                      # private immutable managed outbound attachments
└── config/
    ├── accounts.enc
    ├── master.key
    ├── gateway.token
    └── oauth/
```

When run from a source checkout without `MAILOKF_ROOT_DIR`, MailOKF stores all local state in the
repository's ignored `lcl_data/` directory. Set `MAILOKF_ROOT_DIR` to use a different portable root;
this is required when the application is installed outside a checkout or when a CLI, daemon, gateway,
and MCP client must share a separately managed location. Paired `MAILOKF_DATA_DIR` and
`MAILOKF_CONFIG_DIR` overrides remain available as an explicit legacy split mode. Use `mailokf storage show` to inspect resolution and `mailokf storage
consolidate --root <path>` after stopping all MailOKF processes to migrate split state safely. To move an already-unified root, keep every MailOKF process stopped and run `mailokf storage relocate --source-root <old-root> --root <new-root>`; add `--move` only when verified source removal is intended. Both workflows reject WAL/SHM, links, special files, and overlapping roots, use a private restart journal, verify every destination byte, and publish a newly root-bound `mailokf.json` only after the destination is complete. Relocation targets one local or cloud-mounted filesystem root and does not implement direct object-storage semantics.

This layout is local-filesystem preparation, not an Azure Blob or S3 implementation. SQLite and
its WAL/SHM/lock semantics require a local filesystem and are not object-storage compatible.

## Local quick start

Install Python 3.11+ and [`uv`](https://docs.astral.sh/uv/), then use the locked environment:

```bash
git clone https://github.com/pat229988/OKF_MAIL.git
cd OKF_MAIL
uv sync --locked
uv run mailokf init
uv run mailokf doctor
uv run mailokf account list
```

Optional dependencies are explicit:

```bash
# Outlook/Microsoft 365 device-code authorization
uv sync --locked --extra outlook

# CocoIndex + sqlite-vec semantic and hybrid search
uv sync --locked --extra vectors

# Both
uv sync --locked --extra outlook --extra vectors
```

Then follow the [Gmail](docs/GOOGLE_CLOUD_SETUP.md), [Outlook](docs/MICROSOFT_ENTRA_SETUP.md), or [local IMAP/iCloud](docs/LOCAL_USAGE.md#add-an-account) setup; test the account; and run an initial sync. CLI, daemon, gateway, and MCP clients must use the same `MAILOKF_ROOT_DIR`.

### Docker Compose local modes

Compose deliberately separates the continuously running synchronization daemon from the
client-attached MCP stdio process. Build the image and initialize the unified named root volume:

```bash
docker compose build
docker compose run --rm -T mailokf-stdio init
```

Run the supported long-running polling mode in the background:

```bash
docker compose up -d mailokf-daemon
docker compose logs -f mailokf-daemon
```

Do not run the stdio service detached. Configure an MCP client to create an attached, non-TTY
container for each client session (replace the Compose path with the checkout's absolute path):

```json
{
  "mcpServers": {
    "mailokf-docker": {
      "command": "docker",
      "args": [
        "compose",
        "-f",
        "/absolute/path/to/OKF_MAIL/docker-compose.yml",
        "run",
        "--rm",
        "-T",
        "mailokf-stdio"
      ]
    }
  }
}
```

Compose injects literal policy defaults from `.env.example`, followed by an optional ignored
`.env` file for local overrides. The container storage root is `/mailokf`, and both services share
one named volume mounted on a directory owned by nonroot UID `10001`. The
Compose configuration neither publishes an HTTP port nor permits unauthenticated HTTP, even if
`.env` requests it. `docker compose down` preserves the volumes; adding `--volumes` permanently
removes the local database, retained mail, encrypted account vault, and generated master key.

## MCP tool surface

The current contract defines **38 tools** (25 compatibility tools plus 13 extensions) and **four resource templates**. Test sources are maintained under `tests/`; `RELEASE_MANIFEST.json` inventories them by path and hash instead of freezing a brittle function or module count here.

### Original compatible tools

```text
email_list_accounts       email_add_account          email_remove_account
email_test_account        email_list_folders         email_search
email_get                 email_get_thread           email_get_attachment
email_send                email_reply                email_forward
email_draft_create        email_draft_list           email_move
email_transfer            email_delete               email_mark
email_label               email_folder_create        email_get_labels
email_get_categories      email_batch_delete         email_batch_move
email_batch_mark
```

### MailOKF extensions

```text
mail_sync_initial         mail_sync_now              mail_sync_all
mail_sync_status          okf_validate_bundle        okf_rebuild_bundle
knowledge_search          knowledge_get              vector_index_update
vector_search             mail_attachment_list       mail_attachment_stage
mail_attachment_create
```

### Resources

```text
okf://accounts/{account_id}/index
okf://accounts/{account_id}/messages/{message_id}
okf://accounts/{account_id}/threads/{thread_id}
mailokf://accounts/{account_id}/sync-status
```

## Provider synchronization design

| Provider | Initial synchronization | Incremental cursor | Outbound path |
|---|---|---|---|
| Gmail | `messages.list` + `messages.get(format=raw)` | `historyId` + `history.list`; full resync on expired history | MIME attachments through Gmail drafts/send API |
| Outlook | folder enumeration + message delta + raw MIME `$value` | one opaque `@odata.deltaLink` per folder, immutable IDs | Graph small attachment POST or large upload session on drafts |
| iCloud / IMAP | folder UID scan + `BODY.PEEK[]` | `UIDVALIDITY`, known UID high-water state, periodic reconciliation | MIME attachments through authenticated SMTP and IMAP Drafts |

## Raw mail and attachments

The default policy is intentionally asymmetric:

- raw `.eml` files are temporary and deleted only after a successful local commit;
- attachment bytes are retained in a SHA-256 blob store;
- OKF files contain attachment metadata and `attachment://sha256/<digest>` references;
- normalized text and HTML bodies are stored in SQLite; OKF remains the portable text projection;
- quarantine contains messages that could not be safely converted.

Set `MAILOKF_RETAIN_RAW_MIME=true` only for debugging or legal-retention requirements and protect that directory accordingly.

## Search model

1. **FTS5** provides exact and explainable sender, subject, body, and attachment-name retrieval.
2. **CocoIndex** watches the OKF directory and incrementally reprocesses only changed Markdown files.
3. **Sentence Transformers** creates local embeddings.
4. **sqlite-vec** stores and queries vectors in a `vec0` virtual table.
5. **Hybrid search** merges keyword and semantic ranks using reciprocal-rank fusion.

## Security boundaries

- Provider credentials are encrypted locally with AES-256-GCM.
- Outbound send/reply/forward tools require `confirmed=true` and a nonblank idempotency key.
- Pending or unknown provider outcomes block automatic retry with the same key.
- Email contents are untrusted data and never authorize tool calls.
- Attachment paths supplied for outgoing email are validated as local files.
- Raw MIME staging uses private file permissions and quarantine-on-failure semantics.
- Message failures are durably retried and bounded permanent failures are dead-lettered.
- Bare numeric IMAP UIDs require explicit source-folder context.
- Permanent Gmail deletion requires stored `https://mail.google.com/` scope; the default setup flow does not request it.
- Separate tenant/account roots must be maintained in multi-user deployments.
- Do not expose streamable HTTP outside localhost without MCP authorization, TLS, and tenant-aware access control.

## Development

Use the locked Python 3.11+ environment:

```bash
uv sync --locked --extra dev --extra outlook
uv run python -m compileall -q src tests scripts
uv run python scripts/snapshot_mcp_contract.py --check
uv run pytest -q
uv run mailokf --help
```

See [`docs/MCP_SURFACE.md`](docs/MCP_SURFACE.md) for the stable interface map and [`docs/MCP_CONTRACT.md`](docs/MCP_CONTRACT.md) for contract guarantees.

The test suite covers full and incremental synchronization, remote deletion reconciliation, MIME cleanup, attachment persistence, OKF rendering and validation, FTS5, encrypted credentials, and thread references.

## Important production work

Before production rollout, complete provider-specific and operational validation, particularly:

- live provider contract tests;
- OAuth verification/consent and app publishing;
- Graph throttling and large-folder pagination;
- IMAP server compatibility and UID edge cases;
- push notifications/webhooks;
- multi-tenant authorization;
- backup, retention, erasure, and audit policies;
- semantic retrieval evaluation.

## Attribution

MailOKF is released under the [MIT License](LICENSE). Its stable MCP interface, local-first storage model, and OKF knowledge layer are documented in [`docs/MCP_SURFACE.md`](docs/MCP_SURFACE.md) and [`docs/MCP_CONTRACT.md`](docs/MCP_CONTRACT.md).