# Substack Vault

**English** · [中文](./README.zh-CN.md)

> Collect Substack newsletters from **Zoho Mail**, archive them locally, and analyze via **MCP Streamable HTTP** with cross-author search.

**v0.1.0** — Initial release: Zoho sync, Web UI, article library, and MCP tools.

## Features

| Module | Description |
| --- | --- |
| **Mail ingestion** | Zoho Mail OAuth (Self Client + Grant Code), full Substack email body |
| **Session archive** | Paste your own Substack session cookie to import subscriptions and pull historical (including paid) posts you already subscribe to |
| **Subscriptions** | Enable/disable feeds, inbox scan for new authors, link orphans by `sender_email` |
| **Article library** | List/card views, filter by author/date/keyword; drawer reader, copy, export Markdown |
| **AI analysis** | DeepSeek (`deepseek-v4-flash`): saved schemes, date range + authors + keywords, cross-author comparison |
| **Sync engine** | Manual/scheduled incremental sync, configurable lookback, sync log |
| **Web UI** | Dashboard, subscriptions, articles, AI analysis, accounts, sync log, settings; light/dark theme |
| **MCP** | Streamable HTTP endpoint for Cursor and other agents; cross-author topic search |
| **Local-first** | SQLite storage, encrypted OAuth/LLM tokens on disk, binds to `127.0.0.1` by default |

## Requirements

- Python **3.11+**
- Zoho Mail account ([accounts.zoho.com](https://accounts.zoho.com))
- A **Self Client** app in [Zoho API Console](https://api-console.zoho.com)
- (Optional) [DeepSeek](https://platform.deepseek.com/) API key for in-app AI analysis

## Quick start

```bash
git clone https://github.com/flap4fun/substack-vault.git
cd substack-vault

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
# Optional: OCR for image-only Substack emails (e.g. weekly image posts)
pip install -e ".[ocr]"

cp .env.example .env               # Windows: copy .env.example .env
```

### Optional: image OCR (Tesseract)

Some paid newsletters put the body in images. Vault OCRs those **from Zoho email HTML** (not the public Substack site, which may be paywalled).

1. Install [Tesseract](https://github.com/tesseract-ocr/tesseract) with language packs **`chi_tra`** and **`eng`**
2. `pip install -e ".[ocr]"`
3. Re-sync articles to re-fetch email HTML when needed and OCR images

### Optional: historical archive (session cookie)

Email sync only sees recent inbox mail. To pull older posts from publications you already pay for:

1. Sign in to Substack in your browser
2. Copy `connect.sid` / `substack.sid` (DevTools → Application → Cookies) or export a JSON cookie array
3. In **Settings**, paste the cookie and save (Vault probes `profile/self` and encrypts it locally)
4. In **Subscriptions**, use **Import from Substack** then **Pull history**

Do not commit cookies. This version has no password / OTP auto-login. Use only your own subscribed content.

Start the server:

```bash
substack-vault
```

Open **[http://127.0.0.1:8765](http://127.0.0.1:8765)** in your browser. Use this URL directly — do not serve `web/` with Live Server or another static file server.

### Docker (optional)

Run the whole app (including Tesseract OCR) in a container — no Python, venv, or Tesseract install on the host:

```bash
cd substack-vault
cp .env.example .env        # compose reads secrets from this file
docker compose up -d --build
```

Open **[http://127.0.0.1:8765](http://127.0.0.1:8765)**. The SQLite database and encrypted keys live in the Docker named volume `substack-vault_vault-data`, so data survives image rebuilds.

Useful commands:

```bash
docker compose logs -f        # follow logs
docker compose restart         # restart the app
docker compose down            # stop (data kept)
docker compose down -v         # stop AND delete all data
```

Notes:

- The host port is bound to `127.0.0.1` only. Keep it that way — the MCP endpoint has **no authentication**.
- Fill in `VAULT_ZOHO_*` (and optionally DeepSeek credentials in **Settings**) before connecting; compose passes the `VAULT_ZOHO_*` values from your `.env` into the container.
- OCR (Tesseract `chi_tra`+`eng`) is already inside the image, so image-only posts work out of the box.
- Substack outbound access uses `VAULT_PROXY` from `.env` (for example, `http://host.docker.internal:7890`); it does not depend on the Windows system-proxy toggle and tries direct access when the proxy is unavailable.
- Linux users who prefer direct file access can bind-mount instead of the named volume: replace the `volumes:` block in `docker-compose.yml` with `- ~/.substack-vault:/data`, after `sudo chown -R 1000:1000 ~/.substack-vault`.

## Connect Zoho Mail

Self Client apps **do not** support browser OAuth redirects. You must generate a Grant Code manually.

### 1. Create a Self Client in Zoho API Console

- Type: **Self Client**
- Redirect URI: `http://localhost:8765/oauth/zoho/callback` (must match `.env`)
- Scopes (paste when generating code):

```
ZohoMail.messages.READ,
ZohoMail.accounts.READ,
ZohoMail.folders.READ,
offline_access
```

> `offline_access` is required for refresh tokens; without it you must re-authorize when tokens expire.

### 2. Connect in the Web UI

1. Open **邮箱账号** (Mail accounts) → **连接 Zoho Mail**
2. Select your data center (default: Global `.com`)
3. Enter Client ID, Client Secret, and Grant Code
4. After connecting, use **手动同步** (Sync now) in the top bar

Credentials can also go in `.env` (see Configuration). UI-saved credentials are stored locally as well.

### 3. Subscriptions and sync

1. **订阅管理** (Subscriptions) → **扫描发现** (Scan inbox) to discover Substack senders
2. Enable subscriptions you want to sync
3. **手动同步** or wait for the background job (default: every 30 minutes)

## Web UI

| Page | Purpose |
| --- | --- |
| **Dashboard** | Article/subscription stats, recent sync activity |
| **Subscriptions** | Author list, enable toggle, inbox scan, article count & last update |
| **Articles** | Filters, resizable reading drawer, copy/export Markdown; **Send to AI analysis** |
| **AI analysis** | Saved schemes, authors/keywords/date range → DeepSeek Markdown report |
| **Mail accounts** | Zoho connection status, read-only sync policy |
| **Sync log** | Per-run counts, duration, errors |
| **Settings** | Theme, DeepSeek API base URL + key, compact list |

### AI analysis (DeepSeek)

1. Open **Settings** → **AI / DeepSeek**, set API Key (default base URL `https://api.deepseek.com`)
2. Open **AI analysis**, pick or edit a scheme (seeded: “跨作者观点对比”)
3. Choose date range, 1–N authors, 1–N keywords (e.g. tickers), then **Run**
4. Or from **Articles**, set filters and click **Send to AI analysis**

Model is fixed to **`deepseek-v4-flash`**. The API key is Fernet-encrypted locally (same as Zoho secrets).

Display timezone defaults to **Asia/Shanghai (UTC+8)**; the database stores UTC.

## Cursor MCP

With `substack-vault` running, add to Cursor MCP settings:

```json
{
  "mcpServers": {
    "substack-vault": {
      "url": "http://127.0.0.1:8765/mcp"
    }
  }
}
```

### Tools

| Tool | Description |
| --- | --- |
| `list_subscriptions` | List all Substack subscriptions |
| `list_articles` | List articles by author slug or keyword |
| `get_article` | Full text and metadata by ID |
| `search_articles` | Search titles and bodies |
| `find_topic_across_authors` | Same topic across authors for viewpoint comparison |
| `get_sync_status` | Sync state and recent logs |

Example: ask an agent to compare how different authors cover a topic, or summarize the past week’s posts.

## Configuration

`.env` example (`VAULT_` prefix):

```env
# Zoho OAuth
VAULT_ZOHO_CLIENT_ID=
VAULT_ZOHO_CLIENT_SECRET=
VAULT_ZOHO_REDIRECT_URI=http://localhost:8765/oauth/zoho/callback

# Server
VAULT_HOST=127.0.0.1
VAULT_PORT=8765

# Optional Substack proxy fallback; direct Substack access is also built in
VAULT_PROXY=

# Sync (optional)
VAULT_SYNC_INTERVAL_MINUTES=30
VAULT_SYNC_LOOKBACK_DAYS=90
VAULT_SYNC_SUBSCRIBED_ONLY=true
```

| Variable | Default | Description |
| --- | --- | --- |
| `VAULT_HOST` | `127.0.0.1` | Bind address |
| `VAULT_PORT` | `8765` | Port |
| `VAULT_SYNC_INTERVAL_MINUTES` | `30` | Background sync interval (minutes). Can also be changed from Settings → Scheduled sync (30–1440) without restart; the UI value takes precedence |
| `VAULT_SYNC_LOOKBACK_DAYS` | `90` | Initial / lookback window (days) |
| `VAULT_SYNC_SUBSCRIBED_ONLY` | `true` | Sync only enabled subscriptions |
| `VAULT_PROXY` | empty | Optional proxy for Substack requests; failures automatically fall back to direct access |

## Data directory

| Platform | Path |
| --- | --- |
| Windows | `%USERPROFILE%\.substack-vault\` |
| macOS / Linux | `~/.substack-vault/` |

Contains `vault.db` (SQLite), encryption key, and OAuth credentials. **Do not commit this directory.**

## Development

```bash
pip install -e ".[dev]"
pytest
```

Layout:

```
src/substack_vault/
  api/          FastAPI REST + static Web
  connectors/   Zoho Mail connector
  parsers/      Substack email parser
  sync/         Sync engine
  storage/      SQLAlchemy models & DB
  mcp/          MCP Streamable HTTP
web/            Frontend static assets
tests/          Unit & API tests
openspec/       Specs & change history (OpenSpec)
```

## Roadmap

- [ ] Outlook and other mail providers
- [ ] RSS body enrichment
- [ ] Article tags and advanced filters
- [ ] Analysis run history / streaming UI / more LLM providers
- [ ] Multi-user / remote deployment

## Limitations

- **Zoho Mail only** (international and regional data centers)
- **Single-machine local** deployment; no multi-tenant or cloud sync
- Substack HTML parsing depends on email templates; unusual layouts may lose formatting
- Image-body posts need local **Tesseract** (`chi_tra`+`eng`); OCR is best-effort
- Paid full text: recent mail via Zoho; older / paywalled posts via your own session cookie in Settings. Never commit cookies
- In-app AI supports **DeepSeek official** only; model fixed to `deepseek-v4-flash`
- MCP endpoint has **no authentication** — do not expose to the public internet

## License

[MIT](LICENSE) © 2026 flap4fun