# xhs-sync — 小红书 → Notion / Obsidian

Pulls the notes you've saved on Xiaohongshu (小红书) into **Notion**, **Obsidian**, or
both, sorts them into categories automatically, and keeps them in sync. Run it by hand,
put it on a schedule, or drive it from an AI assistant over MCP.

```
小红书 收藏/点赞/笔记
        │
        ▼
   collect  ──►  categorize  ──►  Notion   (one page per note)
  (browser)      (rules + Claude)  Obsidian (one Markdown file per note)
        │                              │
        └──────────  state.db  ◄────────┘
              (dedupe — re-runs are cheap)
```

## What you get

- **Your saved notes, in your own tools.** Title, full body, images, author, hashtags,
  like/save counts and a link back to the original.
- **Automatic categories.** Every note lands in one of 15 categories (美食, 旅行, 穿搭,
  美妆, 家居, 健身健康, 学习成长, 职场, 数码科技, 理财, 育儿, 摄影, 宠物, 兴趣手作, 其他)
  plus a subcategory, searchable tags, and a one-line summary. Fully editable — see
  [Categories](#categories).
- **Idempotent syncs.** A note is only rewritten when its content actually changed. Your
  own edits in Obsidian are preserved.
- **Works offline-ish.** Without an Anthropic API key it falls back to keyword rules, which
  are free and surprisingly decent.

## Install

```bash
git clone <this repo> xhs-sync && cd xhs-sync
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[all]'
playwright install chromium          # one-time browser download
```

Then set up your config and credentials:

```bash
xhs-sync init                        # writes config/config.yaml
cp .env.example .env                 # then fill in your tokens
```

## Set up the destinations

You need at least one. Both is fine.

### Obsidian

Just point at your vault — nothing else to install:

```bash
# in .env
OBSIDIAN_VAULT=/Users/you/Documents/MyVault
```

Notes are written to `<vault>/小红书/<Category>/<YYYY-MM>/<title>-<id>.md` with YAML
frontmatter (`category`, `tags`, `url`, `likes`, …), so Dataview and the graph view pick
them up immediately. Images are downloaded into the vault by default so the notes work
offline.

### Notion

1. Create an integration at <https://www.notion.so/my-integrations> and copy the secret.
2. Create (or pick) a database, open it, and connect your integration:
   **`···` → Connections → your integration**.
3. Copy the database id from its URL — the 32-character chunk in
   `notion.so/<workspace>/<DATABASE_ID>?v=…`.

```bash
# in .env
NOTION_TOKEN=secret_xxxxxxxx
NOTION_DATABASE_ID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

The database can be empty — on first run the missing columns (`XHS ID`, `Category`,
`Tags`, `Summary`, `Likes`, …) are created for you.

## Set up categorization

Add an Anthropic API key and ambiguous notes get classified by Claude:

```bash
# in .env
ANTHROPIC_API_KEY=sk-ant-xxxx
```

Leave it out and everything falls back to keyword rules. See
[How categorization works](#how-categorization-works) for the tradeoff.

## Use it

```bash
xhs-sync login                       # opens a browser — scan the QR code once
xhs-sync doctor                      # checks everything is wired up
xhs-sync sync --dry-run              # see what it would do
xhs-sync sync                        # do it
```

`login` stores the session in a persistent browser profile under `~/.xhs-sync/`, so you
only do it once (until Xiaohongshu expires it — re-run `login` when a sync says you're
logged out).

### Everyday commands

| Command | What it does |
|---|---|
| `xhs-sync sync` | Sync your 收藏 tab (newest first) |
| `xhs-sync sync --source liked` | Sync 赞过 instead (`posted` for your own 笔记) |
| `xhs-sync sync --limit 50` | Only look at the 50 most recent |
| `xhs-sync sync --sink obsidian` | Write to one destination this run |
| `xhs-sync sync --dry-run` | Show categories, write nothing |
| `xhs-sync add <url>` | Sync one specific note |
| `xhs-sync recategorize` | Re-run categorization over everything (after editing the taxonomy) |
| `xhs-sync status` | What's in the local database, by category |
| `xhs-sync doctor` | Diagnose setup problems |

### Run it on a schedule

`scripts/` has ready-made units — edit the paths inside first.

```bash
# macOS
cp scripts/com.xhs-sync.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/com.xhs-sync.plist

# Linux
cp scripts/xhs-sync.{service,timer} ~/.config/systemd/user/
systemctl --user enable --now xhs-sync.timer

# or plain cron — daily at 09:00
0 9 * * * PROJECT_DIR=$HOME/xhs-sync /bin/bash $HOME/xhs-sync/scripts/sync.sh
```

Scheduled runs need a valid login. When the session expires the run logs it and exits
non-zero rather than failing silently.

## Categories

Categories live in [`config/taxonomy.yaml`](config/taxonomy.yaml). Adding one is three
lines:

```yaml
  - name: 咖啡 Coffee
    description: Espresso gear, bean reviews, brewing methods, cafe hopping.
    color: brown
    keywords: [咖啡, 手冲, espresso, v60, latte art]
    subcategories:
      - name: 器具 Gear
        keywords: [磨豆机, 手冲壶, grinder, portafilter]
```

Both the keyword matcher and Claude read that file, so a new category is picked up by
both. After editing, `xhs-sync recategorize` re-sorts everything you've already synced.

### How categorization works

Three modes, set by `categorize.mode` in `config.yaml` (or `--mode`):

| Mode | Behaviour | Cost |
|---|---|---|
| `rules` | Keyword scoring only | Free, offline |
| `llm` | Every note goes to Claude | ~1 request per 8 notes |
| `hybrid` *(default)* | Keywords when they're confident, Claude otherwise | Usually a handful of requests |

In `hybrid` mode a note skips the API when its top keyword score clears
`rules_confident_at` **and** beats the runner-up by `rules_margin` — i.e. when the answer
is obvious ("空气炸锅鸡翅做法" is 美食, no model needed). Everything vague goes to Claude,
which also writes the summary and the searchable tags. Notes are batched, and the
taxonomy prompt is cached, so a 200-note first sync is a few cents.

Claude's answer is constrained to a JSON schema and mapped back onto a real category from
your taxonomy, so it can't invent one. Note text is passed as data, never as instructions.

If you fix a category by hand — edit the `Category` in Notion, or use
`xhs-sync recategorize` after editing the taxonomy — a manual category is never silently
overwritten by the rules on a later run.

## MCP server

The repo also exposes the sync as an MCP server, so an assistant can run it:

```bash
xhs-sync serve-mcp        # speaks MCP over stdio
```

Register it with Claude Code:

```bash
claude mcp add xhs-sync -- /absolute/path/to/.venv/bin/xhs-sync serve-mcp
```

Or add it to any MCP client's config:

```json
{
  "mcpServers": {
    "xhs-sync": {
      "command": "/absolute/path/to/.venv/bin/xhs-sync",
      "args": ["serve-mcp"]
    }
  }
}
```

Tools: `xhs_sync_now`, `xhs_add_note`, `xhs_search_notes`, `xhs_list_categories`,
`xhs_recategorize` — so you can ask things like *"sync my latest saves"* or *"what travel
notes have I saved about Kyoto?"*

## How collection works

Xiaohongshu has no public API, and its web endpoints are signed in JavaScript that changes
regularly. Reimplementing that signing means re-reimplementing it every few months.

Instead this drives a real Chromium with **your** logged-in session, scrolls your own tab,
and reads the JSON the page fetches for itself. Two consequences worth knowing:

- **It only ever reads your own account's data** — the same notes you see when you open
  the app.
- **It degrades rather than breaks.** The extractor walks whatever JSON arrives looking for
  note-shaped objects instead of pinning to endpoint paths and field names, and falls back
  to reading the DOM, then to the page's embedded state. When Xiaohongshu reshuffles
  things, you'll usually get fewer fields rather than a crash.

Be reasonable with it: sync on a timer, not in a loop. It's your own saved notes, at
human speed.

To use an existing Chrome instead of Playwright's Chromium:

```bash
export XHS_BROWSER_EXECUTABLE="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
```

## What lives where

```
src/xhs_sync/
  collect/extract.py   tolerant JSON → Note (the bit that survives site changes)
  collect/browser.py   Playwright session, scrolling, XHR capture
  categorize/rules.py  offline keyword scoring
  categorize/llm.py    Claude classifier (schema-constrained, batched, cached)
  categorize/engine.py hybrid decision logic
  sinks/obsidian.py    Markdown + frontmatter, preserves your edits
  sinks/notion.py      database upsert, creates missing columns
  pipeline.py          collect → categorize → write → record
  state.py             sqlite dedupe + sync state
  mcp_server.py        MCP over stdio
  cli.py               the xhs-sync command
config/taxonomy.yaml   your categories
```

## Development

```bash
pip install -e '.[dev]'
pytest -q                     # 62 tests, no network or browser needed
pytest -q --live-browser      # also launch a real Chromium
```

The tests cover the extractor against realistic payload shapes, keyword scoring, both
sinks' output, dedupe behaviour, the pipeline, and the MCP protocol layer.

Try it without a Xiaohongshu account at all — `xhs-sync import` takes a JSON file of raw
note payloads and runs the whole pipeline on it:

```bash
xhs-sync import path/to/notes.json --dry-run
```

## Notes and limits

- **Private to you.** Everything runs locally. The only things leaving your machine are the
  writes to your Notion database and, in `hybrid`/`llm` mode, note text sent to the
  Anthropic API for categorization. Use `--mode rules` to send nothing anywhere.
- **Your login can expire.** Re-run `xhs-sync login` when a sync reports it.
- **Videos are linked, not downloaded.** Images are downloaded into the Obsidian vault;
  Notion embeds them by URL (Xiaohongshu's CDN links can expire, so the vault copy is the
  durable one).
- **Notion API version** is pinned to `2022-06-28`.