mobile_claude_connect

A phone-to-PC bridge that gives a Claude Code session running on your PC access to your iPhone's data — photos, videos, live location, contacts, calendar, clipboard — plus a plugin system that lets Claude write and run its own browser automations against logged-in web services.

The phone is only a screen. Every byte stays on your own machine and travels over your own Tailscale tailnet. There is no cloud component, no third-party service, and nothing is uploaded anywhere.

Three pieces:

Architecture

        iPhone                      Tailscale                    Windows PC
 ┌────────────────────┐            ┌───────────┐      ┌──────────────────────────────┐
 │ Claude app         │            │           │      │  Claude Code                 │
 │  (Remote Control)  │───────────────────────────────>│      │                       │
 │                    │            │           │      │      │ stdio                  │
 │ Bridge PWA :8778   │  upload    │  100.x    │      │  mcp_server.py  ── MCP tools  │
 │  - photo picker    │───────────>│ WireGuard │─────>│      │                       │
 │  - run Shortcut    │            │  CGNAT    │      │  server.py  :8778             │
 │  - /login mirror   │            │  range    │      │   /ingest  ──> vault/         │
 │                    │            │           │      │   /drop    ──> vault/drop/    │
 │ Shortcuts app      │  location  │           │      │   /records ──> records table  │
 │  - Claude Bridge   │  contacts  │           │      │   /job/*   ──> job queue      │
 │  - Send to Claude  │───────────>│           │      │   /login/* ──> login_session  │
 └────────────────────┘            └───────────┘      │                              │
                                                      │  indexer.py ── CLIP ── GPU    │
   iCloud for Windows                                 │  caps.py ── Playwright ──┐    │
   photo library  ─────────────────────────────────────> watch_dirs             │    │
                                                      │  bridge.db (SQLite)     │    │
                                                      └─────────────────────────┼────┘
                                                                                │
                                                                  profiles/<service>/
                                                                  (persistent cookies)

Firewall allows 100.64.0.0/10 only — the Tailscale CGNAT range. Nothing is reachable from the public internet.

Related MCP server: remote-control-mcp

MCP tool surface

These are the tools Claude Code sees once the MCP server is registered.

CLIP semantic search

Photos and videos are embedded with openai/clip-vit-large-patch14 (configurable). A photo contributes one image embedding; a video is sampled at three frames spread across the middle 70% of the clip and the frame embeddings are averaged, then re-normalized. Every vector is L2-normalized float32 and stored as a raw BLOB in SQLite, so a query is one numpy matmul over the date/geo-filtered candidate set — tens of milliseconds over tens of thousands of assets.

A search is:

1. Filter by taken_at range and/or a [lat_min, lat_max, lon_min, lon_max] bounding box in SQL.

2. Embed the English text query, matmul against the candidate matrix, sort by cosine similarity, take the top k.

3. Return candidates, not answers — Claude then calls photos_open and looks at them before saying anything.

Two design notes that came out of using this in anger:

• Scores are relative. Cosine similarity lands in roughly 0.15–0.35 for everything. Rank matters; the number does not. There is no threshold at which a hit "is" a match.

• Trip questions are geography, not vision. "Find photos from my Japan trip three years ago" should not ask CLIP what Japan looks like. photos_places clusters GPS tags onto a 0.5° grid (~55 km) and reports each cluster's coordinates and date span, so Claude finds where you actually were and then narrows by content inside that window. A naive country-sized bounding box is genuinely dangerous here: a Japan rectangle of [24, 46, 123, 146] also swallows southeastern Korea.

Videos are stored upright: iPhones film portrait as a 1920×1080 landscape buffer plus a display-matrix rotation that libav does not apply on decode, so indexer._upright() reads frame.rotation and rotates before embedding.

Security model

This exposes your entire photo library and several logged-in accounts on an HTTP port. The controls are deliberately simple and layered:

• Tailnet only. open_firewall.ps1 creates an inbound rule for TCP 8778 restricted to -RemoteAddress 100.64.0.0/10, the Tailscale CGNAT range. Traffic never touches the public internet; it is WireGuard-encrypted end to end by Tailscale.

• Shared bearer token. Every route except the PWA shell, /manifest.json and /health requires the token from config.json, compared with hmac.compare_digest so the check is constant-time.

• Token in the query string is a deliberate compromise. <img src> cannot carry a header, and the phone's Shortcuts actions make headers awkward. This is only acceptable because the traffic never leaves the tailnet. Do not expose this port publicly.

• Path containment. Every uploaded filename goes through safe_name() (which strips directory components and Windows-illegal characters but keeps Unicode, so Korean and emoji filenames survive) and the resolved destination is checked with is_relative_to(VAULT) before anything is written.

• Passwords are never logged. /login/input explicitly filters what it forwards, and login_session passes typed text straight to the browser.

• The phone-side PWA stores the token in localStorage, so you type it once.

The job queue

iOS gives no way for a PC to wake a phone. So the flow is inverted:

1. Claude calls phone_request("location.current"), which inserts a row into jobs with status pending and returns a job id immediately.

2. The phone runs the Claude Bridge Shortcut — by tap, from the PWA's button, or from a scheduled automation. It GETs /job/next, which atomically marks one job taken. With nothing queued the server answers 204 and the Shortcut just exits.

3. The Shortcut branches on the action name, gathers the data, and POSTs it to /records?job=<id> or /job/<id>/done.

4. Claude polls phone_job_status.

A job marked taken that is never completed is returned to pending after job_ttl_min minutes, so a Shortcut killed mid-run does not lose the request.

Claude must tell you to run the Shortcut rather than silently polling — that is written into the tool's docstring.

Capability plugins

A capability is one Python file in capabilities/ that exposes run(**kwargs) -> dict. Its module docstring is the spec — that is what Claude reads to decide whether to call it, so it documents the args, the return shape, and whether a login is needed.

Modules are re-imported on every call (importlib.util.spec_from_file_location + exec_module), so a capability Claude wrote or edited thirty seconds ago is live immediately. No MCP restart, no server restart.

The intent is that a missing integration is not a dead end. Claude has Write and Bash on the machine: it calls capability_howto, writes capabilities/<name>.py, and runs it. What it writes persists and accumulates.

Two ship with the repo: webpage.py (render any URL in a real browser and return its readable text, optionally as a logged-in user) and naver_mail.py (a substantial worked example — it intercepts the web client's own JSON XHRs rather than guessing an API).

Writing one

"""One line saying what this does - this line is what Claude sees in the list.

Args:
    since (str): 'YYYY-MM-DD'. Optional, defaults to 30 days ago.
    limit (int): max rows. Optional, default 20.

Returns:
    {"orders": [{"date","title","price","url"}], "count": int}

Notes:
    Requires a one-time manual login (PROFILE below).
"""

PROFILE = "example"                       # omit for capabilities needing no login
LOGIN_URL = "https://example.com/login"


def run(since: str = "", limit: int = 20) -> dict:
    from caps import browser

    with browser(PROFILE) as page:
        page.goto("https://example.com/orders", wait_until="domcontentloaded")
        if "login" in page.url:
            return {"error": "logged out",
                    "fix": f"call capability_login('{PROFILE}', '{LOGIN_URL}')"}
        rows = page.query_selector_all("li.order")
        if not rows:
            return {"error": "no rows matched 'li.order' - selector likely stale",
                    "url": page.url}
        ...

See capabilities/_TEMPLATE.py for the full annotated version. The rules, which capability_howto returns verbatim:

• Always go through caps.browser(PROFILE). Never launch Playwright directly.

• Fail loudly. An empty list after a site redesign reads as "you have nothing", which the user will act on. Return {"error": ...} naming the selector that missed.

• Never automate payment, order confirmation or money transfer. Gather, decide, prepare the screen — the human taps the last button.

• Never automate a login (below).

Logins are deliberately manual

Automating a sign-in loses to 2FA and captchas, and a pile of failed attempts locks your account. So it is not supported. Instead:

1. Claude calls capability_login(profile, login_url).

2. By default a headless Chromium starts on the PC and its screen is mirrored to your phone at /login — JPEG frames out at ~3 fps, taps and keystrokes posted back. Nothing appears on the PC screen, so it cannot steal focus from a game. Pass on_pc=True for accounts you would rather not type over the bridge.

3. You type the password yourself, once.

4. Cookies persist and every later run reuses them.

The persistence is subtler than "use a persistent profile". Chromium discards session cookies — the ones with no expiry — when the context closes, and for many services (Naver's NID_AUT / NID_SES, for instance) those are the login. A profile directory can look fully populated and still be signed out. So caps.browser() calls load_state() on open and save_state() on close, round-tripping the cookies through profiles/<service>/storage_state.json, and login_session snapshots them every ~10 seconds in case the phone walks away mid-login.

caps.launch_kwargs() prefers channel="chrome" (your real installed Chrome) over Playwright's bundled Chromium, because Google in particular flags the bundled build during sign-in.

Requirements

• Windows (the code uses ctypes.windll for process priority and CREATE_NO_WINDOW)

• Python 3.10+ (uses X | Y type syntax and Path.is_relative_to)

• Tailscale on both the PC and the phone, in the same tailnet

• An NVIDIA GPU is optional but strongly recommended. CLIP runs on CPU (set "device": "cpu") — it is just much slower for a first full index.

• An iPhone. Everything phone-side is the built-in Shortcuts app plus a home-screen PWA; no app to install, no pairing, no jailbreak.

Install

git clone <this repo>
cd mobile_claude_connect

# CUDA build of torch first, if you have an NVIDIA GPU:
pip install torch --index-url https://download.pytorch.org/whl/cu128

pip install -r requirements.txt

# Browser engine for the capability plugins. Separate step - pip does not do this.
playwright install chromium

If you have Google Chrome installed, caps.launch_kwargs() will use it (channel="chrome") in preference to the bundled Chromium, and playwright install chromium becomes optional.

Configure

copy config.example.json config.json
python -c "import secrets; print(secrets.token_urlsafe(24))"

Edit config.json:

• token — paste the generated secret. The phone needs the same value.

• vault_dir — absolute path where phone uploads land.

• watch_dirs — folders scanned for photos in place (nothing is copied). Point this at your iCloud for Windows library, e.g. C:\Users\YOUR_USERNAME\Pictures\iCloud Photos\Photos.

• device — cuda or cpu.

Running without a config.json exits with instructions rather than a stack trace.

Firewall

# Run as Administrator
powershell -ExecutionPolicy Bypass -File open_firewall.ps1

This opens TCP 8778 to 100.64.0.0/10 only and prints the URL to open on the phone. Keep .ps1 files ASCII-only — Windows PowerShell 5.1 reads them in the system ANSI codepage, and non-ASCII characters will corrupt.

Run

Double-click run_bridge.bat (or python server.py). Closing the window stops the bridge, and nothing works from the phone while it is down.

With the bridge up, the index maintains itself: every auto_index_minutes (default 15) the server spawns indexer.py as a short-lived subprocess, so the ~2 GB of CLIP VRAM is handed back between runs instead of being held for as long as the server lives. An idle pass only stats the file tree and never touches the GPU. Output goes to autoindex.log.

For a first bulk import, python backfill.py loops scan-and-embed until the source folder stops growing — useful while iCloud is still downloading a large library. Do not run it at the same time as auto-indexing; two CLIP processes will fight over VRAM.

Register the MCP server

claude mcp add phone -s user -- python C:\path\to\mobile_claude_connect\mcp_server.py

Gotcha: MCP servers are loaded when a session starts. A server added with claude mcp add will not appear in the session you are currently in. Start a new session.

Phone-side setup — the PWA, the bulk photo import, and the Shortcut recipes — is in SETUP.md.

Using it

From a Claude Code session on the PC (including one you are driving from the phone via the Claude app's Remote Control):

Find the photos from my trip three years ago that show a pump machine.

What happens: photos_stats to check coverage → photos_places to find where you actually were and when → photos_search("a pump machine", date_from=..., date_to=...) to rank candidates → photos_open on the top hits → Claude looks at the JPEGs and answers from what it sees. CLIP narrows; Claude judges.

Korean (or any non-English) queries are translated to a short English phrase before the search, because CLIP's text encoder is English-only.

What's not included

This repository is code only. Everything that made the running instance useful is personal data and is not here:

• bridge.db — the photo index. You start with an empty index. Point watch_dirs at a photo folder and run python indexer.py (or let auto-index do it) to build your own. The first run downloads the CLIP model, about 1.7 GB.

• vault/ — uploaded photos, videos and share-sheet drops.

• preview/ — JPEGs rendered by photos_open.

• profiles/ — Playwright browser profiles, i.e. live logged-in sessions. You log in yourself, once per service, via capability_login.

• config.json — holds the bearer token. Copy config.example.json.

All of these are in .gitignore. Keep them there.

Known limits

• Call history and SMS cannot be retrieved. iOS exposes no API for either, to any app. A native app would not help; the only route is local backup extraction, which needs lockdown pairing.

• The PC must be on. The phone is a screen, not a peer.

• The /login mirror is ~3 fps JPEG at 1366×900 — fine for a login form, not for browsing.

• The phone-side PWA UI is in Korean (static/index.html, static/login.html). The server, the MCP tools and everything Claude reads are in English.

License

This is a source-only repository. The code here is MIT — see LICENSE — and it is the only thing this repo actually distributes. Everything else arrives on your machine from somewhere else: pip install -r requirements.txt pulls the wheels from PyPI, the CLIP weights come from Hugging Face, and the browser comes from playwright install. No third-party binary is redistributed here. Copyleft obligations attach to redistribution, so for the ordinary case — clone it, install it, run it on your own PC — almost nothing below is something you have to do, and the rest is written down for the day you package this into something you hand to someone else.

Running from source (what almost everyone does)

Nothing to comply with. The permissive licences in the table ask only that notices survive if you copy code out of a dependency into your own project. Two things are still worth knowing before you make plans, and neither of them is a copyleft question.

The CLIP weights have no licence, and that is not the same as being permissive. This is the one item here that can affect what you do with the thing while merely running it. The transformers library is Apache-2.0, but the weights are a separate artifact with separate terms. The Hugging Face repo openai/clip-vit-large-patch14 contains no LICENSE file and declares no license field in its model-card metadata. The upstream openai/CLIP code repository is MIT (© 2021 OpenAI), but that licence text covers the code in that repository and says nothing about checkpoints. So: whether commercial use of these weights is permitted is unverified. The model card also states its own position plainly — "Any deployed use case of the model — whether commercial or not — is currently out of scope", with the model intended for research into robustness and generalisation, and surveillance and facial recognition named as permanently out of scope. That is the authors' stated intent rather than a licence grant or prohibition, but if you are deciding whether to build a product on this, it is the most direct thing they have said. Swapping clip_model for a model with explicit licence metadata (several openly-licensed CLIP variants exist) is the clean way out.

Automating a third-party site is governed by that site's terms, not by this licence. The capability plugins drive logged-in web sessions — naver_mail.py is the shipped example, and the obvious next ones (Coupang Eats, Gmail) are the same shape. Whether you may script an account you hold is a question for that service's terms of use and any applicable computer-access law; the MIT licence on this code grants you nothing there. The same goes for any third-party MCP server you register alongside this one: it carries its own licence and its own service terms.

Dependency licences

The stack is mostly permissive, but two of the wheels ship copyleft binaries, one ships proprietary NVIDIA libraries, and one component has no declared licence at all. Every "see below" in the right-hand column is a redistribution obligation, not a run-time one.

If you build and redistribute a binary

Packaging this into an installer, a Docker image or any other artifact you hand to someone else is what turns the marked rows above into work.

PyAV's wheel is LGPL FFmpeg with GPL encoders sitting next to it — the distinction is finer than it looks. FFmpeg is LGPL-2.1-or-later by default and goes GPL when built with --enable-gpl, which upstream requires for x264 and x265. PyAV's wheels are not built with that flag: the bundled avutil reports libavutil license: LGPL version 3 or later and its configure line shows --enable-version3 --enable-libx264 --enable-libx265 with no --enable-gpl. That is possible because PyAV patches FFmpeg's configure to move libx264 and libx265 out of the GPL list into the version-3 list. The FFmpeg core is therefore LGPL-3.0-or-later — but x264 and x265 are themselves GPL-2.0-or-later, and av.libs/ ships them as libx264-165.dll and libx265.dll. So pip install av still puts GPL code in your dependency graph, whatever the FFmpeg core says. This project only reads video duration, rotation and sample frames and never touches an encoder, so the practical fix if you redistribute is to drop those two DLLs, or build from the sdist (pip install --no-binary av av) against your own FFmpeg. Two smaller gaps worth knowing: the wheel's licenses/ directory contains only PyAV's own BSD-3-Clause text and none of the FFmpeg, x264 or x265 notices, and delvewheel renames the FFmpeg DLLs with hash suffixes, which cuts against FFmpeg's own compliance checklist.

pillow-heif is the same shape, plus a patent question. Its Python code is BSD-3-Clause, but the binary wheel bundles libheif and libde265 under LGPL-3.0 and the x265 encoder under GPL-2.0 — which is why its PyPI classifier reads "GNU General Public License v2" even though its declared licence field says BSD-3-Clause. This project only decodes HEIC from the phone, so the GPL encoder is never called, but it is still in the wheel. Separately and independently of copyright: HEVC/H.265, the codec inside HEIC, is covered by patent pools whose administrators state that products with HEVC encoding or decoding functionality typically need a licence. That is a patent matter, not a software-licence one, and it is aimed at products that ship decoders commercially — a personal server is not the target — but it is a real obligation that no open-source licence resolves for you.

Playwright is Apache-2.0, and on x64 the browser it fetches is not open-source Chromium. Chromium's own source is BSD-3-Clause over a large third-party licence set (Blink under BSD/LGPL, Mozilla-derived code under MPL/GPL/LGPL, and several hundred more, aggregated at chrome://credits). But playwright install chromium no longer downloads that on x64. Playwright's browsers.json names the download "Chrome for Testing", and cdn.playwright.dev redirects to Google's own chrome-for-testing-public bucket. The artifact is Google-built and Google-branded, and it bundles the Widevine CDM, whose licence states plainly that it "is not open source software" and may not be distributed without a separate agreement with Google. Only the linux-arm64 build is still Playwright's own plain Chromium. Playwright's public docs still describe the default as open-source Chromium; the shipped code says otherwise. Separately, caps.launch_kwargs() prefers channel="chrome" — your real installed Google Chrome, governed by the Google Chrome Terms of Service rather than any open-source licence. The upshot for redistribution: bundle neither. Let the user install their own browser, which is what this repo already does. Unverified: whether Google exempts Chrome for Testing from the consumer Chrome terms.

A CUDA PyTorch build is not purely BSD. PyTorch itself is BSD-3-Clause, but the CUDA wheel this README's install step recommends — the one from download.pytorch.org/whl/cu128, not PyPI, whose Windows wheel is CPU-only — ships NVIDIA's CUDA runtime and cuDNN binaries. Inspecting an installed torch ...+cu128 shows 22 of them in torch/lib (cuBLAS, cuDNN, cuFFT, cuRAND, cuSOLVER, cuSPARSE and friends). Their wheel metadata declares LicenseRef-NVIDIA-Proprietary, and the CUDA EULA allows redistributing the runtime only under its own conditions: your application must add material functionality, the binaries must not be modified, they must be reachable only by your application, and you may not use the SDK in a way that would subject it to an open-source licence. cuDNN adds a supplement that overrides the base terms where they conflict. None of that is BSD. Running locally is untouched by any of it; the CPU-only wheel avoids the question entirely.

Nothing here is legal advice.