PhonePilot Local

Your iPhone, beside your AI conversation.

A local iPhone console for macOS: live screen preview, mouse and keyboard control, and MCP tools that reuse the same device session. Includes a reusable agent skill. No PhonePilot account or additional model API key is required.

简体中文 · Compatibility · Architecture

Experimental software. Physical-device operation has been verified on one iPhone/iOS/Xcode combination. See the tested scope before relying on another configuration. The preview above uses a synthetic demo. Start with USB: wireless tunnel establishment failed in the recorded physical test.

Features

• Live HEVC preview with an explicit periodic-screenshot fallback.

• An iPhone-style frame that follows the received image's aspect ratio, orientation, and available space. No extra notch or Dynamic Island is drawn over the image.

• Tap, swipe, long press, text entry, app launching, and hardware buttons.

• Pause/resume controls; pausing cancels current input while keeping the preview available.

• Save PNG screenshots and record the console canvas without audio.

• Ten stdio MCP tools, plus a skill for opening and using the console in a compatible AI app.

• Light/dark appearance and layouts for wide windows and narrow side panels.

PhonePilot uses pymobiledevice3 CoreDevice services, rather than Apple's iPhone Mirroring window. The console does not bundle an AI model. AI processing and usage limits belong to your chosen AI application.

Related MCP server: idh

Requirements

• macOS and Python 3.11 or newer; the recorded physical-device test used Apple silicon.

• uv.

• Full Xcode with developer-device support compatible with your iPhone's iOS version.

• A USB-connected, unlocked iPhone that already trusts this Mac and has Developer Mode enabled.

Initial trust, pairing, Developer Mode, and device-support preparation are manual steps. PhonePilot does not change them automatically. Node.js is needed only for frontend development checks, not for running the console.

Quick start

Download or clone this repository and open a terminal in its directory:

uv sync --locked --python 3.14
uv run --locked phonepilot

Open http://127.0.0.1:8765/ in a browser, or in your AI app's browser side panel. Connect and unlock the phone, select the detected device in 设备设置 / Device settings, and choose 连接 iPhone / Connect iPhone. Wait for a current picture before controlling it. The console interface is currently in Simplified Chinese.

On macOS, you can also run 首次安装.command, followed by 启动 PhonePilot.command. The launcher reuses or starts a background hub and opens the console. Closing the launcher does not stop the hub. The terminal quick-start command above stays in the foreground and can be stopped with Control-C.

Try the UI without a phone

uv run --locked phonepilot --demo --port 8766 --state-dir var-demo

Open http://127.0.0.1:8766/ and connect the demo device. Demo mode never discovers or controls real hardware and is labelled throughout the UI. Keep it separate from your physical-device session.

Use it from an AI app

Agent skill

For a compatible app that discovers user skills under ~/.agents/skills:

python3 tools/install_skill.py

The installer links the skill to this checkout, preserves an existing different skill, and does not modify global MCP settings. Keep the checkout in place; moving it requires updating the link. Skill discovery and browser-control availability depend on the app.

Example request:

用 PhonePilot 打开手机侧栏，先告诉我当前打开的是什么 App。

MCP

Start the console, connect your phone, then open 智能体 / Agent → 智能体接入设置 and copy the generated MCP configuration into your AI app's settings. The configuration uses your own checkout and environment paths; it contains no session token. Reload the MCP connection as required by your app.

Tools: phone_status, phone_screenshot, phone_tap, phone_swipe, phone_long_press, phone_type_text, phone_press_key, phone_press_button, phone_list_apps, phone_open_app.

All tools reuse the console's connection. Screenshot-based pointer tools require a fresh observation_id; it expires after 60 seconds and is invalidated by control actions. Re-observe the screen after an action to verify the outcome.

Privacy and control

The server listens on 127.0.0.1, checks local Host/Origin values, and requires a random session token for device APIs. Session files live in var/ with restrictive permissions and must not be shared. Non-ASCII text entry uses the phone clipboard and replaces its current contents; PhonePilot does not read the old clipboard.

Screenshots, input text, and app lists are not written to disk by ordinary API calls. Explicit screenshot/recording downloads do create files. The in-memory activity log contains operation types and times, not input text. Screenshots supplied to an AI app are processed under that app's rules; local device control does not imply local AI inference. See SECURITY.md.

Known limits

No audio playback/recording, multi-touch, automatic unlock, automatic pairing, or automatic Developer Mode setup. Physical-device recording, non-ASCII input, wireless operation, multi-device operation, and long-duration stability remain unverified. Camera/microphone use and developer-service faults can interrupt a session. Adapting the preview to a new screen shape is not a claim of protocol compatibility with that device.

Development

uv sync --locked --extra dev --python 3.14
uv run --locked --extra dev pytest -q
npm ci
npm test
npm run format:check
uv build
uv run --locked --extra dev python tools/package_release.py --output dist/phonepilot-local-source.zip

The Python integration tests use simulated devices; frontend tests exercise frame fitting and pointer mapping. Neither test suite connects to a physical iPhone. GitHub Actions is configured to run checks on macOS. A local pass does not mean the hosted workflow has run. See CONTRIBUTING.md.

License and credits

GPL-3.0-or-later. Transport and HID functionality builds on pymobiledevice3 at the pinned revision recorded in pyproject.toml and uv.lock. Keep that pin when reproducing the verified implementation: this adapter uses upstream internal APIs. See THIRD_PARTY.md for attribution. This project is not affiliated with Apple or OpenAI.

Recovery and parcel checks

The launcher reuses a running hub or starts it in the background and opens the console. Closing the launcher does not stop the service. Setup uses the locked environment and checks the selected full Xcode. After explicit connection, the hub probes device availability, waits for the same device on interruption and bounds reconnect attempts. Pause is preserved; uncertain input is never replayed. Video failures use screenshots while retrying live video. A hub restart requires device selection again.

The bundled skill includes a read-only Taobao parcel-check workflow: inspect each order and split shipment, report observed logistics events and incomplete coverage. An AI client still executes the workflow; copying its prompt does not start a built-in agent. Personal logistics data is excluded from source packages.

USB and Wi-Fi

Choose either transport in Device settings. For first-time wireless setup, connect an unlocked, trusted iPhone over USB, select it and explicitly choose 为这部手机准备无线连接 / Prepare wireless. This saves a private pairing record locally. Then choose Wi-Fi, keep the phone and Mac on the same network and refresh. Existing network-paired devices can also be discovered. Normal connection and recovery never pair a new device. Wi-Fi mode cannot silently fall back to USB, and recovery retains the selected transport. Wireless code has simulated regression coverage. In the recorded physical test, existing-pair identity validation passed but wireless tunnel establishment failed; cable-free control has not passed acceptance.

If discovery is unavailable, enter the phone’s private IPv4 address in Wi-Fi settings. The existing pairing and returned device identity must match the selected target. A saved pairing does not establish availability; update the address if DHCP changes it.