iospilot

An MCP server that lets an LLM agent drive a real iPhone (iOS 27+) from Windows, Linux or macOS, with nothing installed on the phone: no WebDriverAgent, no XCTest runner, no signing, no Mac.

It talks to Apple's CoreDevice developer services over USB through opendevice:

• Screen as text: the accessibility tree of the foreground app with frames, compressed into a short numbered element list. One read takes ~1 s.

• Input: HID touch (tap, long press, drag, swipe) and a virtual keyboard, as if a finger and a Bluetooth keyboard were used. Text of any script is entered by pasting.

• Waits like a person: the phone streams its screen as HEVC at ~60 fps even when nothing moves. A frame that repeats the last picture is a few hundred bytes, any visible change is far more, so the frame sizes tell when the screen has stopped changing, with no decoding. Every action waits for that, then returns the new element list. One round trip per step.

This is the iPhone sibling of droidpilot and keeps its tool surface where it can.

Requirements

• An iPhone on iOS 27 or later (screen streaming and HID input over CoreDevice start there), with Developer Mode on (Settings → Privacy & Security → Developer Mode).

• USB cable, the phone unlocked and trusting this computer.

• Windows: Apple's device drivers (the Apple Devices app from the Microsoft Store, or iTunes). Linux: usbmuxd.

• Python 3.10+.

The developer disk image must be mounted once per phone boot. opendevice can do it:

opendevice mounter mount        # or: pymobiledevice3 mounter auto-mount

Related MCP server: mcp-ios

Setup

git clone https://github.com/WaziXuan/iospilot && cd iospilot
python -m venv .venv && .venv/bin/pip install -e .      # Windows: .venv\Scripts\pip install -e .

Register it with your MCP client, e.g. Claude Code:

claude mcp add iospilot -- /path/to/iospilot/.venv/bin/python -m iospilot

With several phones connected, pass --udid <udid> (or set IOSPILOT_UDID).

Quick checks without an agent:

python scripts/mcp_call.py                    # list the tools
python scripts/mcp_call.py observe            # what is on screen
python scripts/smoke.py                       # end-to-end run inside the Settings app

What the agent sees

app: Settings (com.apple.Preferences)  screen: 402x874pt  elements: 14
[e1] Header "Settings" @(79,141) 126x38 header
[e2] Button "Zixuan W, Apple Account, iCloud+, and more" @(201,213) 370x90
[e6] Switch "Airplane Mode" ='0' @(201,470) 370x50 off
[e7] Button "Wi-Fi, HKU-SH" @(201,520) 370x50
[e13] SearchField "Search" @(185,822) 346x28 edit

• [eN] is a handle for action tools; it is valid for the latest list only.

• Coordinates are iOS points. @(x,y) is where a tap lands: the centre of the element's visible part, not of its frame, when something covers part of it.

• Flags: edit, on/off, selected, disabled, header, adjustable, embedded (drawn by another process, e.g. a Home Screen widget, whose frames are widget-local).

• The header says keyboard: shown (covers y>=…) when the on-screen keyboard is up; then its action key (Search/Go/Send/Return) is listed as a Key element.

• hidden: N counts elements left out because they are covered or off screen.

Tools

Action tools return ok (settled in N ms; read in M ms) and the new element list.

Measurements

iPhone 17 Pro Max (iPhone19,2), iOS 27.2, USB, Windows 11:

Design notes

Things found on the device that the code depends on:

• First payload only. The accessibility daemon answers a request with an InitialDump (~1 s; labels, traits, frames) and ~0.5 s later a ~300 kB AdditionalData payload. The first is complete for our purposes, so reads return there.

• One connection per read. A connection kept open was still receiving that 300 kB payload while the next tap or a screenshot ran, which garbled the DTX framing over opendevice's userspace TCP stack. A new connection costs 10–100 ms and fixed it. About one request in ten right after a tap gets no answer; it is retried on a new connection after 2.2 s.

• Settling on frame sizes. Still frames are ~500 B on a Settings page, ~1.2 kB on a Home Screen whose clock icon ticks (a ~19 kB frame each second). The still-frame floor follows the recent frames, capped so a transition can't become the new normal. A Home button press blocks ~0.4 s and its animation starts after that, so "nothing changed" is timed from when the action returned, not from when it started.

• Scrolls that stop. iOS takes the release velocity from the last touch samples. Resting the finger without reporting it does nothing; the finger eases in and then keeps reporting its position at 60 Hz for 0.15 s, so lists stop where they were dragged and pages overlap.

• Text by pasting. Letters typed on the (virtual) hardware keyboard go through the active input method: with Pinyin on, "wifi" became an uncommitted composition and later "我i分". The text is put on the pasteboard and pasted with Cmd+V instead. iOS then asks "Settings" would like to paste from "dtpasteboardd"; iospilot taps Allow Paste (exact label only) and then restores the user's clipboard.

• Return only with a keyboard. With no text field focused, a hardware Return activates the control that has keyboard focus and once opened an unrelated Settings page. press_key enter taps the keyboard's action key when one is shown and refuses otherwise.

• Occlusion without z-order. The dump has no z-order. Something smaller over part of an element, or a control over plain content (a tab bar item over the last chat row), counts as on top. iOS 27's glass search bar takes touches ~30 pt above the field it reports, so a band of 40 pt above bottom search fields and narrow floating buttons counts as covered too.

• Role from traits. The role description sometimes arrives only with the second payload, so roles come from UIAccessibilityTraits (including the private toggle bit, 1 << 53).

• Which "Settings". Two installed bundles are named Settings (com.apple.Preferences and com.apple.CarPlaySettings); app lookup prefers user apps, then non-CarPlay, non-internal ones.

Limitations

• iOS 27+ only, and USB only for now (opendevice has a Wi-Fi mode that iospilot doesn't use yet).

• Reading the screen costs ~1 s, far slower than an on-device XCTest runner (~0.1 s); that is the price of installing nothing on the phone.

• Portrait was tested; landscape frames are not handled specially.

• Apps that draw their own UI (games, maps, canvases) expose little or no accessibility tree; use screenshots and point taps there.

• Home Screen widget elements carry widget-local coordinates (flagged embedded).

• The paste prompt is answered for English and Chinese UIs.

• After a phone reboot the developer disk image must be mounted again.

Development

.venv/bin/pip install -e ".[dev]"
.venv/bin/python -m pytest

The unit tests need no device. scripts/smoke.py and scripts/mcp_call.py drive a real phone.

Related projects

• opendevice: the CoreDevice client this builds on.

• droidpilot: the same idea for Android emulators.

• iphone-use, SideTap, mobile-mcp: iPhone control through an XCTest runner (WebDriverAgent or their own) on the phone; faster reads, but the runner must be built, signed and installed.

License

MIT