# iospilot

An MCP server that lets an LLM agent drive a real iPhone (iOS 27+) from Windows, Linux or macOS,
with **nothing installed on the phone**: no WebDriverAgent, no XCTest runner, no signing, no Mac.

It talks to Apple's CoreDevice developer services over USB through
[opendevice](https://github.com/oops-workplace/opendevice):

- **Screen as text**: the accessibility tree of the foreground app with frames, compressed into a
  short numbered element list. One read takes ~1 s.
- **Input**: HID touch (tap, long press, drag, swipe) and a virtual keyboard, as if a finger and a
  Bluetooth keyboard were used. Text of any script is entered by pasting.
- **Waits like a person**: the phone streams its screen as HEVC at ~60 fps even when nothing moves.
  A frame that repeats the last picture is a few hundred bytes, any visible change is far more, so
  the frame sizes tell when the screen has stopped changing, with no decoding. Every action waits
  for that, then returns the new element list. One round trip per step.

This is the iPhone sibling of [droidpilot](https://github.com/WaziXuan/droidpilot) and keeps its
tool surface where it can.

## Requirements

- An iPhone on **iOS 27 or later** (screen streaming and HID input over CoreDevice start there),
  with **Developer Mode** on (Settings → Privacy & Security → Developer Mode).
- USB cable, the phone unlocked and **trusting this computer**.
- Windows: Apple's device drivers (the *Apple Devices* app from the Microsoft Store, or iTunes).
  Linux: `usbmuxd`.
- Python 3.10+.

The developer disk image must be mounted once per phone boot. opendevice can do it:

```bash
opendevice mounter mount        # or: pymobiledevice3 mounter auto-mount
```

## Setup

```bash
git clone https://github.com/WaziXuan/iospilot && cd iospilot
python -m venv .venv && .venv/bin/pip install -e .      # Windows: .venv\Scripts\pip install -e .
```

Register it with your MCP client, e.g. Claude Code:

```bash
claude mcp add iospilot -- /path/to/iospilot/.venv/bin/python -m iospilot
```

With several phones connected, pass `--udid <udid>` (or set `IOSPILOT_UDID`).

Quick checks without an agent:

```bash
python scripts/mcp_call.py                    # list the tools
python scripts/mcp_call.py observe            # what is on screen
python scripts/smoke.py                       # end-to-end run inside the Settings app
```

## What the agent sees

```text
app: Settings (com.apple.Preferences)  screen: 402x874pt  elements: 14
[e1] Header "Settings" @(79,141) 126x38 header
[e2] Button "Zixuan W, Apple Account, iCloud+, and more" @(201,213) 370x90
[e6] Switch "Airplane Mode" ='0' @(201,470) 370x50 off
[e7] Button "Wi-Fi, HKU-SH" @(201,520) 370x50
[e13] SearchField "Search" @(185,822) 346x28 edit
```

- `[eN]` is a handle for action tools; it is valid for the latest list only.
- Coordinates are iOS **points**. `@(x,y)` is where a tap lands: the centre of the element's
  visible part, not of its frame, when something covers part of it.
- Flags: `edit`, `on`/`off`, `selected`, `disabled`, `header`, `adjustable`, `embedded`
  (drawn by another process, e.g. a Home Screen widget, whose frames are widget-local).
- The header says `keyboard: shown (covers y>=…)` when the on-screen keyboard is up; then its
  action key (Search/Go/Send/Return) is listed as a `Key` element.
- `hidden: N` counts elements left out because they are covered or off screen.

## Tools

| Tool | What it does |
|---|---|
| `observe` | Element list, optionally with a screenshot |
| `screenshot` | JPEG; at the default width of 402 px, 1 px = 1 point |
| `tap`, `tap_text` | Tap a handle, `x,y` points, or the first element whose label/value contains a text |
| `long_press` | Press and hold (context menus) |
| `swipe` | From one handle/point to another; `hold_ms` turns it into a drag |
| `scroll` | `down`/`up`/`left`/`right`, by ~60% of the screen or of an element, without fling by default |
| `type_text` | Enter text into the focused field (any script); `clear=True` empties it first |
| `press_key` | `back`, `home`, `lock`, volume keys, `enter` (taps the keyboard's action key), arrows… |
| `wait_for`, `wait_idle` | Wait for a text, or for the screen to stop changing |
| `app_launch`, `app_stop`, `app_list` | Apps by name ("Settings", "微信") or bundle id |
| `status` | Device, iOS version, screen size, latest timings |

Action tools return `ok (settled in N ms; read in M ms)` and the new element list.

## Measurements

iPhone 17 Pro Max (iPhone19,2), iOS 27.2, USB, Windows 11:

| | |
|---|---|
| Connect (tunnel + stream) | ~0.5 s |
| Read the element list | 0.9–1.2 s |
| Tap → settled → new list | ~1.5–2.5 s (opening a page ~2 s, no change ~1.4 s) |
| Scroll a list | ~2 s |
| Enter text (paste + iOS's paste prompt) | ~2.5 s |
| Screenshot | ~1.5 s |

## Design notes

Things found on the device that the code depends on:

- **First payload only.** The accessibility daemon answers a request with an *InitialDump*
  (~1 s; labels, traits, frames) and ~0.5 s later a ~300 kB *AdditionalData* payload. The first
  is complete for our purposes, so reads return there.
- **One connection per read.** A connection kept open was still receiving that 300 kB payload
  while the next tap or a screenshot ran, which garbled the DTX framing over opendevice's
  userspace TCP stack. A new connection costs 10–100 ms and fixed it. About one request in ten
  right after a tap gets no answer; it is retried on a new connection after 2.2 s.
- **Settling on frame sizes.** Still frames are ~500 B on a Settings page, ~1.2 kB on a Home
  Screen whose clock icon ticks (a ~19 kB frame each second). The still-frame floor follows the
  recent frames, capped so a transition can't become the new normal. A Home button press blocks
  ~0.4 s and its animation starts after that, so "nothing changed" is timed from when the action
  returned, not from when it started.
- **Scrolls that stop.** iOS takes the release velocity from the last touch samples. Resting the
  finger without reporting it does nothing; the finger eases in and then keeps reporting its
  position at 60 Hz for 0.15 s, so lists stop where they were dragged and pages overlap.
- **Text by pasting.** Letters typed on the (virtual) hardware keyboard go through the active
  input method: with Pinyin on, "wifi" became an uncommitted composition and later "我i分". The
  text is put on the pasteboard and pasted with Cmd+V instead. iOS then asks
  *"Settings" would like to paste from "dtpasteboardd"*; iospilot taps *Allow Paste* (exact
  label only) and then restores the user's clipboard.
- **Return only with a keyboard.** With no text field focused, a hardware Return activates the
  control that has keyboard focus and once opened an unrelated Settings page. `press_key enter`
  taps the keyboard's action key when one is shown and refuses otherwise.
- **Occlusion without z-order.** The dump has no z-order. Something smaller over part of an
  element, or a control over plain content (a tab bar item over the last chat row), counts as
  on top. iOS 27's glass search bar takes touches ~30 pt above the field it reports, so a band
  of 40 pt above bottom search fields and narrow floating buttons counts as covered too.
- **Role from traits.** The role description sometimes arrives only with the second payload,
  so roles come from UIAccessibilityTraits (including the private toggle bit, 1 << 53).
- **Which "Settings".** Two installed bundles are named Settings (`com.apple.Preferences` and
  `com.apple.CarPlaySettings`); app lookup prefers user apps, then non-CarPlay, non-internal ones.

## Limitations

- iOS 27+ only, and USB only for now (opendevice has a Wi-Fi mode that iospilot doesn't use yet).
- Reading the screen costs ~1 s, far slower than an on-device XCTest runner (~0.1 s); that is
  the price of installing nothing on the phone.
- Portrait was tested; landscape frames are not handled specially.
- Apps that draw their own UI (games, maps, canvases) expose little or no accessibility tree;
  use screenshots and point taps there.
- Home Screen widget elements carry widget-local coordinates (flagged `embedded`).
- The paste prompt is answered for English and Chinese UIs.
- After a phone reboot the developer disk image must be mounted again.

## Development

```bash
.venv/bin/pip install -e ".[dev]"
.venv/bin/python -m pytest
```

The unit tests need no device. `scripts/smoke.py` and `scripts/mcp_call.py` drive a real phone.

## Related projects

- [opendevice](https://github.com/oops-workplace/opendevice): the CoreDevice client this builds on.
- [droidpilot](https://github.com/WaziXuan/droidpilot): the same idea for Android emulators.
- [iphone-use](https://github.com/leeguooooo/iphone-use), [SideTap](https://github.com/ucsandman/sidetap),
  [mobile-mcp](https://github.com/mobile-next/mobile-mcp): iPhone control through an XCTest runner
  (WebDriverAgent or their own) on the phone; faster reads, but the runner must be built, signed
  and installed.

## License

MIT