# jev-mobile-pilot

Let coding agents drive iOS Simulators and Android devices without spending their own tokens on every tap.
Code reads the screen's accessibility tree, [TypeSafe](https://typesafe.ai)'s Jev model picks each step,
and your agent only plans and answers the occasional question.

> [!IMPORTANT]
> - **Platforms:** iOS Simulator (macOS with Xcode, through AXe) and Android emulators and devices (through adb).
>   Real iPhones are not supported. Real Android phones work only if the person who starts the server allows them.
> - **Screen text leaves your machine.** Each decision sends the labels and values on the screen, your goal,
>   hints and param names to TypeSafe's API. Screenshots are never sent to Jev. See [What is sent where](#what-is-sent-where).
> - This is an independent project. It is not affiliated with or endorsed by TypeSafe, Apple or Google.

```text
your agent ──mobile_task("Open Settings > General > About")──▶ jev-mobile-pilot
                                                                │  loop:
                                                                │   read the accessibility tree → compact list of elements
                                                                │   Jev: which element next? is the goal done?
                                                                │   safety check → tap / type / scroll
◀── done · 2 steps · 5 Jev calls · $0.00017 · 6 s ──────────────┘
```

## Why

General device tools for agents ([below](#related-projects)) leave every step to the calling model:
read the screen, pick a button, tap, read again. On a long flow that is many turns of a large model.
jev-mobile-pilot hands those small decisions to Jev, which answers typed questions (pick one of these options,
yes or no) in a fraction of a second for a fraction of a cent, and comes back to your agent only when
it is unsure, blocked, or about to do something it cannot undo.

## Requirements

- Node.js 22 or later
- A TypeSafe API key in the environment variable `TYPESAFE_API_KEY`
- For iOS: macOS with Xcode, an iOS Simulator runtime and a booted simulator, and
  [AXe](https://github.com/cameroncooke/AXe) (tested with 1.8.0): `brew install cameroncooke/axe/axe`
- For Android: `adb` from the Android SDK platform-tools (tested with 37.0.1) on your `PATH` (or set `ADB_PATH`)
  (`brew install --cask android-platform-tools` on macOS), and an emulator or a device with USB debugging on.
  Tested on macOS with Android 16.

You need only the tools for the platform you use. Keep the key in your shell profile or a secrets manager,
not in a repository or an MCP config that is committed.

## Install

jev-mobile-pilot is not published to npm. Build it from source:

```sh
git clone https://github.com/heyk1njo/jev-mobile-pilot.git
cd jev-mobile-pilot
npm ci
npm run build
node dist/cli.js doctor
```

`doctor` checks AXe and adb and their versions, the devices that are ready, the API key (with one tiny Jev call)
and the safety settings. One of AXe or adb is enough; the other shows as a warning.

### Claude Code

```sh
claude mcp add -s project jev-mobile-pilot -- node /absolute/path/to/jev-mobile-pilot/dist/cli.js mcp
```

Claude Code starts the server with its own environment, so an exported `TYPESAFE_API_KEY` is enough.

### Other MCP clients

Run `node /absolute/path/to/jev-mobile-pilot/dist/cli.js mcp` as a stdio server. If your client does not pass its
environment to MCP servers, set `TYPESAFE_API_KEY` in the client's server config and keep that file out of git.

## Use it from an agent

The MCP server has eight tools:

| Tool | What it does | Calls Jev |
| --- | --- | --- |
| `mobile_devices` | List simulators, emulators and devices with their ID, platform and state | no |
| `mobile_observe` | The current screen, one line per element, with refs `e1`, `e2`... | no |
| `mobile_act` | One action: `tap` a ref, `type` text, press a `key`, `scroll`, go `back`, `launch` an app | no |
| `mobile_find` | The three elements that best match a description | yes |
| `mobile_task` | Run a multi-step goal until it is done or needs you | yes |
| `mobile_answer` | Continue a task that asked a question or paused | yes |
| `mobile_screenshot` | A PNG for your agent to look at (not sent to Jev) | no |
| `mobile_doctor` | Check the setup | one tiny call |

Every tool but `mobile_devices` and `mobile_doctor` takes an optional `device`: a simulator UDID or an adb serial.
With only one device ready, it can be left out. `launch` takes a bundle ID on iOS and a package name on Android.

A typical call:

```json
{
  "goal": "Start a new contact and type the name into the First name field",
  "params": { "name": { "value": "Taro" } },
  "hints": ["Contacts has a + button at the top right"]
}
```

Write the goal, plan and hints in English. Values to type go in `params`, never in the goal; mark passwords
and tokens `"secret": true`. For flows that cross several screens, pass a `plan` of milestones:
`[{ "do": "Open General", "done_when": "The General screen is shown" }, ...]`.
Write `done_when` as something visible on the screen ("The Profile tab is selected", "A General heading is shown"),
not a conclusion such as "the user's own profile is shown": Jev only sees the screen's text and controls.

`mobile_observe` output looks like this (an illustrative screen):

```text
app: Settings (pid 4242) · 402x874 · screen 9f2c41d0a7b3
e1 button "Settings"
- heading "General"
e2 button "About"
e3 button "Software Update"
e4 switch "Background App Refresh" value="1"
e5 button "Transfer or Reset iPhone" (off-screen below)
```

### Task results

`mobile_task` and `mobile_answer` run for up to about 45 seconds and return one of these statuses, the latest screen,
cost so far, and the path of a trace file:

| Status | Meaning | What to do |
| --- | --- | --- |
| `done` | Jev judged the goal reached from the screen | Check the screen before relying on it |
| `needs_input` | A question for you (see below) | Answer with `mobile_answer` |
| `paused` | The 45-second window ran out | Call `mobile_answer` with only `task_id` |
| `budget_exhausted` | `max_steps` (default 20) or `budget_usd` (default 0.05) reached | `mobile_answer` with `more_steps`, or stop |
| `failed` | Something went wrong that a retry will not fix | Read the message and the trace |

Questions carry a `kind`:

| Kind | When |
| --- | --- |
| `pick` | Jev is not confident which element is next; the options are listed with probabilities |
| `confirm` | The next step looks irreversible and irreversible steps are allowed |
| `refused` | The next step looks irreversible and they are not allowed |
| `not_found` | Nothing on the screen matches, even after scrolling |
| `stuck` | The screen stopped changing |
| `blocker` | A login wall, permission prompt or similar needs a person |
| `app_changed` | Another app came to the front |
| `action_failed` | The device tool reported an error; the action is not sent again on the same screen unless you pick it |

`mobile_answer` takes `pick` (an option or any ref on the returned screen), `hint`, `plan`, `more_steps`,
`confirm` or `abort`. `confirm: true` only answers a pending `confirm` question. With nothing but `task_id`, it
reads the screen again and goes on.

## Use it from the command line

```sh
jev-mobile-pilot devices
jev-mobile-pilot observe --device <ID>
jev-mobile-pilot run "Open Settings > General > About"
jev-mobile-pilot run "Type the name into the First name field of a new contact" --param name=Taro
jev-mobile-pilot run "Sign in" --param user=me@example.com --secret password=MY_APP_PASSWORD --submit password
```

(Use `node dist/cli.js` in place of `jev-mobile-pilot` unless you ran `npm link`.)
With more than one device ready, pass `--device`. Add `--allow-physical` to drive a real Android phone. `run` answers a `paused` result itself and stops at any question.
Exit codes: `0` done, `1` failed or error, `2` needs input, `3` budget exhausted. `jev-mobile-pilot --help` lists every option.

## Safety

- **Irreversible steps are refused by default.** Before each tap and each Return key press (a `submit` param,
  `mobile_act` with `key: "return"`, or text that contains a newline, which presses Return as it is typed), a word list
  (pay, buy, send, delete, discard, sign out, like, follow, comment, and Japanese equivalents) and a Jev yes/no
  question judge whether the step can be undone. If Jev cannot answer, the step counts as irreversible.
  Text fields are exempt.
- Only the person who starts the server can allow them, with `--allow-irreversible` or
  `JEV_MOBILE_PILOT_ALLOW_IRREVERSIBLE=1`. No tool argument can. Even then each such step returns a `confirm` question,
  and `run` on the command line never confirms.
- **Real Android phones are off by default.** They are listed, but every other tool refuses them unless the server
  is started with `--allow-physical` or `JEV_MOBILE_PILOT_ALLOW_PHYSICAL=1`. A device counts as a real phone unless its
  system properties say it is an emulator, and the device list is read again on every call. A real phone is signed in to real accounts:
  a wrong tap can message, follow or buy for real, and its screen text goes to TypeSafe. Prefer an emulator.
- An action whose result is unknown (the device tool timed out or failed) is not sent again unless you pick it.
- One call at a time per device, and one unfinished task per device: starting another fails and names the task
  to finish or abort first. Budgets cap steps and spend for every task, and the 45-second window is checked mid-step.
- This is a guard rail for a test device, not a security boundary. Use test accounts and no real payment methods.

## What is sent where

| Data | Goes to |
| --- | --- |
| Screen text (up to 40 lines of text and 40 visible controls per decision; `mobile_find` sends up to 250 controls), goal, plan, hints, param names | TypeSafe's API |
| Values of non-secret params, after they are typed (they appear on screen) | TypeSafe's API |
| Secret params | Nowhere. They are typed on the device and replaced with `[secret:name]` everywhere else |
| Screenshots | Only to your agent, when it calls `mobile_screenshot` |
| Traces (one JSON Lines file per task) | Your disk, `$TMPDIR/jev-mobile-pilot/` by default |

On Android the status bar and notifications (`com.android.systemui`) are left out. Windows over the app, such as
dialogs and the keyboard, are read as part of the screen and sent like the rest of it.
Add `--redact REGEX` (or `JEV_MOBILE_PILOT_REDACT`, one pattern per line) to replace matching text with `[redacted]`
before it is sent or written. Do not point the tool at screens showing data you may not share with a third party.

## Cost

jev-mobile-pilot estimates cost as input tokens × $0.042 per million and stops at `budget_usd`. Check TypeSafe's
current pricing; the estimate can differ from your bill. In testing on iOS 27 simulators, short tasks
(two or three steps, about five Jev calls) cost about $0.0002 and took 6 to 18 seconds.

## Limitations

Both platforms:

- `type` sends ASCII only. Japanese and other text that needs an input method cannot be typed.
- There is no way to clear a field; `type` appends.
- Secret values shorter than 4 characters are not masked in screen text, because they would match ordinary words.
- `done` is Jev's judgment from the screen text. Check the screen.
- The on-screen keyboard is removed from the screen view.
- Only what the accessibility tree exposes is visible. Custom-drawn views without accessibility are not.

iOS, found with AXe 1.8.0, Xcode 27 and iOS 27:

- AXe's default tap does not register, so taps use `--tap-style physical`.
- The home button does not work; use `launch` with a bundle ID to switch apps.
- `back` is a drag from the left edge, because AXe's edge swipe does not work.
- React Native views with a label but no accessibility role are shown as `cell` with a warning. Setting
  `accessibilityRole` in the app fixes both the warning and the guess.
- Default iOS traits come back empty from AXe, so kinds come from element types.

Android, found with platform-tools 37.0.1 and Android 16 on a Pixel:

- Reading the screen (`uiautomator dump`) takes about 2.5 seconds, and fails while the screen keeps animating;
  it is retried twice.
- `back` is the system Back key.
- The keyboard is found with `dumpsys input_method` and `dumpsys window`. If it is shown but its frame cannot be
  read, the lower 45% of the screen is treated as keyboard.
- Whether `uiautomator dump` includes windows over the app (dialogs, the keyboard) may depend on the Android version.
- Text goes through the active keyboard app. If it is set to a mode that converts input, such as Gboard's
  Japanese kana mode, ASCII comes out converted or full-width. Switch the keyboard to an English layout first.
- `input text` turns `%s` into a space and has no escape for it, so text containing `%s` is refused.
- Text is passed to the device's `input text` command. On the device, other processes can see that command line
  while it runs, so prefer test credentials there too.

## Related projects

- [AXe](https://github.com/cameroncooke/AXe): the CLI this project uses to read and drive the iOS Simulator.
- [XcodeBuildMCP](https://github.com/cameroncooke/XcodeBuildMCP), [ios-simulator-mcp](https://github.com/joshuayoes/ios-simulator-mcp),
  [mobile-mcp](https://github.com/mobile-next/mobile-mcp) and agent-device: general device and simulator control
  for agents. The calling model decides each step. They cover far more (building, real iPhones, recording).
- [Aben25/jev-sim](https://github.com/Aben25/jev-sim): mobile-simulator computer use with Jev, iOS and Android.
- jevtest (YAML-defined flows on Android emulators) and droidrun/mobile-jev (an Android agent with Jev).

jev-mobile-pilot is narrower: the accessibility tree only, Jev for each step, and a task API that stops and asks
instead of guessing.

## Development

```sh
npm ci
npm test           # unit and engine tests with a fake device and a fake Jev
npm run typecheck
npm run build
```

Tests need no simulator, no device, no AXe, no adb and no API key. See [CONTRIBUTING.md](CONTRIBUTING.md) for the manual
check on a real simulator or emulator, and [SECURITY.md](SECURITY.md) to report a vulnerability.

## License

[MIT](LICENSE). Jev and TypeSafe are names of their owners. Apple, iOS and Xcode are trademarks of Apple Inc.
Android is a trademark of Google LLC.