phone-use
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
Phone Use
Give your agent a phone. A free, open-source CLI for the iOS Simulator and Android.

Website · Docs · npm · Cloud waitlist
Phone Use lets a coding agent see and operate a phone. It reads the screen as named elements, taps and types by label, and reports after every action whether the screen changed. It runs on your own machine with no account, as a CLI, an MCP server and a TypeScript SDK.
It also maps an app into a reusable graph as it explores, then navigates that map deterministically, with zero model calls.
Quick start
Phone Use is free on your own machine and needs no account. With nothing else set up, every verb drives the iOS Simulator on your Mac:
curl -fsSL https://phoneuse.dev/install | bash
open -a Simulator # macOS + Xcode; give it a few seconds to boot
phone-use observe
phone-use open "Weather"Or let your coding agent set itself up. Paste this into Claude Code, Codex or Cursor:
Install phone-use with `curl -fsSL https://phoneuse.dev/install | bash` and add its skill with `npx skills add Derivate-Inc/phone-use-skills --yes`. If `phone-use` is not found afterwards, use the path the installer printed. Run `phone-use doctor`, then open the iOS Simulator (or run `phone-use connect android` for an Android device), run `phone-use observe`, and tell me what is on the screen.The two pieces on their own:
npx skills add Derivate-Inc/phone-use-skills # teach any skills-compatible agent the CLI
claude mcp add --scope project phone-use -- npx -y phone-use mcp # or register the MCP serverThe same verbs drive four kinds of phone:
# nothing at all → the local iOS Simulator
phone-use connect android # an Android device or emulator over adb (macOS, Linux)
phone-use connect # the real iPhone on your desk (macOS + iPhone Mirroring)
phone-use create ios # a cloud iPhone: waitlist, https://phoneuse.dev/waitlistCloud phones are on a waitlist. They run on our hardware and there are only
a few. phone-use create from an account that is not approved creates nothing:
it prints how to continue on the local simulator and where to join the
waitlist, and exits non-zero so a script
notices. Everything that runs on your own machine sends us nothing.
connect is the only way to reach real App Store apps and real accounts —
simulators cannot run them. See
docs/real-iphone.md for what that path
can and cannot do.
Drive a device directly with the SDK (macOS + Xcode):
import { ios } from '@phone-use/sdk';
const device = await ios.launch(); // dedicated simulator, deleted on close()
await device.apps.open('Settings');
await device.tap('General');
await device.close();Or run the full harness from a checkout:
bun install
# start the MCP server (also serves the live viewer at localhost:4936)
bun run mcp
# one-off natural-language task (needs ANTHROPIC_API_KEY)
bun run task "open Settings and go to Accessibility"
# benchmarks: deterministic layer / full LLM harness (subscription via headless claude)
bun run bench
bun run bench:agentRegister with an MCP client (already wired in .mcp.json for Claude Code):
claude mcp add --scope project phone-use -- bun /path/to/phone-use/packages/phone-use/src/index.ts mcpRelated MCP server: mobile-mcp
How it is built
The runtime drives the iOS Simulator and real iPhones on macOS, and Android devices or emulators over adb on any OS. The SDK keeps the device transport behind a private backend boundary, and the transport is ours end to end: an XCTest runner for iOS simulators and a small on-device helper for Android, with zero third-party runtime dependencies.
Packages
Package | What it is |
Typed device runtime SDK: | |
The harness on top of the SDK: MCP server + CLI + embeddable agent loop, permission floor, App Cartographer maps, skills, live viewer, trace artifacts. Requires Bun. |
Why this exists
Desktop "computer use" got good. Phones stayed unexplored — partly because you can't "view source" on a native app; you have to physically drive the UI. Phone Use closes that gap with a driver tuned for phones (accessibility + vision + coordinate taps), an interactive cursor you can watch live, and an App Cartographer that turns exploration into a durable, shareable map.
What's inside
Driver & interaction (packages/phone-use/src/driver.ts)
Fast capture — direct
simctlJPEG (~110ms, 4× faster than the toolkit default)Compressed observation — filters accessibility noise, adds geometry; a Hacker News page goes from 485 noisy nodes to ~134 clean ones
Action ladder — deep links → verified taps → scroll → type → system-alert handling → zoom-on-region for tiny targets
Vision fallback — Set-of-Marks screenshots + raw coordinate taps for canvas/map screens accessibility can't see
Change evidence on every tap — the runner compares the accessibility tree just before and after the tap and reports whether it changed (or that it could not tell), so most no-op taps show up at once. It is evidence of a change, not proof the intended thing happened. A stale or ambiguous ref, or a control something covers, is refused instead of tapped
Interactive cursor & live viewer (packages/phone-use/src/annotate.ts, packages/phone-use/src/live.ts)
A real visible pointer with move-then-click, element boxes, and click pulses composited into every frame
A browser live viewer (
localhost:4936): MJPEG screen video + a 60fps client-side cursor overlay — watch the agent drive in real time
Cloud iOS sessions also expose an authenticated Open phone page with native H.264 video (30fps target), browser decoding, and automatic reconnection.
App Cartographer — the novel core (packages/phone-use/src/explore.ts, packages/phone-use/src/appmap.ts)
Autonomous crawl → a graph of screens (nodes) and transitions (edges), keyed by a structural signature stable across dynamic content
Directed search — for a specific destination, a depth-first crawl that stops the moment it's reached (a depth-2 setting is ~3 taps, not a full app map); full breadth-first crawl available for complete maps
Deterministic navigation —
goto("Fonts")computes the shortest tap-path and executes it with zero model calls; cached routes replay in a few secondsPersistent & self-enriching — maps save to
~/.phone-use/maps/<bundle>.json, reload across sessions, and grow richer every time they're used; crawled edges carry semantic docs (what tapping each element does —opens "General",reveals …), so the map is functional knowledge, not just topologyShareable — exploration compounds across agents —
export_atlasbundles the learned maps + skills into one portable file; another phoneimport_atlases it and navigates instantly with zero crawl (a fresh phone didgoto Aboutin 5s from an inherited map vs ~23s to build one). Imported maps merge, so knowledge accretes.inspect_atlaspreviews what a bundle would add (new / enriches / redundant per app, new vs existing skills) before you trust it — and flags a map's age (⚠ may be stalepast 30 days), since maps are stamped with a save timeSelf-extending — auto-builds a map when none exists; crawls deeper on demand for unmapped screens
Self-healing — when a saved path breaks (the app updated), it detects the failure, re-maps, diffs what changed, and routes the new way
Intent-based control — ask & act by name (packages/phone-use/src/task.ts, packages/phone-use/src/apps.ts)
ask("iOS Version")— read any value without knowing which screen (or even which app) it's on: searches every mapped app, resolves the one that has it, navigates, and readstoggle_setting("Full-Screen Previews", on=true)— flip a switch by name, auto-finding its screen; idempotent desired-statetap("Send")— press a control by name (exact or substring): the action verb for buttons like "Done", "Add", "New". Taps in place, scrolling to find it if needed. Complementstoggle_setting(switches) so skills can do create/compose errands, not just read/navigateset_field("Ada", field="First name")— type into a form field by its label/placeholder (the input verb for forms;searchis for search bars). Withtapthis completes compose/create errands — open Contacts → tap Add → set First/Last name → tap Done is one deterministic skill that actually creates the contactsearch("Maps", "coffee")— the input primitive: focus an app's search field and type, revealing it whether it's a large-title bar hidden above the list (Settings) or a collapsed bottom card (Maps). Completes the intent vocabulary: navigate (goto), read (ask), act (toggle_setting), input (search)Whole-phone — any app by natural name (
"Settings"→com.apple.Preferences), sinceopenAppis the cross-app transition and a registry maps names to bundle idsThe atlas becomes a queryable, controllable knowledge base of the phone that gets faster and richer as it's explored —
atlas_overview(orphone-use atlas) shows what's been learned: every mapped app (screens, transitions, freshness) and every saved skill, so an agent can check what it already knows before exploring
Phone Skills — replayable cross-app composites (packages/phone-use/src/skills.ts)
A skill is a named, ordered chain of the intent primitives with variable capture: any step names its result with
as, later steps interpolate it as{{name}}. So a cross-app errand — read a value in app A, use it in app B — is one saved artifactRecorded once, replayed forever with zero model calls — the deterministic composite layer on top of the primitives (the "agent runs your phone errands" payoff)
Resilient replay — each step retries on transient failure (cold-launch timing, mid-animation taps, focus races); safe because an op reports failure only when it didn't act, so a retry never double-acts
Checked, not fire-and-hope — a
verifystep asserts a postcondition ("Delivered" showed, the contact saved), so a skill confirms its own success; with retries it also polls, doubling as wait-for-conditionLearn by doing —
record_skill→ perform the task once →stop_recording: every primitive that succeeds is captured as a step, so the phone learns an errand by watching it, no hand-authoringGeneralize —
parameterize_skillpromotes a recorded literal to a{{param}}slot, so one recording of "open coffee in Maps" replays for any place (run_skill args={place:"sushi"})Skills persist to
~/.phone-use/skills/<name>.json; run saved-by-name or ad-hoc. MCP:run_skill,save_skill,list_skills,record_skill,stop_recording,parameterize_skillVerified: a skill that asks Settings for the model name, captures it, then searches Maps for it runs end-to-end — cross-app, with substitution
Live atlas (packages/phone-use/src/live.ts, packages/phone-use/src/vizbus.ts)
The viewer shows the phone streaming on one side and the app map drawing itself in real time on the other, as the agent crawls
Surfaces (packages/phone-use/src/mcp.ts, packages/phone-use/src/agent.ts)
MCP server —
observe(delta-rendered),batch,verify,press/press_at,zoom_region,apps,map_app,goto,ask,toggle_setting, skills & atlas tools, … usable by Claude Code, Codex, or any MCP clientBuilt-in agent loop — Claude drives the phone directly (needs an API key); same harness rules as the MCP path (milestones, two-strike, reflection-on-demand)
Benchmark
The device layer has a regression eval (eval/): 33 probes on a test app that records what it actually received, so the tool never grades itself, plus a multi-agent tier. Its headline is the false-success rate (the tool said OK, the app says otherwise). phone-use was tuned against these probes, so its 0% there is a regression bar, not evidence of reliability in general. It measures single device actions, not whether an agent completes real tasks. bun run bench below is a smoke test of the high-level verbs.
bun run bench runs a suite of phone tasks, each a skill that ends in a verify text check on the final screen, from a clean state, timed, across the capability classes. Latest simulator run:
Phone Use benchmark — 6/6 passed, avg ~6.3s/task
✓ read-ios-version read 3658ms goto Settings→About, verify "26.1"
✓ deep-nav-accessibility navigate 2860ms goto Accessibility, verify "Vision"
✓ search-open-maps search+act 8269ms search Maps "coffee" → open, verify
✓ toggle-setting toggle 3581ms set a switch by name (idempotent)
✓ phone-wide-ask ask-anywhere 5832ms resolve which app holds a fact, read it
✓ create-contact create 13506ms Add → fill name → Done → verifybench --json emits machine-readable results for CI (exit code nonzero on any failure). It is a repeatable smoke test on stock apps the tool was tuned on; its checks are text on the final screen (and the toggle task trusts its own result), so passing it does not show generalization. Building it surfaced (and fixed) real races the piecemeal tests missed.
bun run smoke runs the headline-feature regression guards in one command — the recorder over the real MCP wire, the shared-atlas inherit-and-navigate round-trip, and the full skill lifecycle (record → parameterize → replay → forget) — each asserting live on the simulator.
Docs
Product documentation lives with the site under
apps/landing/app/docs/ and is served at
https://phoneuse.dev/docs. Run bun run docs:dev and open
http://localhost:3000/docs.
Technical write-ups are in docs/: start with
docs/README.md. The novel capability is written up in
docs/15-app-cartographer.md.
Verified end-to-end
Driven by Claude Code through the MCP server alone, in one ~33s session on the simulator:
"Tell me this iPhone's iOS version and model name (use
ask), then turn on Full-Screen Previews (toggle_setting on=true)."
The agent called ask("iOS Version") → 26.1 and ask("Model Name") → iPhone 16e (both auto-resolved to Settings → About without being told the screen), then toggle_setting("Full-Screen Previews", on=true) → auto-found the switch and set it 0 → 1. No screen names, no bundle ids, no step-by-step navigation.
Every layer — driver, cursor, live viewer, map building, deterministic goto, self-extension, self-healing, whole-phone ask/toggle, and the live atlas — is verified on the iOS Simulator.
Status
iOS Simulator and Android (device or emulator): the main paths. Every layer above is verified on the iOS Simulator.
Real iPhone: works through iPhone Mirroring on macOS 15 or newer (
phone-use connect). It reads the screen with text recognition, so it is rougher than the simulator; real-iphone.md lists what it can and cannot do. Driving a physical iPhone with the XCTest runner (Developer Mode + code signing) is not wired up yet.Cloud phones: on a waitlist.
Phone Use is free, Apache-2.0, and comes as it is.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Launch and verify high-fidelity mobile simulations from an AI agent.
Let your AI agent control real iPhones: screenshots, tap, swipe, type and open apps.
Let AI agents control Android and iOS devices on Mobilerun Cloud: screen, input, apps, workflows.
Open a React Native app in a simulated phone: read the screen, tap, type, set device state.
Related MCP Servers
- AlicenseBqualityDmaintenanceEnables AI agents to inspect, log, and control React Native apps on Android emulators or iOS simulators, including UI tree, tap, scroll, and hot reload.2246 npm5MIT
- AlicenseAqualityCmaintenanceEnables agents and LLMs to interact with native iOS and Android applications and devices via structured accessibility snapshots or coordinate-based taps, supporting simulators, emulators, and real devices.3945,799 npmApache 2.0
- AlicenseBqualityBmaintenanceEnables AI agents to automate Android and iOS devices, including emulators and physical devices, by capturing screenshots, reading UI accessibility trees, performing touch and text input, and managing app lifecycle and logs.184MIT
- AlicenseAqualityBmaintenanceEnables agents and LLMs to automate, inspect, and scrape native iOS and Android apps across simulators, emulators, and real devices through a platform-agnostic interface using accessibility snapshots or screenshot-based coordinates. It provides device, app, screen, input, log, and crash management tools, including optional cloud device reservation.34Apache 2.0