Skip to main content
Glama

Phone Use

Give your agent a phone. A free, open-source CLI for the iOS Simulator and Android.

Phone Use: give your agent a phone

Website · Docs · npm · Cloud waitlist

Phone Use lets a coding agent see and operate a phone. It reads the screen as named elements, taps and types by label, and reports after every action whether the screen changed. It runs on your own machine with no account, as a CLI, an MCP server and a TypeScript SDK.

It also maps an app into a reusable graph as it explores, then navigates that map deterministically, with zero model calls.

Quick start

Phone Use is free on your own machine and needs no account. With nothing else set up, every verb drives the iOS Simulator on your Mac:

curl -fsSL https://phoneuse.dev/install | bash

open -a Simulator         # macOS + Xcode; give it a few seconds to boot
phone-use observe
phone-use open "Weather"

Or let your coding agent set itself up. Paste this into Claude Code, Codex or Cursor:

Install phone-use with `curl -fsSL https://phoneuse.dev/install | bash` and add its skill with `npx skills add Derivate-Inc/phone-use-skills --yes`. If `phone-use` is not found afterwards, use the path the installer printed. Run `phone-use doctor`, then open the iOS Simulator (or run `phone-use connect android` for an Android device), run `phone-use observe`, and tell me what is on the screen.

The two pieces on their own:

npx skills add Derivate-Inc/phone-use-skills                       # teach any skills-compatible agent the CLI
claude mcp add --scope project phone-use -- npx -y phone-use mcp   # or register the MCP server

The same verbs drive four kinds of phone:

                          # nothing at all → the local iOS Simulator
phone-use connect android # an Android device or emulator over adb (macOS, Linux)
phone-use connect         # the real iPhone on your desk (macOS + iPhone Mirroring)
phone-use create ios      # a cloud iPhone: waitlist, https://phoneuse.dev/waitlist

Cloud phones are on a waitlist. They run on our hardware and there are only a few. phone-use create from an account that is not approved creates nothing: it prints how to continue on the local simulator and where to join the waitlist, and exits non-zero so a script notices. Everything that runs on your own machine sends us nothing.

connect is the only way to reach real App Store apps and real accounts — simulators cannot run them. See docs/real-iphone.md for what that path can and cannot do.

Drive a device directly with the SDK (macOS + Xcode):

import { ios } from '@phone-use/sdk';

const device = await ios.launch(); // dedicated simulator, deleted on close()
await device.apps.open('Settings');
await device.tap('General');
await device.close();

Or run the full harness from a checkout:

bun install

# start the MCP server (also serves the live viewer at localhost:4936)
bun run mcp

# one-off natural-language task (needs ANTHROPIC_API_KEY)
bun run task "open Settings and go to Accessibility"

# benchmarks: deterministic layer / full LLM harness (subscription via headless claude)
bun run bench
bun run bench:agent

Register with an MCP client (already wired in .mcp.json for Claude Code):

claude mcp add --scope project phone-use -- bun /path/to/phone-use/packages/phone-use/src/index.ts mcp

Related MCP server: mobile-mcp

How it is built

The runtime drives the iOS Simulator and real iPhones on macOS, and Android devices or emulators over adb on any OS. The SDK keeps the device transport behind a private backend boundary, and the transport is ours end to end: an XCTest runner for iOS simulators and a small on-device helper for Android, with zero third-party runtime dependencies.

Packages

Package

What it is

@phone-use/sdk

Typed device runtime SDK: ios.launch()/connect() → Device, observe/act with portable re-resolvable Actions, typed error tree, backend registry, /testing doubles, simulator() controls (logs, recording, permissions, push). Zero runtime dependencies; runs on Node >= 22 or Bun.

phone-use

The harness on top of the SDK: MCP server + CLI + embeddable agent loop, permission floor, App Cartographer maps, skills, live viewer, trace artifacts. Requires Bun.

Why this exists

Desktop "computer use" got good. Phones stayed unexplored — partly because you can't "view source" on a native app; you have to physically drive the UI. Phone Use closes that gap with a driver tuned for phones (accessibility + vision + coordinate taps), an interactive cursor you can watch live, and an App Cartographer that turns exploration into a durable, shareable map.

What's inside

Driver & interaction (packages/phone-use/src/driver.ts)

  • Fast capture — direct simctl JPEG (~110ms, 4× faster than the toolkit default)

  • Compressed observation — filters accessibility noise, adds geometry; a Hacker News page goes from 485 noisy nodes to ~134 clean ones

  • Action ladder — deep links → verified taps → scroll → type → system-alert handling → zoom-on-region for tiny targets

  • Vision fallback — Set-of-Marks screenshots + raw coordinate taps for canvas/map screens accessibility can't see

  • Change evidence on every tap — the runner compares the accessibility tree just before and after the tap and reports whether it changed (or that it could not tell), so most no-op taps show up at once. It is evidence of a change, not proof the intended thing happened. A stale or ambiguous ref, or a control something covers, is refused instead of tapped

Interactive cursor & live viewer (packages/phone-use/src/annotate.ts, packages/phone-use/src/live.ts)

  • A real visible pointer with move-then-click, element boxes, and click pulses composited into every frame

  • A browser live viewer (localhost:4936): MJPEG screen video + a 60fps client-side cursor overlay — watch the agent drive in real time

Cloud iOS sessions also expose an authenticated Open phone page with native H.264 video (30fps target), browser decoding, and automatic reconnection.

App Cartographer — the novel core (packages/phone-use/src/explore.ts, packages/phone-use/src/appmap.ts)

  • Autonomous crawl → a graph of screens (nodes) and transitions (edges), keyed by a structural signature stable across dynamic content

  • Directed search — for a specific destination, a depth-first crawl that stops the moment it's reached (a depth-2 setting is ~3 taps, not a full app map); full breadth-first crawl available for complete maps

  • Deterministic navigation — goto("Fonts") computes the shortest tap-path and executes it with zero model calls; cached routes replay in a few seconds

  • Persistent & self-enriching — maps save to ~/.phone-use/maps/<bundle>.json, reload across sessions, and grow richer every time they're used; crawled edges carry semantic docs (what tapping each element does — opens "General", reveals …), so the map is functional knowledge, not just topology

  • Shareable — exploration compounds across agents — export_atlas bundles the learned maps + skills into one portable file; another phone import_atlases it and navigates instantly with zero crawl (a fresh phone did goto About in 5s from an inherited map vs ~23s to build one). Imported maps merge, so knowledge accretes. inspect_atlas previews what a bundle would add (new / enriches / redundant per app, new vs existing skills) before you trust it — and flags a map's age (⚠ may be stale past 30 days), since maps are stamped with a save time

  • Self-extending — auto-builds a map when none exists; crawls deeper on demand for unmapped screens

  • Self-healing — when a saved path breaks (the app updated), it detects the failure, re-maps, diffs what changed, and routes the new way

Intent-based control — ask & act by name (packages/phone-use/src/task.ts, packages/phone-use/src/apps.ts)

  • ask("iOS Version") — read any value without knowing which screen (or even which app) it's on: searches every mapped app, resolves the one that has it, navigates, and reads

  • toggle_setting("Full-Screen Previews", on=true) — flip a switch by name, auto-finding its screen; idempotent desired-state

  • tap("Send") — press a control by name (exact or substring): the action verb for buttons like "Done", "Add", "New". Taps in place, scrolling to find it if needed. Complements toggle_setting (switches) so skills can do create/compose errands, not just read/navigate

  • set_field("Ada", field="First name") — type into a form field by its label/placeholder (the input verb for forms; search is for search bars). With tap this completes compose/create errands — open Contacts → tap Add → set First/Last name → tap Done is one deterministic skill that actually creates the contact

  • search("Maps", "coffee") — the input primitive: focus an app's search field and type, revealing it whether it's a large-title bar hidden above the list (Settings) or a collapsed bottom card (Maps). Completes the intent vocabulary: navigate (goto), read (ask), act (toggle_setting), input (search)

  • Whole-phone — any app by natural name ("Settings" → com.apple.Preferences), since openApp is the cross-app transition and a registry maps names to bundle ids

  • The atlas becomes a queryable, controllable knowledge base of the phone that gets faster and richer as it's explored — atlas_overview (or phone-use atlas) shows what's been learned: every mapped app (screens, transitions, freshness) and every saved skill, so an agent can check what it already knows before exploring

Phone Skills — replayable cross-app composites (packages/phone-use/src/skills.ts)

  • A skill is a named, ordered chain of the intent primitives with variable capture: any step names its result with as, later steps interpolate it as {{name}}. So a cross-app errand — read a value in app A, use it in app B — is one saved artifact

  • Recorded once, replayed forever with zero model calls — the deterministic composite layer on top of the primitives (the "agent runs your phone errands" payoff)

  • Resilient replay — each step retries on transient failure (cold-launch timing, mid-animation taps, focus races); safe because an op reports failure only when it didn't act, so a retry never double-acts

  • Checked, not fire-and-hope — a verify step asserts a postcondition ("Delivered" showed, the contact saved), so a skill confirms its own success; with retries it also polls, doubling as wait-for-condition

  • Learn by doing — record_skill → perform the task once → stop_recording: every primitive that succeeds is captured as a step, so the phone learns an errand by watching it, no hand-authoring

  • Generalize — parameterize_skill promotes a recorded literal to a {{param}} slot, so one recording of "open coffee in Maps" replays for any place (run_skill args={place:"sushi"})

  • Skills persist to ~/.phone-use/skills/<name>.json; run saved-by-name or ad-hoc. MCP: run_skill, save_skill, list_skills, record_skill, stop_recording, parameterize_skill

  • Verified: a skill that asks Settings for the model name, captures it, then searches Maps for it runs end-to-end — cross-app, with substitution

Live atlas (packages/phone-use/src/live.ts, packages/phone-use/src/vizbus.ts)

  • The viewer shows the phone streaming on one side and the app map drawing itself in real time on the other, as the agent crawls

Surfaces (packages/phone-use/src/mcp.ts, packages/phone-use/src/agent.ts)

  • MCP server — observe (delta-rendered), batch, verify, press/press_at, zoom_region, apps, map_app, goto, ask, toggle_setting, skills & atlas tools, … usable by Claude Code, Codex, or any MCP client

  • Built-in agent loop — Claude drives the phone directly (needs an API key); same harness rules as the MCP path (milestones, two-strike, reflection-on-demand)

Benchmark

The device layer has a regression eval (eval/): 33 probes on a test app that records what it actually received, so the tool never grades itself, plus a multi-agent tier. Its headline is the false-success rate (the tool said OK, the app says otherwise). phone-use was tuned against these probes, so its 0% there is a regression bar, not evidence of reliability in general. It measures single device actions, not whether an agent completes real tasks. bun run bench below is a smoke test of the high-level verbs.

bun run bench runs a suite of phone tasks, each a skill that ends in a verify text check on the final screen, from a clean state, timed, across the capability classes. Latest simulator run:

Phone Use benchmark — 6/6 passed, avg ~6.3s/task
  ✓ read-ios-version         read          3658ms   goto Settings→About, verify "26.1"
  ✓ deep-nav-accessibility   navigate      2860ms   goto Accessibility, verify "Vision"
  ✓ search-open-maps         search+act    8269ms   search Maps "coffee" → open, verify
  ✓ toggle-setting           toggle        3581ms   set a switch by name (idempotent)
  ✓ phone-wide-ask           ask-anywhere  5832ms   resolve which app holds a fact, read it
  ✓ create-contact           create       13506ms   Add → fill name → Done → verify

bench --json emits machine-readable results for CI (exit code nonzero on any failure). It is a repeatable smoke test on stock apps the tool was tuned on; its checks are text on the final screen (and the toggle task trusts its own result), so passing it does not show generalization. Building it surfaced (and fixed) real races the piecemeal tests missed.

bun run smoke runs the headline-feature regression guards in one command — the recorder over the real MCP wire, the shared-atlas inherit-and-navigate round-trip, and the full skill lifecycle (record → parameterize → replay → forget) — each asserting live on the simulator.

Docs

Product documentation lives with the site under apps/landing/app/docs/ and is served at https://phoneuse.dev/docs. Run bun run docs:dev and open http://localhost:3000/docs.

Technical write-ups are in docs/: start with docs/README.md. The novel capability is written up in docs/15-app-cartographer.md.

Verified end-to-end

Driven by Claude Code through the MCP server alone, in one ~33s session on the simulator:

"Tell me this iPhone's iOS version and model name (use ask), then turn on Full-Screen Previews (toggle_setting on=true)."

The agent called ask("iOS Version") → 26.1 and ask("Model Name") → iPhone 16e (both auto-resolved to Settings → About without being told the screen), then toggle_setting("Full-Screen Previews", on=true) → auto-found the switch and set it 0 → 1. No screen names, no bundle ids, no step-by-step navigation.

Every layer — driver, cursor, live viewer, map building, deterministic goto, self-extension, self-healing, whole-phone ask/toggle, and the live atlas — is verified on the iOS Simulator.

Status

  • iOS Simulator and Android (device or emulator): the main paths. Every layer above is verified on the iOS Simulator.

  • Real iPhone: works through iPhone Mirroring on macOS 15 or newer (phone-use connect). It reads the screen with text recognition, so it is rougher than the simulator; real-iphone.md lists what it can and cannot do. Driving a physical iPhone with the XCTest runner (Developer Mode + code signing) is not wired up yet.

  • Cloud phones: on a waitlist.

Phone Use is free, Apache-2.0, and comes as it is.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      C
      maintenance
      Enables agents and LLMs to interact with native iOS and Android applications and devices via structured accessibility snapshots or coordinate-based taps, supporting simulators, emulators, and real devices.
      39
      45,799 npm
      Apache 2.0
    • A
      license
      B
      quality
      B
      maintenance
      Enables AI agents to automate Android and iOS devices, including emulators and physical devices, by capturing screenshots, reading UI accessibility trees, performing touch and text input, and managing app lifecycle and logs.
      18
      4
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables agents and LLMs to automate, inspect, and scrape native iOS and Android apps across simulators, emulators, and real devices through a platform-agnostic interface using accessibility snapshots or screenshot-based coordinates. It provides device, app, screen, input, log, and crash management tools, including optional cloud device reservation.
      34
      Apache 2.0