ghosthand
Automates Apple devices and apps: booting simulators, listing and targeting connected iPhones, interacting with installed apps (e.g. Settings) to change device state such as enabling dark mode, and capturing screenshots of the running screen.
Drives an iOS Simulator or a real iPhone the way a person would: reads the accessibility tree, takes screenshots, taps, swipes, types, presses Home, launches/terminates apps and opens URLs or deep links, optionally running an autonomous agent that completes natural-language tasks on the device.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@ghosthandopen Settings on the simulator and turn on dark mode"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
ghosthand
Natural-language automation for iOS: an agent (Claude) drives an iOS Simulator or a real iPhone the way a person would.
Runner/(Swift): an XCUITest app that runs on the device and serves a small HTTP API: accessibility tree, screenshots, taps, swipes, typing, buttons, app launching.ghosthand/(Python): drives simulators (simctl) and iPhones (devicectl), starts and talks to the runner, exposes everything as an MCP server, and includes an autonomous agent (Claude via the Anthropic API) and a web console.packages/ghosthand-client/: zero-dependency Python SDK to drive the console from pytest or CI.
Requirements
macOS with Xcode (tested with Xcode 27.1) and an iOS Simulator runtime (
xcodebuild -downloadPlatform iOS)uv (
brew install uv)
Related MCP server: ios-agent-driver
Getting started
uv sync
uv run ghosthand boot "iPhone 17" # boot a simulator
uv run ghosthand runner start # build (first time) and start the runner
uv run ghosthand tree # visible elements, numbered
uv run ghosthand tap -i 4 # tap element [4]
uv run ghosthand launch com.apple.Preferences
uv run ghosthand swipe up
uv run ghosthand type "hello"
uv run ghosthand screenshot screen.png
uv run ghosthand doctorDefault target: the only booted simulator, else the only reachable iPhone. Otherwise pick one with --device "iPhone 17" or GHOSTHAND_DEVICE.
The runner is rebuilt automatically when its Swift sources change. Its logs are in ~/.ghosthand/runners/.
On a real iPhone
One-time setup:
Xcode > Settings > Accounts: add your Apple ID. A free account works ("Personal Team").
iPhone: connect it over USB and accept "Trust This Computer". Afterwards, Wi-Fi works too.
Developer Mode: Settings > Privacy & Security > Developer Mode (requires a restart).
Auto-Lock: Settings > Display & Brightness > Auto-Lock > Never while automating. A locked iPhone cannot be driven; the agent stops with a clear message.
uv run ghosthand devices # the iPhone must be "reachable"
uv run ghosthand runner start -d "My iPhone" # builds, signs, installs and starts the runnerOn the very first launch, iOS refuses apps from an unknown developer: on the iPhone, go to Settings > General > VPN & Device Management > Apple Development: > Trust, then start again. The CLI prints these instructions when it detects the problem.
How it works:
the runner is signed with your team (set
GHOSTHAND_TEAMto force one) and bundle IDs unique to your account;with a free account, signing expires after 7 days: the runner is rebuilt automatically after 6 days;
the Mac reaches the runner through the CoreDevice tunnel that Xcode opens to the iPhone (USB or Wi-Fi). Its IPv6 address is resolved once at startup, because each mDNS lookup of
<udid>.coredevice.localtakes several seconds;screenshots are downscaled to point size and JPEG-encoded on the iPhone: about 30 KB instead of 1.7 MB.
With Claude Code (MCP)
uv run ghosthand mcp install # = claude mcp add --scope user ghosthand -- uv --directory <here> run ghosthand mcp serveRestart Claude Code, then ask for example: "On the simulator, open Settings and turn on dark mode." The runner starts on the first call. No API key is needed: Claude Code does the thinking.
Tool | Purpose |
| numbered element list + annotated screenshot (scaled to points) |
| tap by |
| a direction, or start and end points |
| type into the focused field |
| Home button |
| open / quit an app |
| URL or deep link |
| installed apps |
| list targets / get one ready (and start its runner) |
| hand a whole task to the built-in agent |
Each action returns the refreshed element list, saving a round trip.
An index always refers to the latest observation returned to the model.
Autonomous agent
export ANTHROPIC_API_KEY=sk-ant-... # or `ant auth login`
uv run ghosthand run "Turn on dark mode in Settings"
uv run ghosthand traces # recent runsEach step is one independent request to the model: the task, a text journal of previous steps and the current
screen (numbered elements + annotated screenshot). The model answers with exactly one action
(tap_element, tap_point, long_press_element, swipe, swipe_points, type_text, press_home,
launch_app, open_url, wait or finish) and a one-sentence reason.
Old screenshots are never resent: only the text journal grows, so the cost per step stays flat.
An action with no visible effect is reported to the model ("the screen did not change"), and so are errors.
Stops on
finish, the step budget (--max-steps, default 30), a model refusal, or 3 answers without an action.Model:
claude-opus-5-5atloweffort by default (--model,--effort, orGHOSTHAND_MODEL/GHOSTHAND_EFFORT). Server-side refusal fallbacks (fallbacks: "default") are enabled.
Reliable mode
uv run ghosthand run "Turn on dark mode in Settings" --mode reliablePlan: before acting, one request produces 2 to 6 checkpoints, each with the evidence expected on screen.
Verification: the agent claims a checkpoint with
checkpoint_done; an independent reviewer, which took no action and sees only the screen and the journal, confirms or rejects it.finish(success=true)is reviewed the same way: after two rejections the task fails instead of reporting a false success.Replanning:
replanwhen the situation no longer matches the plan.Slower and more expensive than
fast(one extra request per checkpoint), but a reported success is a seen success.
There is no risk guard: the prompts tell the model not to complete purchases, deletions or messages unless the task asks for them, and nothing else enforces it. Supervise runs on a real iPhone.
Target check (before every tap)
The model decides from a screen captured seconds earlier; animations, loading or scrolling can move or replace its target in the meantime. Right before tapping, the screen is read again:
an element tapped by index is found again by type, label and identifier: tapped where it is now if it moved, not tapped at all (and reported to the model) if it is gone;
a point tap aimed at a labeled element is cancelled if a different element is now under that point.
The same check applies to ios_tap in the MCP server.
Traces: ~/.ghosthand/traces/<id>/ holds run.json (task, mode, plan, status, summary, duration, tokens),
steps.jsonl (one line per step, with checks) and step-NNN.jpg (the screen the model saw).
From an MCP client: ios_run_task starts the agent in the background, ios_task_status follows it and
ios_task_cancel stops it. The API key must then be in the MCP server's environment
(claude mcp add … -e ANTHROPIC_API_KEY=…).
Web console
uv run ghosthand ui # http://localhost:8000, opens the browserLive screen of the simulator or iPhone: click to tap, hold for a long press, drag to swipe; Home, typing, opening an app or a URL; starting and stopping the runner.
Agent: run a task (fast or reliable mode, step budget) and follow the plan and steps live.
Traces: replay any run step by step (screen the agent saw, reason, outcome, checks, plan); ← → keys to navigate.
Elements: the numbered list exactly as the model sees it.
The console listens on 127.0.0.1 only and refuses requests from other hosts or other sites (DNS rebinding, CSRF),
since it controls your phone.
API (used by the page and the SDK): GET /api/devices, POST /api/devices/{udid}/runner,
GET /api/devices/{udid}/screen.jpg, GET /api/devices/{udid}/elements, POST /api/devices/{udid}/action,
GET|POST /api/tasks, GET /api/tasks/{id}, POST /api/tasks/{id}/cancel,
GET /api/traces, GET /api/traces/{id}, GET /api/traces/{id}/step-NNN.jpg.
Client SDK (tests, CI)
from ghosthand_client import GhosthandClient
def test_dark_mode():
result = GhosthandClient().run("Turn on dark mode in Settings", mode="reliable")
assert result.succeeded, result.summarySee packages/ghosthand-client/README.md.
Tests
uv run pytest # no device or API key needed: fake model and fake runnerRunner API
Every request must send Authorization: Bearer <token>. runner start creates the token and stores it in ~/.ghosthand/runners/. On the simulator the listener also accepts only connections from this Mac; an iPhone is reached through the CoreDevice tunnel, with the same token.
Route | Purpose |
| status, current app, whether the iPhone is locked |
| size in points, scale, orientation |
| full-resolution PNG, or a point-sized JPEG |
| raw accessibility tree of the foreground app (+ system alerts) |
| tap or long press, in points |
| swipe |
| type into the focused field |
|
|
| open an app |
| quit an app |
| state of each app |
| installed apps, used to find the foreground one (sent at startup) |
| open a URL or deep link |
| stop the runner |
Editing the Xcode project
Runner/GhosthandRunner.xcodeproj is generated from Runner/project.yml with XcodeGen: cd Runner && xcodegen generate.
License
MIT, see LICENSE.
Available Tools
14 toolsios_appsBRead-only
List installed apps: bundle ID and display name.
| Name | Required | Description | Default |
|---|---|---|---|
| device | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, so the safety profile is covered and the description only needs to add context. It does add the return shape (bundle ID + display name), but says nothing about whether apps come from a booted simulator vs a specific device, or what happens if no device is available.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One short sentence, front-loaded with the action and immediately followed by the payload. Nothing is wasted or buried.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read-only list tool with no output schema, the essentials are close to covered, but the undefined device parameter leaves a real gap: an agent cannot tell whether to omit it, pass a UDID, or pass a booted-simulator identifier.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single 'device' parameter has 0% schema description coverage and the description never mentions it — no semantics for what a device value looks like, whether it is required, or what the null default selects. The prose about bundle ID/display name describes output, not the input parameter, so it does not compensate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('List installed apps') and even names the returned fields (bundle ID, display name), which lets an agent distinguish it from ios_devices (devices, not apps) and ios_launch (launching, not listing). It does not explicitly call out those siblings, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: an agent can infer this is the discovery step before ios_launch/ios_terminate, but nothing states when to call it, whether a device must be booted first, or what to do instead for other listing needs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_bootA
Get a target ready: boot a simulator (name like "iPhone 17", or UDID) or pick an iPhone, then start the automation runner on it.
The first start compiles the runner (a minute or two). On an iPhone it must be unlocked, and the very first time the user must trust the developer certificate in Settings > General > VPN & Device Management.
| Name | Required | Description | Default |
|---|---|---|---|
| device | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well: it discloses the lengthy first compile, the iPhone unlock requirement, and the one-time developer certificate trust step. It does not cover idempotency or what happens if the device is already booted, but the prerequisites are unusually well surfaced.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core action and follows with relevant prerequisites. The caveats earn their place, and there is little wasted text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter setup tool with no output schema and no annotations, the description covers the key operational context an agent needs. It could mention return value or already-booted behavior, but nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for the single device parameter. It does so by explaining accepted formats: a simulator name like "iPhone 17" or a UDID, plus an iPhone selection option, though the exact accepted value shape remains slightly ambiguous.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a concrete verb and resource: boot a simulator or select an iPhone, then start the automation runner. It is distinguishable from siblings like ios_devices and ios_observe, though it does not name them explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied as a setup step before automation actions, and the description notes the first-start compile delay. However, it does not explicitly say when to use this versus ios_devices or whether it is required before every session.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_devicesARead-only
List simulators and real iPhones, and which ones are ready (booted / reachable).
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already tells the agent this is a safe read, so the description's only added value is naming the readiness states returned (booted / reachable). It does not clarify whether powered-off simulators are included, whether real devices are discovered over the network, or what ordering to expect.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with the verb and resource first and the readiness qualifier second. Every clause earns its place; nothing is padded or repeated from the title.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, read-only list tool with no output schema, stating what is enumerated and what readiness means is essentially sufficient. A sentence on discovery mechanics (e.g., whether it requires Xcode/simctl or scans the network) would close the remaining gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is no parameter semantics to document and the baseline of 4 applies. The description correctly introduces no filtering concepts that might imply hidden arguments.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('List') and concrete resources ('simulators and real iPhones'), plus the attribute surfaced (booted / reachable). It is clearly distinguishable from siblings like ios_apps, ios_boot, and ios_observe, which act on a device rather than enumerating devices.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: an agent can infer this is the discovery step before ios_boot/ios_observe, but the description never says 'call this first to obtain a device identifier' or name any alternative or exclusion. No prerequisites or conditions are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_launchA
Open an app by bundle ID (see ios_apps). relaunch=True restarts it from scratch.
| Name | Required | Description | Default |
|---|---|---|---|
| device | No | ||
| relaunch | No | ||
| bundle_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does disclose one real behavioral trait: relaunch=True restarts the app from scratch rather than bringing it to foreground. It is silent on whether the call blocks until the app is running, what happens if the app is not installed, and whether device defaults to a currently booted simulator.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, the core action is front-loaded, and the pointer to ios_apps is inlined rather than padded into a separate sentence. Nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple three-parameter launch tool with no output schema and no annotations, the description covers the essential action but leaves the device parameter, failure behavior, and post-launch state undocumented. It is adequate but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate: it explains bundle_id (app identifier, discoverable via ios_apps) and relaunch (fresh restart). The third parameter, device, gets no explanation at all, leaving a clear gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Open an app') plus the identifying mechanism ('by bundle ID'), which is enough to separate it from listing tools like ios_apps. It does not explicitly contrast with siblings such as ios_open_url or ios_boot, so it falls short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The parenthetical '(see ios_apps)' gives a concrete prerequisite for obtaining the bundle ID, and relaunch is framed as an option. However, nothing says when to prefer this over ios_open_url (deep links) or how it relates to ios_terminate, so usage is only implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_observeARead-only
See the screen: numbered list of visible elements, plus a screenshot.
Args: screenshot: Include a screenshot (scaled to points, so pixel = tap coordinate). annotate: Draw element indexes on the screenshot. device: Simulator name or UDID; defaults to the only booted one.
| Name | Required | Description | Default |
|---|---|---|---|
| device | No | ||
| annotate | No | ||
| screenshot | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With readOnlyHint=true already covering the safety profile, the description adds useful behavioral context: it explains the numbered output, screenshot inclusion, annotation behavior, and the important coordinate mapping that pixel equals tap coordinate. It does not cover failure behavior when no booted device exists or multiple devices are ambiguous, but it is substantially informative.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose, followed by compact parameter notes under "Args:". Every sentence adds operational value with no redundant filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a three-parameter observation tool with no output schema, the description explains what is returned, how the screenshot relates to tap coordinates, what annotation does, and how device selection defaults. Nothing essential for correct invocation is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must carry parameter meaning, and it does: screenshot controls screenshot inclusion, annotate controls index drawing, and device accepts a simulator name or UDID with a clear default. The screenshot explanation also adds the critical scaling/coordinate detail not present in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: "See the screen," and immediately specifies the return shape as a numbered element list plus screenshot. This clearly distinguishes observation from action-oriented siblings such as ios_tap, ios_swipe, and ios_type.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The observation purpose is implied by "See the screen," so an agent can infer this is the inspection step before interacting. However, the description does not explicitly say when to use it versus alternatives or mention prerequisites such as needing a booted simulator.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_open_urlC
Open a URL or deep link (https://…, maps://…, app schemes).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| device | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It doesn't say whether the target device must be booted, which app handles the URL, whether the call blocks until the page loads, or what happens on an unhandled scheme — all material for a device-side navigation action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with zero filler; the verb, resource, and accepted formats are all in the first clause.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no annotations and no output schema, the description should cover preconditions and effects, but it omits device targeting, prerequisite device state, and result behavior. For a two-parameter action tool this leaves significant gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It adds useful meaning for `url` via scheme examples, but the `device` parameter — which selects the target simulator/device and defaults to null — is never mentioned or explained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Names a specific verb (Open) and resource (URL or deep link) and gives concrete scheme examples (https://, maps://, app schemes). This distinguishes it reasonably from ios_launch, though the description never explicitly states the difference between opening a URL and launching an app.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No statement of when to prefer this over ios_launch or how it relates to other device-control siblings. The examples of URL schemes imply usage but never give conditions or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_pressC
Press a hardware button. "home" goes to the home screen.
| Name | Required | Description | Default |
|---|---|---|---|
| button | Yes | ||
| device | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It discloses the effect of the home button, but it does not explain device requirements, whether the device must be booted, or any side effects or response behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences with the core action front-loaded. Every sentence contributes directly to understanding what the tool does.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool with no annotations and no output schema, the description is still incomplete: it leaves the 'device' parameter undocumented and omits prerequisites such as whether the target device must already be booted.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains that button='home' goes to the home screen, but it never mentions the optional 'device' parameter or how target selection works.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Press a hardware button.' It also identifies the one supported button, 'home', and its effect. It is clear against screen taps, but it does not explicitly name sibling alternatives such as ios_tap.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no explicit guidance on when to use this tool versus ios_tap or other iOS interaction tools. The only usage context is the implicit fact that this presses a hardware button, with no exclusions or alternative-routing advice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_run_taskA
Hand a whole task to the built-in agent (Claude via the Anthropic API) instead of driving step by step.
It runs in the background; poll with ios_task_status. Needs ANTHROPIC_API_KEY (or ant auth login)
in the server's environment.
Args: task: What to do, in plain language, e.g. "Turn on dark mode in Settings". max_steps: Step budget before giving up. wait_seconds: Block up to this long for the result (0 returns immediately). mode: "fast" acts directly; "reliable" plans checkpoints and has each one verified (slower, surer). device: Simulator name or UDID.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | fast | |
| task | Yes | ||
| device | No | ||
| max_steps | No | ||
| wait_seconds | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden and does well: it discloses background execution, polling with ios_task_status, API-key prerequisites, step budget, blocking wait behavior, and the fast/reliable mode tradeoff. It does not mention device side effects or irreversibility, but the major operational traits are covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core distinction, then follows with essential operational notes and a compact Args block. Every section contributes useful information without unnecessary repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-parameter tool with no annotations and no output schema, the description covers invocation, prerequisites, background behavior, and all parameters. It leaves some gaps around what device state changes may occur and what the eventual result contains, but the polling reference covers result retrieval adequately.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must document the parameters, and it does: task meaning and example, max_steps as budget, wait_seconds including 0 behavior, mode semantics with both enum values, and device as simulator name or UDID. This adds substantial meaning beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action (hand a whole task to the built-in agent) and resource, and explicitly contrasts it with driving step by step. This lets an agent distinguish ios_run_task from the low-level tap/swipe/type siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear usage guidance: use this instead of step-by-step control, poll results with ios_task_status, and configure ANTHROPIC_API_KEY or ant auth login. The alternative tool and the background/polling workflow are both named explicitly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_swipeA
Swipe: either a direction from the screen center, or explicit points.
The direction is the finger's: "up" scrolls content down (reveals what is below). To go back in most apps, swipe from x=2 to x=250 at mid-height.
Args: direction: up, down, left or right. from_x: Start x in points (with from_y, to_x, to_y instead of direction). from_y: Start y in points. to_x: End x in points. to_y: End y in points. duration: Gesture duration in seconds; shorter is a faster fling. device: Simulator name or UDID.
| Name | Required | Description | Default |
|---|---|---|---|
| to_x | No | ||
| to_y | No | ||
| device | No | ||
| from_x | No | ||
| from_y | No | ||
| duration | No | ||
| direction | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations, so the description carries the full burden — and it discloses two non-obvious traits: direction follows the finger ("up" scrolls content down) and duration controls fling speed. It omits what happens if neither direction nor points are supplied, and says nothing about prerequisites such as a booted simulator or foregrounded app.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the two operating modes before the Args block, and the counterintuitive direction semantics are called out early. The Args block is tight and every line adds meaning; nothing is padded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter, zero-annotation, no-output-schema gesture tool, the description covers all parameters, both invocation modes, and the main gotcha. It stops short of describing device resolution behavior when device is null and of any failure semantics, which an agent would want before firing a gesture.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it mostly does: it enumerates the four direction values, gives units (points) for all four coordinates, flags that point mode is used 'instead of direction', and explains duration semantically. It still doesn't state that the two modes are mutually exclusive by rule, nor mention the 0.3s default or what happens when nothing is passed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb+resource ('Swipe') and immediately splits it into its two modes: a named direction from screen center, or explicit from/to points. That immediately separates it from ios_tap and ios_press, which take single points, though it never names those siblings explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides one concrete recipe ('to go back in most apps, swipe from x=2 to x=250 at mid-height'), which is genuine usage guidance. However, it never states when to choose swipe over ios_tap or ios_press, or when a gesture is inappropriate. Usage is implied rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_tapA
Tap an element by [index] from the latest observation, or a point (x, y) in points.
Args: index: Element index from the latest ios_observe or action result. x: Horizontal position in points (with y, instead of index). y: Vertical position in points. hold_seconds: Press duration for a long press (e.g. 1.0); 0 for a normal tap. device: Simulator name or UDID.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | ||
| y | No | ||
| index | No | ||
| device | No | ||
| hold_seconds | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations at all, the description carries the full behavioral burden. It does disclose the long-press behavior via hold_seconds and its 0-default, which is genuinely useful. But it says nothing about failure modes (stale index, out-of-bounds coordinates), whether the tap requires a booted device, what the action returns, or whether it blocks until the UI settles.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The intent is front-loaded in the first sentence, with parameters following in a scannable Args block. Every line earns its place; the only mild redundancy is restating the parameter names already in the schema keys.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a five-parameter UI-mutation tool with no annotations and no output schema, the description covers targeting semantics and long-press duration well enough to invoke correctly. What remains missing is precondition and error behavior, which matters for a device-control action but is a smaller gap than the parameter documentation it does provide.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, yet the Args block documents every one of the five parameters with units ('in points'), the default for hold_seconds ('0 for a normal tap'), the source of index, and the mutual exclusivity of index versus x/y. This fully compensates for the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence states a specific verb (tap) plus two precise targeting modes: an element index from the latest observation, or an (x, y) point in points. An agent can tell this apart from ios_swipe and ios_press by the verb. It stops short of explicitly contrasting itself with the sibling ios_press, but the resource and scope are unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implicit workflow guidance is present: the index parameter is described as coming from 'the latest ios_observe or action result', which tells the agent the observation must precede the tap. However, there is no explicit statement of when to choose index vs coordinates, when to prefer ios_press or ios_swipe instead, or what preconditions (booted simulator, fresh observation) must hold.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_task_cancelB
Stop an agent task after its current step.
| Name | Required | Description | Default |
|---|---|---|---|
| task_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It usefully discloses that cancellation happens after the current step, but it omits permissions, reversibility, idempotency, and what happens to task output or state.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single front-loaded sentence with no wasted words. It immediately states the operation and its key timing constraint.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a task-cancellation mutation with no annotations and no output schema, the definition is too thin. It communicates the core action but leaves important operational details undocumented, especially task_id sourcing and cancellation side effects.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter, task_id, has 0% schema description coverage and is not explained in the description. The phrase 'agent task' loosely connects it to a task identifier, but the description does not state where to obtain it or its expected format.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Stop an agent task.' It also clarifies timing with 'after its current step,' which distinguishes this from simply checking status or starting a task, though it does not explicitly name sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied: call this to stop an agent task. However, the description does not say when to use it versus checking ios_task_status, nor does it mention whether already-completed tasks can be canceled.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_task_statusARead-only
Status and latest steps of an agent task; without task_id, a list of all tasks.
| Name | Required | Description | Default |
|---|---|---|---|
| task_id | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The readOnlyHint annotation already establishes that this is a safe, non-mutating read. The description adds that the return payload includes 'latest steps' and that omitting task_id returns all tasks, which is useful since there is no output schema, but it says nothing about ordering, volume, or limits on the list-all mode.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single semicolon-joined sentence that front-loads the primary behavior and immediately qualifies the optional parameter. No filler, nothing redundant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-optional-parameter read tool with a readOnly annotation, the description covers purpose, both operating modes, and the shape of the result despite the absence of an output schema. The remaining gap is provenance of task_id and any bound on the listing mode.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% and task_id has no description in the schema, so the description must carry the meaning — and it does, by explaining the omitted/null default behavior (list all tasks). What it omits is where a task_id comes from or its format, preventing a top score.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Names a specific resource (an agent task) and states exactly what is returned: status plus latest steps. It also covers the dual retrieval mode, so an agent can tell it apart from the mutating siblings ios_run_task and ios_task_cancel, though it does not name them explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The clause 'without task_id, a list of all tasks' gives an explicit condition for choosing between the single-task and list modes, which is real usage guidance. However, it never states when to prefer this over ios_run_task (to check progress after starting) or ios_task_cancel, leaving sibling routing to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_terminateC
Quit an app.
| Name | Required | Description | Default |
|---|---|---|---|
| device | No | ||
| bundle_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are supplied, so the description carries the full behavioral burden. It doesn't say whether the app's unsaved state is discarded, whether the kill is forced or graceful, what happens if the app isn't running, or what errors can occur -- all critical for a destructive mutation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three words is not conciseness here, it is under-specification; nothing is front-loaded because there is essentially no content. Every needed detail was omitted rather than compressed.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With two undocumented parameters, no annotations, and no output schema, the description needed to carry everything and instead says almost nothing. An agent cannot reliably call this tool from the definition alone.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description mentions neither parameter. It doesn't clarify that bundle_id must match an installed app (obtainable via ios_apps) or what the optional device selector means in a multi-device context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
"Quit an app" gives a verb plus resource, so the basic intent is inferable, but it never says this is the inverse of ios_launch or that the target is an installed iOS app identified by bundle id. Compared with siblings like ios_launch and ios_apps, the distinction is only implied.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this versus ios_launch, ios_apps, or the task tools, and no preconditions (e.g., app must be running, device targeting). An agent must guess at the appropriate context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ios_typeA
Type text into the field that has keyboard focus (tap the field first). Use "\n" for Return.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| device | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It usefully discloses the focus prerequisite and the "\n" newline convention, but is silent on whether typed text appends or replaces existing content, what error occurs without focus, and what the call returns.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, front-loaded with the action and target, with the prerequisite and newline convention packed into compact parentheticals. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter typing tool with no output schema, the core interaction is adequately described. The unexplained `device` parameter and the absence of any failure/return behavior leave meaningful gaps given zero annotation coverage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must explain both parameters. It partially covers `text` via the newline note but says nothing about `device` — an agent cannot tell whether it selects a simulator, a physical device, or defaults to a single connected target.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (type) and target (the field with keyboard focus), which cleanly separates it from ios_tap, the tool the parenthetical tells you to call first. The distinction from siblings is implied rather than named outright, which keeps it just short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear precondition — the field must already have keyboard focus, so tap it first — plus the convention for sending Return. It does not address failure cases (e.g., what happens with no focus) or when to prefer ios_press or ios_open_url instead.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
14 tool updates
v0.1.0- First observed
ios_apps - First observed
ios_boot - First observed
ios_devices - First observed
ios_launch - First observed
ios_observe - First observed
ios_open_url - First observed
ios_press - First observed
ios_run_task - First observed
ios_swipe - First observed
ios_tap - First observed
ios_task_cancel - First observed
ios_task_status - First observed
ios_terminate - First observed
ios_type
TDQS
Scored across 14 tools
Each tool maps to a distinct action in the observation→interaction loop (observe, tap, swipe, type, press), and the agent-mode tools (ios_run_task / ios_task_status / ios_task_cancel) are clearly separated from manual driving. The only soft overlaps are ios_launch vs ios_open_url (an app scheme URL could effectively launch an app) and ios_press vs dedicated navigation, which an agent could occasionally misselect.
All 14 tools use a uniform ios_ prefix with snake_case, giving a highly predictable namespace. Most are verb or verb_noun (ios_boot, ios_tap, ios_open_url, ios_run_task), with a few noun-only list/query names (ios_devices, ios_apps, ios_task_status) that follow a conventional read-only pattern but are a minor deviation.
14 tools sits squarely in the well-scoped range and every tool maps to a needed primitive: discovery, boot, observation, four input gestures, app lifecycle, URL handling, plus an optional agent layer with status/cancel. Nothing looks redundant or padded for the scope of iOS device automation.
The core loop is fully covered end-to-end: list devices, boot, observe screen (elements + screenshot), interact (tap/swipe/type/press), and app lifecycle (apps, launch, terminate, open_url), plus an agent mode with status and cancellation. Minor gaps remain—no install/uninstall of apps, no orientation/rotation, no explicit wait/sleep or clipboard/keyboard-dismiss primitive—but agents can work around these with existing gestures.
Maintenance
Related MCP Connectors
Give an AI agent an iPhone to see, tap and type on over MCP. Free early access (Simulator).
Let your AI agent control real iPhones: screenshots, tap, swipe, type and open apps.
Cloud iPhone Simulators for AI agents: build the Xcode project, read the screen, tap by label, test.
Melaya is a remote MCP server. It gives an assistant hands on your own Android phone and browser: it reads the screen through the accessibility tree, then taps, types and navigates inside the apps and sites you allow-list, with no per-app API. It also builds, schedules and runs agent pipelines across 6k+ connected tools. OAuth 2.1, nothing to install.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceAn MCP server that lets AI agents control iOS and Android devices (tap, scroll, type, take screenshots, read UI trees, and run code). Works with multiple devices at the same time.113 npm53MIT
- AlicenseAqualityDmaintenanceAn MCP server that lets an AI agent drive the iOS Simulator in a loop, enabling tapping, typing, swiping, reading the screen via the accessibility tree, and verifying app state.14MIT
- AlicenseNot gradedqualityDmaintenanceComprehensive MCP server for iOS Simulator automation, enabling AI agents to control simulators via tap, swipe, type, accessibility inspection, app debugging, and device management.35 npm2MIT
- AlicenseNot gradedqualityDmaintenanceMCP server that drives the iOS Simulator via accessibility-tree snapshots with stable element refs, enabling LLMs to interactively find, tap, type, and inspect UI elements at sub-200ms latency.48 npmMIT