esp32-iphone-agent
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
Drives a real iPhone through an ESP32 Agent board. Loop: screenshot, decide, one action, screenshot again.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| screenshotA | The iPhone's screen now (after the last action has settled). Coordinates for the other tools are pixels of this image. |
| tapB | Tap at x, y (screenshot pixels). |
| longpressC | Press and hold at x, y. |
| swipeA | Swipe from x1, y1 to x2, y2 (rests at the end, so lists stop there). |
| type_textA | Type text into the focused field. ASCII only: Chinese is typed as pinyin (then commit a candidate with press_key space or a tap). |
| press_keyB | Press a key or combo: enter, esc, tab, backspace, space, up, down, left, right, cmd+a, shift+tab ... |
| press_buttonA | Press a hardware button. Only HOME (go to the Home Screen). |
| device_infoC | Board status: mirroring (ap), video frozen (fz), app busy (busy), USB, BLE link. |
| phone_audioA | What the iPhone played in the last 30 s (a video, a call, an app's voice) as words, or why it can't be heard (listening off, the app not sharing, no Wi-Fi). The words can be partial or misheard. |
| historyA | What was done on the phone: action list (the newest runs) or show (one run's steps, in 0-1000 screen units, each with the screenshot it was done on). A run ends after 10 minutes idle. |
| intent_listB | Saved intents: id, name, prompt, number of hints and steps. |
| intent_showB | One saved intent: prompt, hints and its recorded steps (0-1000 screen units) with their reference screens. |
| intent_saveB | Save what worked as an intent: a short prompt, up to 6 hints, and the replayable steps of a run (from history). Same name replaces. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 13 tools
Most tools are clearly distinct: screenshot, tap, longpress, swipe, type_text, and the intent_* trio each target a unique action. The main risk is press_key vs press_button, which read similarly as 'press X', though the descriptions explicitly separate keyboard keys from the hardware HOME button. history vs intent_show is also close but differentiated by live-run vs saved-intent scope.
Conventions are mixed: some tools are bare verbs (tap, swipe), some are verb_noun snake_case (type_text, press_key, press_button), some are bare nouns (screenshot, history, device_info), and the intent group uses noun-first (intent_list, intent_show, intent_save) — the opposite order from press_*/type_*. Still readable, but the inconsistent verb/noun ordering prevents a strong pattern score.
13 tools is well within the ideal 3-15 range and each one earns its place: input primitives, observation (screenshot, phone_audio, device_info), audit (history), and intent persistence. No redundant or filler tools are present.
The surface covers the full act-observe loop: screenshot, gestures, text/keyed input, device status, audio feedback, history, and intent save/replay. Minor gaps exist — no intent_delete (only overwrite-by-name), no direct app-launch/dismiss affordance beyond the HOME button, and clipboard/paste is absent — but agents can work around these via UI automation.