iphone-use

Overview Schema Source Related Servers Discussions Admin

phone_elements

Read-only

Read the current iPhone screen as text instead of screenshots, then filter accessibility elements by query or selector and return row indexes plus a snapshot token for tapping.

Instructions

The current screen as text — use this instead of screenshots (an order of magnitude cheaper). Rows carry kind, label, identifier, value, rect and state, plus a snapshot token for phone_tap_element; indexes and snapshot tokens are valid for this read only. query keeps only rows containing that text (label, identifier, value, placeholder) and limit caps the rows. selector ANDs exact kind/identifier/name/label/value/enabled fields and case-sensitive label_contains; optional selector.index (integer0..199) selects one match after scope/query and before limit, retaining the original snapshot row index. Unknown earlier candidates or incomplete tree scope refuse the ordinal. name matches a non-empty identifier, otherwise the label; value_present checks whether the snapshot supplies a string value (including empty/truncated); false means omitted/null on non-secure rows, not a complete AX null; protected SecureTextField rows match neither boolean. It filters AX rows, not OCR. Queries preserve each row's index and the snapshot, so taps still work. Screen text shows at most 200 rows and 32 KiB of row text; an omission notice means a partial view, not absence or list completion. Narrow with query when needed. include_hidden=true expands off-screen/not-drawn rows instead of summarizing five labels; these rows are marked hidden and must be scrolled into view before acting. On the first read in a newly entered app a registry block lists saved flows for it: if one does the task, phone_flow_run it instead of tapping step by step. An alert block means a system alert is up: answer it with an alert step, taps on alerts do not act. ocr=auto adds text read from pixels when the tree has no interactive rows (games, canvases), ocr=force always; such rows have no element, tap their x/y with phone_tap. value_truncated marks an incomplete preview; use phone_read_value for the bounded full AX value. Requires phone_status drivable=true.

Input Schema

Name	Required	Description
`ocr`	No	Text read from a fresh screenshot (Vision OCR) for screens the tree cannot describe (games, canvases, video): `auto` adds it only when the tree has no interactive rows, `force` always, `off` (default) never.
`limit`	No	Return at most this many rows; the answer says how many were left out.
`query`	No	Keep only rows whose label, identifier, value or placeholder contains this text (case-insensitive). Rows keep their original index.
`instance`	No	Installed local instance name, such as i13 or default. Omit to use the configured daemon. No fallback on failure.
`selector`	No	Literal case-sensitive field filters combined with AND; preserves original snapshot indexes. Requires literal criteria; optional index0..199 chooses one match in scoped/query tree order before limit. Row index remains the full snapshot index; incomplete or unknown earlier matches refuse an unproven rank. Filters AX elements, not separate OCR text.
`include_hidden`	No	Show off-screen/not-drawn rows individually in the text, with their original indexes and hidden state, instead of a five-label summary. The 200-row / 32 KiB text budget still applies. Scroll them into view before acting. Default false.

Schema Changelog

Changes observed during successful MCP inspections.

Changed8 schema fields changedv0.27.34Nov 2, 2026

addedInput schema / $defs / ElementSelector / description

Added value: +"A literal filter over exposed snapshot metadata.\n\n```\nlet selector: core::element_selector::ElementSelector =\n    serde_json::from_str(r#\"{\"value_present\": true}\"#).unwrap();\nlet row = serde_json::from_str(r#\"{\"kind\":\"TextField\",\"value\":\"\"}\"#).unwrap();\nassert!(selector.validate().is_ok());\nassert!(selector.matches(&row));\n```"

addedInput schema / $defs / ElementSelector / properties / index

Added value: +{
+  "$ref": "#/$defs/MatchOrdinal",
+  "description": "Zero-based ordinal among matching AX rows, 0..199; not a snapshot row index.\nRequires other criteria and a proven tree-order rank."
+}

addedInput schema / $defs / ElementSelector / properties / name

Added value: +{
+  "description": "Exact name: non-empty identifier when present, otherwise label; case-sensitive.",
+  "type": [
+    "string",
+    "null"
+  ]
+}

changedInput schema / $defs / ElementSelector / properties / value / description

Previous value: -"Exact exposed accessibility value; known truncated previews never match.\nAn empty string matches a present empty value."New value: +"Exact exposed accessibility value; truncated, unavailable and secure values are unknown, never matches.\nAn empty string matches a present empty value."

addedInput schema / $defs / ElementSelector / properties / value_present

Added value: +{
+  "description": "Whether the snapshot supplies a string value, including empty or truncated\nstrings. False matches an omitted/null non-secure snapshot value, not an App's\nnull AX value. Protected SecureTextField rows match neither boolean.",
+  "type": [
+    "boolean",
+    "null"
+  ]
+}

addedInput schema / $defs / MatchOrdinal

Added value: +{
+  "description": "Zero-based ordinal among matches, never a returned snapshot row index.",
+  "maximum": 199,
+  "minimum": 0,
+  "type": "integer"
+}

addedInput schema / properties / limit / minimum
```
Added value: +0
```

changedInput schema / properties / selector / description

Previous value: -"Literal case-sensitive field filters combined with AND; preserves original snapshot indexes.\nRequires at least one field; filters AX elements, not separate OCR text."New value: +"Literal case-sensitive field filters combined with AND; preserves original snapshot indexes.\nRequires literal criteria; optional index0..199 chooses one match in scoped/query tree order before limit.\nRow index remains the full snapshot index; incomplete or unknown earlier matches refuse an unproven rank.\nFilters AX elements, not separate OCR text."

Changed2 schema fields changedv0.27.21Oct 31, 2026

addedInput schema / $defs

Added value: +{
+  "ElementSelector": {
+    "additionalProperties": false,
+    "properties": {
+      "enabled": {
+        "description": "Exact enabled state. A missing state never matches either boolean.",
+        "type": [
+          "boolean",
+          "null"
+        ]
+      },
+      "identifier": {
+        "description": "Exact accessibility identifier; case-sensitive.",
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "kind": {
+        "description": "Exact accessibility kind, e.g. Button (XCUIElementTypeButton also accepted).",
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "label": {
+        "description": "Exact label, preserving case, whitespace and newlines.",
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "label_contains": {
+        "description": "Case-sensitive substring of the label. Combined with other fields using AND.",
+        "type": [
+          "string",
+          "null"
+        ]
+      },
+      "value": {
+        "description": "Exact exposed accessibility value; known truncated previews never match.\nAn empty string matches a present empty value.",
+        "type": [
+          "string",
+          "null"
+        ]
+      }
+    },
+    "type": "object"
+  }
+}

addedInput schema / properties / selector

Added value: +{
+  "anyOf": [
+    {
+      "$ref": "#/$defs/ElementSelector"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "description": "Literal case-sensitive field filters combined with AND; preserves original snapshot indexes.\nRequires at least one field; filters AX elements, not separate OCR text."
+}

Changed1 schema field changedv0.27.3Oct 27, 2026

addedInput schema / properties / instance

Added value: +{
+  "description": "Installed local instance name, such as i13 or default. Omit to use the configured daemon. No fallback on failure.",
+  "type": "string"
+}

First observedv0.21.0Oct 24, 2026

TDQS

A4.2/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=true, so safety is covered. The description adds substantial context beyond that: snapshot tokens are valid for this read only, 200-row / 32 KiB text budgets, omission notices mean partial views not absence, hidden rows must be scrolled into view before acting, and alert blocks block taps. That is genuinely useful behavioral context, though the return format is described in dense, hard-to-parse prose rather than structured disclosure; a 3 reflects solid added value over the annotation baseline without being exemplary.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loading is excellent — the cheapest-vs-screenshot claim and the sibling reference come first — but the remainder is a single dense paragraph mixing return shape, filter semantics, budgeting, alerts, registry, OCR and the value_truncated handoff. Multi-topic content this important (alerts block taps, hidden rows need scrolling) is buried mid-paragraph where an agent scanning for call conditions can miss it.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 6-parameter read tool with no output schema, the description covers return row fields, size/row budgets, truncation semantics, hidden-row behavior, alert handling, registry discovery, and OCR fallbacks — everything needed to call and interpret it. Nothing material is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real semantic layering: it explains the evaluation order (selector.index applies after scope/query and before limit) and that the ordinal retains the original snapshot row index, plus that ordinal matching refuses when earlier candidates are unknown or scope is incomplete. It also clarifies that value_present covers empty/truncated strings and that SecureTextField rows match neither boolean.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a concrete verb+resource: 'The current screen as text', and immediately distinguishes itself from phone_screenshot ('use this instead of screenshots'). It also names the exact return shape (kind, label, identifier, value, rect, state, snapshot), so an agent knows precisely what it gets without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit routing to alternatives: phone_tap_element consumes the snapshot token, phone_read_value for full bounded values, phone_flow_run when the registry block shows a saved flow, phone_tap with x/y for OCR rows. It also states preconditions ('Requires phone_status drivable=true') and when to narrow ('Narrow with query when needed').

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

phone_elements

Instructions

Input Schema

Schema Changelog

TDQS

Other Tools