Read screen text
read_screen_textExtracts visible on-screen text with on-host OCR, including images, canvases, WebViews, and native dialogs, and returns each text run with tap coordinates.
Instructions
Read the visible text off the screen with on-host OCR (Apple Vision). Works without a Metro connection and sees text the layout tree cannot: images, canvases, WebViews, native dialogs. Returns one line per recognized text run as <text> @x,y — x,y is the centre in tap units and can be passed straight to tap {x,y}. Lines that also exist in the app's JS layout are marked [tree]: prefer tap {text} for those, since an OCR box is just text, not a control. Optional: search (substring filters, case-insensitive), languages (e.g. ["de","en"]; auto-detected by default), compare_tree (default true). macOS host only; the first call compiles a small Vision helper (~10–30s, cached).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| device | No | ||
| search | No | ||
| languages | No | ||
| compare_tree | No |