screenshot
Capture the main display or a named window as a base64 JPEG with pixel dimensions, giving a visual fallback for locating and clicking UI elements when text or accessibility data is missing.
Instructions
Visual fallback after ax_read and, for missing rendered text, OCR. Capture the whole main display or a single window with window="" — and return a base64 JPEG plus a text note with its pixel dimensions. ALL coordinate-taking tools (mouse_move, *_click, scroll) expect coordinates in THIS image's pixel space — origin (0,0) top-left, x right, y down — so read target positions directly off the returned image; subsequent clicks are mapped through it automatically. PREFER window="" (substring of its title or app name) over the whole display whenever you are working inside one app: a full-display shot is downscaled to ~1280px wide, so small UI (list rows, sidebar items, buttons) becomes only a few pixels — too small to read or click accurately. A window capture is larger and sharper, returns a smaller image (fewer pixels → less context, faster turn), and clicks map into the window automatically. marks is ON by default: it boxes+numbers actionable elements (needs Accessibility) and lists each as [N] for compatibility with click_mark. Prefer a fresh ax_read + ax_activate for semantic controls. A magenta coordinate grid (for reading x/y) is shown automatically when marks is off and hidden when it is on; pass grid=true to force both, or marks=false for pure coordinate mode. If a target is still too small to click, use zoom_region to magnify part of this image.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| grid | No | Overlay a labeled coordinate grid (magenta rules + pixel labels) for reading off click coordinates. Omitted by default it follows the AX-first rule: OFF when `marks` is on (you click by number, not coordinates), ON when `marks` is off (coordinate mode). Pass grid=true to force it on alongside marks, or grid=false to suppress it. | |
| marks | No | Set-of-Mark: number every actionable UI element (buttons, links, fields) and list each as `[N] role "label"`, so you can activate it with click_mark(number=N) — the most reliable targeting, no coordinate guessing. Defaults ON (AX-first). Needs Accessibility permission. Covers native controls AND web content — real links/buttons on semantic pages, and on div-rendered pages (e.g. webmail) the list rows are numbered too (their click lands via a coordinate at the row center). Only canvas/game- style surfaces with no AX come back empty. Pass marks=false for pure coordinate mode. | |
| window | No | Capture only a single on-screen window instead of the whole display — a case-insensitive substring of the window title or app name (e.g. "Safari", "Settings"). Smaller, sharper image = less context and better click precision. Subsequent clicks map to this window automatically. |