observe
Capture the frontmost window's text elements and screenshot so you can plan click actions on visible UI.
Instructions
Capture the frontmost window and return a screenshot plus a textual listing of its text elements. Use this once before planning a batch of actions. TWO SURFACES: the elements come in two labelled blocks, because a step searches exactly one of them. 'elements in focus window (N)' is the focus window — the default search space of every step, and all a plain target can reach; text in a background window is deliberately not listed and cannot be targeted. 'chrome (menu bar / system, M)' is the macOS menu bar, status items and any open menu or popover; a step reaches those only with target.scope='chrome' — including the step right after opening a menu, since the dropped-down menu is chrome too, not part of the window. The chrome block is printed even when it is empty (M=0), with the reason. The reply also contains: the frontmost app name; which window was read (window); every window that app has open (open_windows) — quote a piece of one of those titles as target.window in execute_plan to work in a window other than the frontmost; whether this window is already known to the interface graph (known_window); the screen size in both pixels (OCR/element coordinates) and logical points (click coordinates); which area was scanned for single-character labels (single_character_scan); and, when there is text on neither surface, how much of it there is (off_surface — counted, not listed, because no step can act on it). Each element reads 'text @ (cx,cy) [conf 0.97]' where (cx,cy) is the element center in pixels. Element text is what you target in execute_plan. A PNG screenshot of the whole display is attached for anything OCR cannot name (icons, images).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||