observe
Capture the frontmost window's text elements and screenshot so you can plan click actions on visible UI.
Instructions
Capture the frontmost window and return a screenshot plus a textual listing of its text elements. Use this once before planning a batch of actions. TWO SURFACES: the elements come in two labelled blocks, because a step searches exactly one of them. 'elements in focus window (N)' is the focus window — the default search space of every step, and all a plain target can reach; text in a background window is deliberately not listed and cannot be targeted. 'chrome (menu bar / system, M)' is the macOS menu bar, status items and any open menu or popover; a step reaches those only with target.scope='chrome' — including the step right after opening a menu, since the dropped-down menu is chrome too, not part of the window. The chrome block is printed even when it is empty (M=0), with the reason. The reply also contains: the frontmost app name; which window was read (window); every window that app has open (open_windows) — quote a piece of one of those titles as target.window in execute_plan to work in a window other than the frontmost; whether this window is already known to the interface graph (known_window); the screen size in both pixels (OCR/element coordinates) and logical points (click coordinates); which area was scanned for single-character labels (single_character_scan); and, when there is text on neither surface, how much of it there is (off_surface — counted, not listed, because no step can act on it). Each element reads 'text @ (cx,cy) [conf 0.97]' where (cx,cy) is the element center in pixels. Element text is what you target in execute_plan. A PNG screenshot of the whole display is attached for anything OCR cannot name (icons, images).
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "observe"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||