Observe the screen
observeInspect desktop app and browser GUIs via accessibility tree to list interactive elements with refs and pixel boxes, capture screenshots or Set-of-Mark views, and expand containers for automation.
Instructions
Observe the GUI: interactive elements with refs (e7) and [x,y,w,h] boxes in image pixels. mode auto returns the tree, or a Set-of-Mark screenshot when the tree is thin; screenshot/som attach an image. root_ref expands a container; scope='screen' lists windows for cross-app work. Start every step with this.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| app | No | App to observe; default: the last observed window or the frontmost app | |
| mode | No | auto: tree, or Set-of-Mark screenshot when the tree is thin | auto |
| page | No | Set-of-Mark page (80 marks each) | |
| scope | No | screen only for cross-app work (deny-listed apps masked) | window |
| root_ref | No | Expand this container: list all its descendants | |
| window_id | No | Window from apps(action='windows') | |
| max_elements | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||