ui
Read a window's accessibility tree to find UI controls by name and get their click points, enabling interaction without screenshots or pixel guessing.
Instructions
Read a window's accessibility tree (AT-SPI) and return its elements
with GLOBAL click points, so you can target a control by NAME with no
screenshot and no pixel guessing. window is an address from desktop
(default: the focused window). name filters to elements whose
accessible name contains it (case-insensitive); actionable (default)
keeps only interactive roles (buttons, entries, menu items, ...).
Returns [{role, name, x, y, clickable}] where x,y is the click point:
focus the window, then click it with pointer (the window must be
visible to receive the click), or do both in one call with click_ui.
Controls that carry a CURRENT VALUE also
report it: value (text typed into an entry, or a slider/spinner
number), percent for a slider's position, checked for a box or
toggle. Password fields never report contents, and many dropdowns
expose no value at all, so read the screen with screenshot when a
rendered value matters. Not every app exposes a tree (terminals, and
Electron/Chrome without --force-renderer-accessibility, expose little
or nothing); when it does not, fall back to screenshot + zoom.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | ||
| window | No | ||
| actionable | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |