accessibility
Drive app windows via the OS a11y tree. Find elements by name/role, invoke, toggle, set values, query state for reliable automation across DPI/layout shifts.
Instructions
START HERE to see and drive a window: the OS accessibility tree — read element names, find by name/role, invoke, toggle, expand/collapse, set value, select an option, query state. Pick an action: read_tree, find, get_element, focused, invoke, focus, set_value, get_value, expand, collapse, toggle, select, state, list_children, wait_for, compile_ui, find_button, find_field, smart_click, smart_type, smart_read. Always preferred over computer.click(x,y) when the target has a name — more reliable across DPI, window resize, layout shifts. click, set-value and get-value are aliases of invoke, set_value and get_value. read_tree / find return one line per element — [Role] "name" @x,y w×h (SCREEN coordinates: click them with computer click space:"screen"), plus value= when there is one; page past "+N more" with offset. invoke, set_value, toggle and select change the app (pressing "Delete" deletes); a destructive target name (Send, Delete, Pay) needs the user's confirmation. For sparse/ambiguous UIs: compile_ui fuses a11y+OCR into one ranked map of elements with stable ids; find_button/find_field locate a target semantically and return an {element_id, snapshot_id} you pass straight to invoke/set_value (survives layout shifts, no coordinates).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | "replace" (default with a target): select all of the field, then type. "append": type at the cursor, keeping the content (default without a target). | |
| name | No | Element name to search for | |
| text | No | The text to type | |
| limit | No | How many elements to list (default 60) | |
| scope | No | Read scope: "focused" for focused element, "window" for active window, "screen" for full screen | |
| value | No | Value for set-value action | |
| action | Yes | Which sub-action to perform. See this tool's description for the enum of valid values. | |
| expect | No | Optional post-conditions to verify after this action (same assertion types as the verify tool: window_title_contains, app_running, element_exists, element_value_contains, clipboard_contains, file_exists, file_contains, ocr_contains, file_changed_since_start). If any FAIL the action returns a DEVIATION and you must adapt. State an OUTCOME you can observe (a window title, a rendered element/chip, a status) — NOT the raw text you typed. | |
| intent | No | What you want to do (submit/cancel/search/login/...) | |
| offset | No | Skip this many elements — page past "… +N more" (default 0) | |
| target | No | Element name/text to click (e.g., "Send", "Submit", "New Email"). Alias: "name". | |
| purpose | No | What the compile is for (Where the action restricts it: one of general | find_text | act.) | |
| timeout | No | Max time in ms (default 5000) | |
| max_cost | No | Hard ceiling on perception cost (default ocr_ok) | |
| processId | No | Optional: limit to a specific process | |
| timeoutMs | No | Default 5000 | |
| element_id | No | Target a compiled element from compile_ui (requires snapshot_id) | |
| intervalMs | No | Default 250 | |
| parentName | No | Alias for "name". Accessibility name of the parent element | |
| controlType | No | UI Automation control type. Either form works: "Button" or "ControlType.Button". An unrecognized type returns no results rather than silently searching unfiltered. | |
| maxChildren | No | Cap on children returned | |
| snapshot_id | No | The compile_ui snapshot the element_id came from (requires element_id) | |
| target_text | No | If set and absent from a11y, pull OCR to find it | |
| automationId | No | Automation ID |