accessibility
Read and drive desktop windows through the OS UI tree to locate controls by name or role, then invoke, set values, toggle, and verify state without screen coordinates.
Instructions
START HERE to see and drive a window: the OS accessibility tree — read element names, find by name/role, invoke, toggle, expand/collapse, set value, select an option, query state. Pick an action: read_tree, find, get_element, focused, invoke, focus, set_value, get_value, expand, collapse, toggle, select, state, list_children, wait_for, compile_ui, find_button, find_field, smart_click, smart_type, smart_read. Always preferred over computer.click(x,y) when the target has a name — more reliable across DPI, window resize, layout shifts. read_tree / find return one line per element — [Role] "name" @x,y w×h (SCREEN coordinates: click them with computer click space:"screen"), plus value= when there is one; page past "+N more" with offset. invoke, set_value, toggle and select change the app (pressing "Delete" deletes); a destructive target name (Send, Delete, Pay) needs the user's confirmation. For sparse/ambiguous UIs: compile_ui fuses a11y+OCR into one ranked map of elements with stable ids; find_button/find_field locate a target semantically and return an {element_id, snapshot_id} you pass straight to invoke/set_value (survives layout shifts, no coordinates).
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"action": "<action>"
},
"name": "accessibility"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | "replace" (default with a target): select all of the field, then type. "append": type at the cursor, keeping the content (default without a target). | |
| name | No | Element name to search for | |
| text | No | The text to type | |
| limit | No | How many elements to list (default 60) | |
| scope | No | Read scope: "focused" for focused element, "window" for active window, "screen" for full screen | |
| value | No | Value for set-value action | |
| action | Yes | Which sub-action to perform. See this tool's description for the enum of valid values. Required: get_element/focus/get_value/expand/collapse/toggle/select/state/list_children: name; set_value: value; find_button: intent; find_field: purpose; smart_type: text. | |
| expect | No | Optional post-conditions to verify after this action (same assertion types as the verify tool: window_title_contains, app_running, element_exists, element_value_contains, clipboard_contains, file_exists, file_contains, ocr_contains, file_changed_since_start). If any FAIL the action returns a DEVIATION and you must adapt. State an OUTCOME you can observe (a window title, a rendered element/chip, a status) — NOT the raw text you typed. | |
| intent | No | What you want to do (submit/cancel/search/login/...) | |
| offset | No | Skip this many elements — page past "… +N more" (default 0) | |
| target | No | Element name/text to click (e.g., "Send", "Submit", "New Email"). Alias: "name". | |
| purpose | No | What the compile is for (Where the action restricts it: one of general | find_text | act.) | |
| timeout | No | Max time in ms (default 5000) | |
| max_cost | No | Hard ceiling on perception cost (default ocr_ok) | |
| processId | No | Optional: limit to a specific process | |
| timeoutMs | No | Default 5000 | |
| element_id | No | Target a compiled element from compile_ui (requires snapshot_id) | |
| intervalMs | No | Default 250 | |
| parentName | No | Alias for "name". Accessibility name of the parent element | |
| controlType | No | UI Automation control type. Either form works: "Button" or "ControlType.Button". An unrecognized type returns no results rather than silently searching unfiltered. | |
| maxChildren | No | Cap on children returned | |
| snapshot_id | No | The compile_ui snapshot the element_id came from (requires element_id) | |
| target_text | No | If set and absent from a11y, pull OCR to find it | |
| automationId | No | Automation ID |