Interact With Node
interact_with_nodeInteract with desktop UI widgets by node ID or label to perform clicks, typing, value changes, and selection via silent AT-SPI actions, falling back to coordinate input and returning UI tree deltas.
Instructions
Interacts with an interactive UI widget by 1D node ID (e.g. 'b1', 'e1') or label.
Executes Level 1 AT-SPI actions silently (without moving the cursor), falling back to
Level 2 coordinate/keyboard input only when AT-SPI is unsupported, and returns the
post-action UI tree delta (modified/appeared widgets). Optional role narrows name matches
like Playwright's getByRole.
Supports actions: click, type, set_value, clear, select_all, select_range, copy,
paste, cut, drag_select, double_click, right_click, hover.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "interact_with_node"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| pid | No | ||
| role | No | Optional accessible role filter (e.g. 'push button', 'entry'). | |
| text | No | ||
| value | No | ||
| action | No | click | |
| target | No | Primary target: compact element ID (e.g. 'b1', 'e1') or widget label. | |
| node_id | No | Deprecated alias for 'target'; prefer passing 'target'. | |
| end_offset | No | ||
| start_offset | No | ||
| settle_timeout_ms | No |