ui_act
Act on UI element refs to click, type, select, scroll, drag, or press keys in Windows apps without moving the mouse or taking user focus.
Instructions
Act on a ref from the latest ui_tree/ui_wait/ui_ocr without moving the mouse or the user's focus. click | type (replace text) | check | uncheck | select (value= inside a list/combo) | expand | collapse | scroll | right-click | double-click | hover | wheel (value 'down 3') | drag (value= or 'dx,dy') | press (value 'Enter', 'ctrl+a', 'Down Down Enter'). press works in the background on classic controls and on a visible text field of a browser or Electron window. focus, and press on other WPF/browser controls (except Enter on a button or link), take the user's foreground: only with allow_foreground after the user agreed; press hands the foreground back once the keys are in, focus keeps it. If an action fails, never fall back to moving the real mouse or bringing the window forward yourself; tell the user. Refs expire when a window is read again.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"ref": "<ref>",
"action": "<action>"
},
"name": "ui_act"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ref | Yes | e.g. e12, or t3 from ui_ocr (click, double-click, right-click, hover, wheel, drag) | |
| value | No | text for type, item for select, keys for press, target ref (e or t) or 'dx,dy' for drag, direction for wheel | |
| action | Yes | ||
| allow_foreground | No | permit taking the user's foreground window |