interact
Script UI interactions using semantic selectors (click, type, key, tab, hover, wheel, drag) and receive emitted intent messages and the updated accessibility tree. Optionally capture a screenshot preview.
Instructions
Drive a UI through scripted interactions (click, type, key, tab, hover, wheel, drag — by semantic selector, never coordinates). Returns the emitted intent messages and the access tree afterwards; set screenshot=true for a preview image.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"description": "<description>",
"steps": "<steps>"
},
"name": "interact"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| size | No | ||
| steps | Yes | An array of interaction steps, e.g. `[{"click":{"role":"button","name":"Add"}}]`. | |
| theme | No | ||
| screenshot | No | Attach a preview image of the UI after the steps. | |
| description | Yes | The UI description: a `fenestra/1` JSON object. |