interact_with_ui
Perform UI interactions (tap, swipe, text input) on Android and iOS apps. Target elements by ID/text or coordinates. Automate mobile app testing with precise UI actions.
Instructions
Perform UI interactions like tap, swipe, or text input. Can target elements by ID/text or by coordinates.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"platform": "<platform>",
"action": "<action>"
},
"name": "interact_with_ui"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| platform | Yes | Target platform | |
| action | Yes | Type of interaction to perform | |
| element | No | Element ID, resource ID, or text to interact with | |
| x | No | X coordinate for coordinate-based interaction | |
| y | No | Y coordinate for coordinate-based interaction | |
| text | No | Text to input (for input_text action) | |
| direction | No | Swipe direction (for swipe action) | |
| durationMs | No | Duration in milliseconds (for long_press and swipe, default: 300) | |
| deviceId | No | Device ID or name (optional) |