computer
Controls the desktop with mouse, keyboard, and screenshot actions for pixel-level GUI automation when named accessibility targets are unavailable.
Instructions
Direct mouse/keyboard/screenshot control. Pick an action: screenshot, screenshot_region, zoom, click, double_click, right_click, middle_click, triple_click, hover, move, move_relative, scroll, scroll_horizontal, drag, drag_path, mouse_down, mouse_up, type, key, key_press, key_down, key_up, wait. screenshot is the LAST RESORT for seeing the screen: first try accessibility read_tree/find/compile_ui (names + positions) or system copy_all_text (exact page text). Coordinates default to image-space pixels of the latest screenshot; pass space:"screen" to click a position reported by accessibility. Prefer accessibility for named targets; use computer only when you need pixel-level control.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"action": "<action>"
},
"name": "computer"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | Left edge X in image-space coordinates | |
| y | No | Top edge Y in image-space coordinates | |
| dx | No | X offset in image-space pixels | |
| dy | No | Y offset in image-space pixels | |
| key | No | Alias for "combo". Alias for combo — accepted for MCP/compound backward-compatibility. | |
| endX | No | End X in image-space | |
| endY | No | End Y in image-space | |
| path | No | Stepped drag path: array of {x,y} points (min 2). When given, startX/startY/endX/endY are ignored. Press at first point, release at last. | |
| shot | No | Optional, image-space pointer actions: the id of the screenshot the coordinates were read off (each screenshot reply names it, e.g. "s3"). Refused when it is not the latest screenshot. | |
| text | No | The text to type | |
| combo | No | Key/combo to press (e.g. "Return", "mod+s"). Space-separate for a sequence. | |
| count | No | 1=single, 2=double | |
| space | No | Coordinate space. Omit (default) → image-space coords from the latest screenshot (the tool scales them to the screen). Pass "screen" → the positions an accessibility tree / find reports (@x,y), passed as reported, unscaled: physical pixels on Windows and Linux, POINTS on macOS (a Retina screenshot's "real" size is twice the point size). | |
| width | No | Width in image-space pixels | |
| action | Yes | Which sub-action to perform. See this tool's description for the enum of valid values. Required: screenshot_region/zoom: x, y, width, height; click/double_click/right_click/middle_click/triple_click/hover/move: x, y; move_relative: dx, dy; scroll: direction; scroll_horizontal: x, y, direction; drag_path: path; type: text; key_down/key_up: combo; wait: seconds. | |
| amount | No | Wheel ticks (default 3) | |
| button | No | Mouse button to click: "left" (default) or "right" | |
| expect | No | Optional post-conditions to verify after this action (same assertion types as the verify tool: window_title_contains, app_running, element_exists, element_value_contains, clipboard_contains, file_exists, file_contains, ocr_contains, file_changed_since_start). If any FAIL the action returns a DEVIATION and you must adapt. State an OUTCOME you can observe (a window title, a rendered element/chip, a status) — NOT the raw text you typed. | |
| height | No | Height in image-space pixels | |
| startX | No | Start X in image-space | |
| startY | No | Start Y in image-space | |
| display | No | Monitor to capture (index from window list_displays). Default: the monitor clawdcursor last worked on, else the primary. | |
| seconds | No | Duration to wait (0.1 to 30) | |
| direction | No | Scroll direction |