computer
Control a GNOME desktop by taking screenshots, moving the mouse, clicking, typing, and pressing keys to complete tasks visually.
Instructions
Mouse, keyboard and screen on the user's GNOME desktop.
Coordinates are pixels in the most recent screenshot (it is downscaled, e.g. 1280x720 for a 1920x1080 screen; the tool maps them back). A screenshot waits briefly for the screen to stop changing. Text outside ASCII is pasted via the clipboard, which is restored afterwards. Actions other than screenshot/zoom return a short confirmation, not an image: take a screenshot when you need to see the outcome.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ref | No | Element ref from read_screen/find; used instead of coordinate. left_click on a ref uses the element's accessibility action when it has one. | |
| text | No | type: the text (any Unicode). key: space-separated combos like 'ctrl+s' or 'Tab Return' | |
| mouse | No | With ref: click with the real pointer even if an accessibility action exists | |
| action | Yes | What to do | |
| region | No | zoom: [x0, y0, x1, y1] in screenshot pixels | |
| repeat | No | key: repeat count | |
| duration | No | wait: seconds | |
| modifiers | No | Held during a click/scroll/drag, e.g. 'ctrl' or 'ctrl+shift' | |
| coordinate | No | ||
| scroll_amount | No | Wheel steps | |
| scroll_direction | No | ||
| start_coordinate | No | left_click_drag start [x, y] |