computer
Drive a real web browser with mouse and keyboard actions: click, type, scroll, screenshot, and press keys to complete tasks.
Instructions
Use a mouse and keyboard to interact with a web browser, and take screenshots. If you don't have a valid tab ID, use tabs_context first to get available tabs.
Whenever you intend to click on an element like an icon, you should consult a screenshot to determine the coordinates of the element before moving the cursor.
If you tried clicking on a program or link but it failed to load, even after waiting, try adjusting your click location so that the tip of the cursor visually falls on the element that you want to click.
Make sure to click any buttons, links, icons, etc with the cursor tip in the center of the element. Don't click boxes on their edges unless asked.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ref | No | Element reference ID from read_page or find tools (e.g., "ref_1", "ref_2"). Required for `scroll_to` action. Can be used as alternative to `coordinate` for click actions. | |
| text | No | The text to type (for `type` action) or the key(s) to press (for `key` action). For `key` action: Provide space-separated keys (e.g., "Backspace Backspace Delete"). Supports keyboard shortcuts using the platform's modifier key (use "cmd" on Mac, "ctrl" on Windows/Linux, e.g., "cmd+a" or "ctrl+a" for select all). | |
| tabId | Yes | Tab ID to execute the action on. Must be a tab in the current group. Use tabs_context first if you don't have a valid tab ID. | |
| action | Yes | The action to perform: * `left_click`: Click the left mouse button at the specified coordinates. * `right_click`: Click the right mouse button at the specified coordinates to open context menus. * `double_click`: Double-click the left mouse button at the specified coordinates. * `triple_click`: Triple-click the left mouse button at the specified coordinates. * `type`: Type a string of text. * `screenshot`: Take a screenshot of the screen. * `wait`: Wait for a specified number of seconds. * `scroll`: Scroll up, down, left, or right at the specified coordinates. * `key`: Press a specific keyboard key. * `left_click_drag`: Drag from start_coordinate to coordinate. * `zoom`: Take a screenshot of a specific region for closer inspection. * `scroll_to`: Scroll an element into view using its element reference ID from read_page or find tools. * `hover`: Move the mouse cursor to the specified coordinates or element without clicking. Useful for revealing tooltips, dropdown menus, or triggering hover states. | |
| region | No | (x0, y0, x1, y1): The rectangular region to capture for `zoom`. Coordinates define a rectangle from top-left (x0, y0) to bottom-right (x1, y1) in pixels from the viewport origin. Required for `zoom` action. Useful for inspecting small UI elements like icons, buttons, or text. | |
| repeat | No | Number of times to repeat the key sequence. Only applicable for `key` action. Must be a positive integer between 1 and 100. Default is 1. Useful for navigation tasks like pressing arrow keys multiple times. | |
| duration | No | The number of seconds to wait. Required for `wait`. Maximum 10 seconds. | |
| modifiers | No | Modifier keys for click actions. Supports: "ctrl", "shift", "alt", "cmd" (or "meta"), "win" (or "windows"). Can be combined with "+" (e.g., "ctrl+shift", "cmd+alt"). Optional. | |
| coordinate | No | (x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates. Required for `left_click`, `right_click`, `double_click`, `triple_click`, and `scroll`. For `left_click_drag`, this is the end position. | |
| scroll_amount | No | The number of scroll wheel ticks. Optional for `scroll`, defaults to 3. | |
| scroll_direction | No | The direction to scroll. Required for `scroll`. | |
| start_coordinate | No | (x, y): The starting coordinates for `left_click_drag`. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| provenance | No | ||
| interactionReceipt | No |