Observe the current computer state: screen geometry, active window, cursor, OCR'd on-screen text and accessibility UI elements. Levels: 'minimal' (fast), 'normal' (screenshot+OCR+accessibility), 'full' (adds focused element details). This is the primary tool for understanding the screen before acting.

Take a screenshot of the whole virtual desktop, a specific monitor or a [left, top, right, bottom] region. Saves it to the persistence directory and returns the path.

Get the accessibility tree of the active (or a named) window as a nested structure of roles, names and bounding boxes. Uses Windows UI Automation or Linux AT-SPI; returns an explanatory error when no accessibility backend is available.

Get the title, application and process of the currently focused window.

Find text anywhere on screen (OCR + accessibility) with fuzzy matching. Returns matches with bounding boxes so you can click them. Tolerates case, arrows and small typos ('Continue' matches 'Continue →').

Find a UI element visually: by image template name (from the configured template directory) or by natural-language description via the optional vision model. Returns a bounding box.

Wait until something changes on screen. Expectations: 'any_change', 'window' (value = window title substring), 'text' (value = OCR text substring). Use after actions to await UI reactions instead of blind sleeps.

Click a UI element. Target can be: visible text ('Continue'), an element id from computer.observe ('element_17'), or coordinates ({'x': 500, 'y': 300}). The engine resolves it via accessibility first, then OCR, then vision - and verifies the result.

Double-click a UI element (same flexible target rules as computer.click).

Right-click a UI element (same flexible target rules as computer.click).

Move the mouse cursor to a target (text, element id or coordinates).

Drag from one target to another (text, element ids or coordinates).

Scroll the mouse wheel. Positive amount scrolls up, negative down. Optionally at specific coordinates (x, y).

Type text at the currently focused input. Handles Unicode and multiline text (clipboard fallback used automatically). The typed content is never logged.

Press a keyboard key ('enter', 'tab', 'escape', 'f5', ...).

Press a key combination. Pass a string like 'ctrl+s' or a list like ['ctrl', 'shift', 't'].

List all top-level OS windows with title, application, pid and state.

Focus a real OS window by (partial) title or application name, e.g. 'Chrome'. Works without any application-specific integration.

Minimize a window by title/application.

Maximize a window by title/application.

Restore a minimized/maximized window.

Close a window gracefully (WM_CLOSE). Requires confirm=true; modal confirmation dialogs remain visible for the agent to handle.

Read the clipboard text.

Write text to the clipboard (useful as a typing fallback).

Launch a GUI application ('chrome', 'code', a .desktop entry, or a full path). Subject to security.allowed_applications, rate limiting and security.confirmation_required (confirm=true by default).

Run a shell command (security.allowlist applies; see computer.backend_status). Requires confirm=true. Output is truncated and credential-redacted.

Report health/availability of every backend (input, screenshot, accessibility, window management, clipboard, OCR, vision) plus security settings and emergency-stop state.

EMERGENCY STOP: immediately halt all automation (checked before every action and attempt). Persists until reset_emergency_stop.

Clear a previously triggered emergency stop.