Capture the Windows screen, a specific display, a specific window (even overlapped via PrintWindow), a screen/window region, or a foreground-plus-related-windows bundle. Returns the image directly plus physical-pixel coordinate metadata and a frame_id for accurate click/drag/scroll projection.

Execute a sequence of up to 50 desktop actions (click, move, drag, scroll, type, paste, key, clipboard, focus, launch, wait, wait_for, screenshot, state, or UI Automation actions like invoke/set_value/get_text) in a single round trip. Aborts immediately at <1ms latency if user E-Stop triggers.

Move to (x, y) and click. Prefer passing from_frame from desktop_screenshot so occlusion/window movement is guarded automatically.

Move the mouse cursor to absolute/frame-projected (x, y) or by relative delta (dx, dy) for hover tooltips or pointer-locked canvases.

Press left button and drag smoothly from (from_x, from_y) to (to_x, to_y), along a multi-point path, or continuing from previous drag (from_last / continue_from).

Scroll vertically or horizontally at (x, y) or current cursor position.

Type Unicode text cleanly using SendInput KEYEVENTF_UNICODE into the active foreground window (use into_hwnd to guard against focus stealing).

Press or hold a key or key chord (e.g. "enter", "ctrl+s", "alt+f4", "ctrl+shift+p"). Blocked OS-escape chords are rejected under whitelist mode.

Enumerate visible Windows application windows (and optionally minimized ones with include_minimized), or bring a window to foreground via focus_hwnd.

Inspect a window's UI Automation accessibility tree (with compact [s#.#] control refs for desktop_ui_action, snapshot delta diffing, pattern filtering, and optional focused/selected/document text observation).

Perform a UI Automation action on a control identified by ref (from desktop_ui) or by selector (hwnd, automation_id, name, control_type, class_name, index).

Read (get), write (set), or clear the Windows text clipboard.

Paste long or multi-line Unicode text reliably via clipboard + Ctrl+V, automatically saving & restoring previous clipboard contents (use require_restore: true to refuse if clipboard contains non-text data).

Launch a Windows application, executable path, Start Menu display name, URL, or Store app with Electron/Node-injection env vars stripped cleanly, and wait for its window.

URGENT: Immediately abort all running and queued desktop actions, kill hung workers if needed, confirm held keys/mouse buttons are released, and optionally latch the E-Stop.

Inspect physical E-Stop overlay state, hotkey config (Esc mode + combo), window whitelist rules, and backend health, OR lock/tighten the window whitelist (lock_whitelist / tighten_whitelist). AI cannot clear an E-Stop or disable/widen an active whitelist.