Capture the screen. Optionally pass x/y/width/height (screenshot coords) to zoom into a region for reading small text; coordinates you act on must still be full-screen screenshot coordinates.

Screen size (real and screenshot space), scale, and pointer position.

Where your keystrokes will go (keys_go_to: the window under your pointer when you have your own pointer) and which window the user has focused. Check this before typing.

Current mouse pointer position in screenshot coordinates.

Click at (x, y). count=2 for double-click, 3 for triple-click. modifiers e.g. ["ctrl"] or ["shift"] are held during the click.

Move the pointer to (x, y) without clicking (e.g. to reveal hover menus).

Press at start, move smoothly to end, release.

Press and hold a mouse button at the current pointer position.

Release a mouse button.

Scroll the wheel at (x, y). amount = number of wheel clicks.

Type text into the focused widget. Any Unicode works (Korean, emoji...). Newlines press Return. expect_window: substring of the focused window's title or class; if it doesn't match, nothing is typed.

Press a key or chord: "Return", "ctrl+c", "ctrl+shift+t", "alt+F4", "super", "Escape", "Page_Down". X keysym names are accepted. expect_window works as in type_text.

Hold key(s) down for seconds (max 10), e.g. for games or key repeat.

Wait for the UI (max 30s), then by default return a screenshot.

Top-level windows known to AT-SPI: app, title, active/visible, box.

List visible widgets as [id] role "name" @(x,y) with center coordinates in screenshot space. Filter by app/window name substring, or active_window_only=True. Ids are valid until the next ui_tree call. Apps only appear if accessibility is enabled (Chrome/Electron need --force-renderer-accessibility).

Activate a ui_tree element. auto = AT-SPI action if available, else a real mouse click at its center.

Replace the contents of an editable ui_tree element. Falls back to focusing it, select-all and typing when the widget isn't directly editable.