Click / invoke an element. Get handle_id from find_elements() or get_tree().

A plain left single click invokes the element's accessibility action (foreground-independent, does not steal focus). If the element exposes no actionable pattern, it falls back to a real pointer click at the element's center, so custom-drawn panels and tab strips that only highlight on a11y Select still get activated.

button="right" opens context menus; clicks=2 is a real double-click (opens list rows / tree items whose a11y invoke only selects them). Both always use real pointer events at the element's center, so make sure the window is on top for those.

Click at absolute screen coordinates (top-left origin, in points).

button: "left" (default), "right" (context menus) or "middle". clicks: 1 (default) or 2 for a real double-click.

Fallback for when accessibility-based clicking can't reach a target — webview content (Tauri/Electron/CEF), custom-drawn UI, OpenGL/canvas, etc.

Delivery is system-global (kCGHIDEventTap on macOS) so the click actually reaches webviews — this was a regression in the first version, which used PID-targeted delivery that silently no-op'd on webview content while still returning ok. The cost is that the click affects whatever window is at (x, y), so make sure the target window is on top first.

window_id is advisory only — the call cannot verify delivery, just that the events were posted.

Drag-and-drop with real pointer events: press at the source, move to the target over duration seconds, release.

Give either screen coordinates (from_x/from_y, to_x/to_y, top-left origin in points) or element handles (from_handle/to_handle, from find_elements or get_tree); a handle is resolved to its on-screen center. You can mix: a handle on one end and coordinates on the other.

For drop targets that have no accessibility path — editors that only accept drops from another panel, list reordering, sliders. Delivery is system-global, so both ends must be visible and the window on top.

Find interactive elements by role and/or name. Returns element handles — pass the 'id' field to click/type_text/etc. Common roles: button, combo box, text, entry, check box, menu item. name is an exact match, not a substring ("Load Listfile…" with its ellipsis, "Export Folder" not "Export"); use get_tree to read names.

Return the accessibility tree for a window as a compact JSON string. Use list_windows() first to get a window_id.

Empty, nameless layout wrappers are pruned automatically, so the tree stays compact even on content-heavy pages while every named or interactive node is preserved. Very large pages are additionally capped to a byte budget so the result always fits the client: when that happens the root object carries "truncated": true and "dropped_nodes" (a count). To see the cut content, pass max_depth to cap recursion depth, or use find_elements() to target specific controls.

max_depth counts the nameless wrappers that pruning hides, and web content (WebView2, WKWebView, Chromium) sits under several of them: a cap below ~12 can stop above a page's content. A node whose children the cap skipped carries "truncated_by_depth": true, so an empty "children" there means "not walked", not "empty". Right after a Tauri/WebView2 window appears its web pane may not exist yet; wait_for_element on a control you expect is the reliable way to know the page has loaded.

Returned as a compact JSON string (not a nested object) so the client's result-size cap counts real content, not indentation whitespace; parse it with a JSON loader.

Get the current text value or state of an element.

Resolve semantic names to live element handle IDs for the current session.

Pass a map of {label: {role, name}} and get back {label: handle_id}. Store the result and use the handle IDs in run_sequence to skip repeated find_elements calls. Re-call after app restarts or window recreation.

Example: learn_layout("1234", { "bold_button": {"role": "push button", "name": "Bold"}, "editor": {"role": "document", "name": ""}, })

List all visible application windows on the desktop.

Fields per entry: id, name, pid, focused, minimized titles: top-level window titles where name is something else (process name on macOS, AT-SPI app name on Linux). bounds: [x, y, width, height] in global screen coords (top-left origin), or null when the platform can't report it. On a Linux Wayland session a native-Wayland app can't know its position, so bounds are null unless it runs under XWayland. Use this to translate screenshot pixels to click_at coordinates. accessible: false signals the platform a11y tree has nothing for this window — find_elements / get_tree will return empty. Skip those tools and use screenshot + click_at directly. Chromium-family windows (Electron/CEF) get an automatic wake poke first — their lazy a11y trees are enabled and re-probed before this is stamped false — so a false here means genuinely canvas-only: custom OpenGL/Vulkan surfaces, raw immediate-mode toolkits without an AccessKit adapter. (egui+accesskit surfaces as accessible=true.)

Inject a key press into the focused element or a specific window. key: 'Down', 'Up', 'Left', 'Right', 'Return', 'Escape', 'Tab', 'Home', 'End', 'PageUp', 'PageDown', 'BackSpace', or any single character. Useful for navigating combo box dropdowns, dismissing dialogs, etc.

Combinations: 'Cmd+A', 'Ctrl+Shift+G', 'Alt+F4', 'Cmd+Shift+G'. 'Cmd' is the platform's shortcut key (Command on macOS, Ctrl on Windows/Linux), so 'Cmd+A' selects all everywhere; 'Super'/'Win' is the literal Windows key. Combinations need window_id and use real input, so that window is brought to the front first. On Linux they need X11 or an XWayland app.

Pass window_id. Without it the key goes to whatever app is in front, which is often your own terminal, and still returns ok.

Execute a list of actions server-side in a single call — no round-trips.

Each step is a dict with an "action" key plus action-specific fields: {"action": "click", "handle": "..."} {"action": "click_at", "x": 100.0, "y": 200.0, "window_id": null} {"action": "type", "handle": "...", "text": "..."} {"action": "key", "key": "Return"} {"action": "select", "handle": "...", "value": "..."} {"action": "get_value", "handle": "..."} {"action": "get_bounds", "handle": "..."} → value [x, y, w, h] or null {"action": "drag", "from_handle": "...", "to_handle": "..."} (or from_x/from_y, to_x/to_y; or dx/dy from the start: drags a splitter a known distance) {"action": "sleep", "seconds": 0.15} {"action": "wait_for_element", "window_id": "...", "role": "...", "name": "...", "state": "...", "timeout": 5.0} {"action": "wait_for_window", "pattern": "...", "timeout": 5.0}

Returns a list of per-step results. Stops at first failure unless stop_on_error is False.

Capture a screenshot as PNG. Last resort — prefer accessibility tools.

Pass window_id to crop to a specific window, or omit for full screen. Pass output_path (e.g. '/tmp/shot.png') to write the PNG to disk and get back {ok, path, width, height} — strongly preferred for any real-window capture, since base64-inline screenshots routinely overflow the response token budget. Without output_path, returns {ok, data, width, height} with base64-encoded PNG bytes.

width/height are pixel dimensions; on retina displays a window with 720×450 logical points captures at 1440×900 px. Divide by the window's point size to recover the scale factor for coordinate math.

Errors with {ok: False, error: ...} if window_id is given but the window cannot be located — does NOT silently fall back to a full-desktop capture.

Caveat on Linux/KWin: per-window capture crops a full-screen shot to the target window's rectangle, because the KDE compositor won't let unprivileged clients capture individual windows. Any other window on top of the target will appear in the crop. Raise or close overlapping windows before capturing if isolation matters. Windows and macOS capture per-window at the compositor level and are unaffected.

Select an option in a combo box or list by its visible text.

Tries the a11y item list, then SetValue on the control, then a keystroke fallback (real click to focus, type the item text, Return). Each strategy is verified by reading the value back; a strategy that leaves the control showing something else counts as a failure, so ok=false means it really did not change. The keystroke fallback needs the window on top.

Start recording the screen (or a specific window) as video. Returns a handle to pass to stop_recording. Crops to the window if window_id is provided.

Stop recording and export.

output_path must end in .gif or .mp4. GIF is transcoded and downscaled to 900px wide at 12 fps, fine for a short clip in a README. MP4 keeps the full-resolution H.264 capture with no re-encode, so use it for anything longer than a minute or two (a 20-minute GIF is both huge and slow to build). Returns the path to the finished file.

Type text into a field. Uses native SetValue — foreground-independent. Falls back to key injection if the element doesn't support SetValue.

SetValue REPLACES the field's content. A window matched by title can be the user's own (single-instance apps such as Windows 11 Notepad reuse their open window), so type only into windows of a process you started.

paste=true also allows typing through the clipboard (Windows, macOS) when the other ways fail. The text is lent to the clipboard, marked out of clipboard history, and cleared afterwards if nothing replaced it; what the user had copied is lost, never restored. Password fields never paste.

Wait until a matching element exists and optionally has the given state. Useful after clicks that trigger async UI changes.

Wait until a window matching title_pattern appears. Returns null on timeout. Useful after triggering an action that opens a dialog.

title_pattern is a case-insensitive substring (not a regex), matched against the listed name and every top-level window title, so a window title works on macOS (listed by process name) and Linux too.