Every open window with id, wm_class, title, geometry, pid and which one has focus. Start here: ids from this list are what type_text and press_keys target (ids change when a dialog is recreated -- a wm_class or title fragment does not). HOW TO DRIVE THIS DESKTOP, because the round trip is the expensive part and the actions are milliseconds: (1) ui_find then ui_press where the app has an accessibility tree -- it cannot miss; (2) find_text for Chrome, Electron and Qt, which expose almost nothing; (3) do_steps when you already know the next few actions, instead of one call each; (4) let the acting tool show you the result rather than following it with a screenshot -- they all do now. A screenshot of the whole screen is the last resort, not the first move.

Look at the screen, one window, or one rectangle. The image comes back in this reply -- there is nothing to Read afterwards. CROP, DO NOT SHRINK: window costs about 1300 tokens and a region strip about 160, against 1843 for the whole desktop, and all three stay legible, while scale below 1 makes small text unreadable for a saving a crop would have made anyway. Passing window AND region means a rectangle measured inside that window. annotate draws grid lines and window boxes labelled in SCREEN coordinates, so the number to pass to pointer_click can be read off the picture instead of estimated. Before reaching for this at all: ui_find and find_text answer "where is X" without an image, and every acting tool already shows you the result.

Look closer at a small area at FULL resolution -- never scaled, unlike screenshot, which fits everything to the model's 1568px ceiling. For a tiny glyph, a hairline border, an icon. Refuses more than half the desktop: zoom exists to spend tokens on FEW pixels.

Move the pointer to an absolute screen position. Exact: this goes to the compositor (org.gnome.Mutter.RemoteDesktop), not through ydotool, so there is no acceleration curve and no closed loop needed. Coordinates are the same ones list_windows and screen_map report.

Click at an absolute screen position. Reports whether it LANDED on anything -- the screen is compared before and after, so a click into dead space says so instead of looking exactly like one that worked -- and shows you the result without a separate screenshot. Also reports whether the keyboard moved as a result. PASS expect_window: the click is refused if something else is under that point, which is the difference between a missed click and a click in someone else's window. Needs no consent dialog, unlike xdotool.

Press at one point, travel, release at another. The travel is real intermediate motion, because a press-and-teleport is not a drag to most toolkits.

Wheel clicks at a point. dy positive scrolls down, dx positive scrolls right.

Where the pointer is. Answers from the compositor when the extension supports it, otherwise from the last position this server set and says which. Never guesses from X, whose answer is stale whenever the pointer is over a Wayland surface.

What a click at this point would hit. Use it before clicking somewhere you inferred from a screenshot. Reports both the compositor's own pick (which respects input shapes, so a click-through overlay is seen through) and every window whose rectangle covers the point.

Everything on screen with the coordinates to reach it: the desktop rectangle, every window top of the stack first with its centre point, where the pointer is, and every pressable widget of the focused application with the exact pixel to click it at. This is the one call that turns 'click the Save button' into a number without looking at an image. Every widget also carries a ref: N -- pass it straight to ui_press(ref) or pointer_click(ref); refs are valid until the next screen_map call, and the result's refs_generation says which call issued them.

Wait until the desktop reaches a state, instead of sleeping a guessed number of seconds. Conditions: window_exists, window_gone, window_focused, focus_changes; text_appears (OCR polls a window for a string -- a reply arriving, a build finishing); widget_exists (an AT-SPI widget matching text/role shows up in app); clipboard_changed (a copy landed); elapsed (just wait N seconds -- for a long install with nothing to poll, and the honest alternative to watching for a string you know will never appear). Returns as soon as it is true, or reports honestly that it timed out. A timeout over 300s is clamped, not refused.

Prove the desktop is in a state, with evidence -- the honest way to END a task. Each assertion comes back passed/failed with what was actually observed; a false assertion is a result, not an error. Give any of: window_exists, window_focused, text_present, widget_exists, clipboard_contains.

Where a visible piece of text is on screen, in coordinates you can click. Reads the pixels with OCR, so it works in Chrome, Electron and Qt apps, which expose almost nothing to ui_find. Try ui_find FIRST -- pressing a real widget cannot miss -- and come here when it returns nothing. About 1.5s for a window; cheaper and more exact than taking a picture and estimating. Blind to icon-only buttons: there is no text in them to read.

Wait until a window or rectangle CHANGES, then show it. For anything wait_for cannot express: a reply arriving, a spinner finishing, a download completing. Polls pixels here instead of making you take blind screenshots and look at each one, and returns as soon as it changes.

Run a short sequence of actions in ONE call and look once at the end. Every separate tool call costs a model round trip of several seconds while the action itself takes milliseconds, so a known sequence -- activate, click, type, press Return, see the result -- belongs here rather than in four calls. Steps run with their own look off; the picture is taken after the last one, or at the step that failed. Use single tools when the next action depends on what the last one revealed. The sequence is validated up front -- a call that cannot finish never starts.

Record the screen, or one window, to an h264 mp4. Use this instead of screenshot whenever the thing being judged MOVES -- an animation, a transition, a scroll, a stutter, a hover state. Stills cannot show motion and bursting them tops out near 5 fps. Goes under the xdg portal straight to org.gnome.Mutter.ScreenCast, so there is no share-your-screen consent dialog, and encodes on the iGPU. Read the result back by pulling frames out with ffmpeg.

Turn a video into ONE image you can actually look at: N frames, evenly spaced, stamped with frame number and timestamp, tiled into a contact sheet. This is the other half of screencast -- a model cannot decode an mp4, so a recording is useless until it becomes stills. Also measures per-frame change and reports duplicate frames, which detects a source repainting slower than the capture rate and catches stutter and frozen output that eyeballing misses. Use from_frame/to_frame to zoom into a fraction of a second once the overview shows where the interesting moment is. Works on any video, not just screencast output.

Focus and raise a window, then confirm focus actually landed there. Returns an error rather than a false success.

Move, resize, close, (un)minimize, (un)maximize, re-workspace or pin a window -- through the compositor, where these are ordinary calls. The result reports the window as it IS afterwards (new geometry, or gone), not just that the call was sent. close that leaves the window standing names the usual reason: an unsaved-changes dialog.

Start an application and confirm it actually arrived: the result carries the NEW window's dict (or, while the screen is locked, the new AT-SPI app) and names which mechanism confirmed. Every real task starts with an app that is not running yet; this is that step, inside the protocol instead of a shell command.

Applications currently on the AT-SPI bus. These names are what ui_tree and ui_find take.

Accessibility tree for one application: roles, names, screen bounds, and which nodes are actionable. Prefer ui_find unless you genuinely need the shape of the whole window.

Find widgets by visible text. THE way to locate something to act on: pressing a real widget through AT-SPI cannot miss and does not care where the window moved to. Paths returned here are valid only while the tree is unchanged -- find, then act.

Read the content of a text widget straight out of the accessibility tree. This is how you VERIFY that something landed, instead of trusting that a keystroke arrived.

PREFERRED way to enter text. Writes through AT-SPI EditableText, which needs no focus and no ydotool: it works on an unfocused window and even while the screen is locked, and it reads the widget back to prove the text landed. Use type_text only when a widget is not AT-SPI-editable.

Invoke a widget's own action through AT-SPI -- the preferred way to act on this desktop. Requires expect_name or expect_role, and refuses if the path no longer points at that widget, so a shifted tree cannot make you press the wrong thing.

Type into a named window. Focus is confirmed first, nothing is typed if it cannot be confirmed, and the widget is read back afterwards to check the right characters arrived. Characters go to the compositor as keysyms, so the keyboard layout cannot transpose them -- the German-QWERTZ hazard that made ydotool type z for y does not apply to this path. ui_set_text is still better where it works: it hands text to the widget and needs no focus at all.

Put text or a file's bytes on the clipboard, and PROVE it landed by reading it back. Goes through the gnome-shell extension so the compositor sets the clipboard itself -- mutter has a measured bug (S-018) where an external client's offer can serve wrong bytes to text requests, so wl-copy is only the fallback and says so when used.

What is on the clipboard. Text by default; types:true lists the offered mimetypes instead. An empty clipboard is a clean result, not an error, and a clipboard owner that never serves its offer is reported after a short deadline instead of hanging.

Send a key combination to a named window, e.g. ctrl+s. Chain several with do_steps rather than one call each. Focus is confirmed first. Ctrl+Alt+F1-F12 is refused: it switches virtual terminal and looks exactly like a frozen machine.

Hold ONE key down for a duration, then release it -- a real press and a separate release, not a tap. For shift-selection, held-key scrolling and games. Blocks for the whole duration; press_keys is the tool for combinations.

Whether each mechanism is usable right now, and what each one will actually do: extension state and which methods the RUNNING shell has (an edited extension does not load until the next login), absolute pointer control, window and AT-SPI counts, keyboard layout, and the XTEST trap. Call this first when something behaves oddly.

Read back the trail of acted tool calls -- every state-changing call is journaled with its arguments, outcome, hit/miss verdict and screenshot hash. Use it to reconstruct what already happened after context loss, or to review an unattended run. Reading tools are not in it.