Screenshot of the active window with a numbered box drawn on each interactable element (Set-of-Mark) — returns the annotated image plus a text legend (#<id> <Role> "<name>"). Pick an element visually, then click it with glass_click_element using its #id (same ids as glass_a11y_snapshot). Chips sit just outside each element so small icon buttons stay visible. The box is only as precise as the toolkit's accessibility geometry (it can drift ~10-20px), but the #id and the click are exact (click_element targets the element's center). Errors if no accessibility tree is available — use glass_screenshot then.

Capture the active window's accessibility tree (semantic elements: role, name, window-relative bounds) as compact text — deterministic, low-token element addressing alongside screenshots. Each line is #<id> <Role> "<name>" (x,y wxh) [states]; pass an #id to glass_click_element. Errors if the backend or app exposes no accessibility tree (e.g. a canvas/black-box app) — fall back to glass_screenshot then.

Save the current frame as a named visual baseline.

Report which operations (input, multi-touch, clipboard, accessibility, window move/resize) can be performed right now on a backend, and any setup a blocked one needs — so you can check before acting instead of hitting an Unsupported error. Each operation reports a live status (supported, degraded — works now at reduced fidelity, note says what's lost; requires_setup — a setup step is missing, note says what; or unsupported — this backend never does it) plus the tools it gates, so a degraded or blocked operation names exactly which tool calls to expect trouble from. Pass backend to query a specific backend by name; omit for the active one. Static — no session required.

Click at window-relative coordinates. button: left|right|middle; count for multi-click. Optional modifiers held during the action, e.g. ["ctrl"] or ["ctrl","shift"] for multi/range-select.

Click an element by its #id from glass_a11y_snapshot (clicks the center of its bounds, via the normal click path). If the element actually renders in a popover owned by a different window than the active one (e.g. an open dropdown's option row), the click is automatically routed into that popover window and the previously-active window is restored afterward. Ids are only valid within the latest snapshot — re-run glass_a11y_snapshot if the UI changed. Optional return: "snapshot" settles the UI then folds a fresh a11y tree into the result (and refreshes the snapshot cache); "settle" waits for the UI to stop changing (text-only); omit or "none" for no observe (default).

Read the app's clipboard as text ("" if empty). Also the cheap text-extraction path: glass_do ctrl+a then ctrl+c, then read here (beats OCR for selectable text). Returns Unsupported where the backend can't provide clipboard access.

Write text to the app's clipboard so it can paste it. Returns Unsupported where the backend can't provide clipboard access.

Diff the current frame against a named baseline; returns change stats + bbox. Set include_image: true to also return the current frame cropped to the changed region (omitted when nothing changed).

Run an ordered sequence of input actions in ONE call (collapsing per-action round-trips), then optionally observe. actions is a list of {"action":"click|move|drag|scroll|type|key|settle", …same fields as the matching tool}; settle waits for the screen to stop changing between steps. Optional then runs after all actions succeed: {settle?, diff?, screenshot?} (text-only unless screenshot/diff image). Fails fast: if an action errors it reports which index failed and how many ran. Use for KNOWN sequences (login, form-fill, menu→item); if you must see a result to choose the next action, don't batch that part.

Diagnose the glass environment and report per-check status + how to fix anything missing. Use this to self-diagnose a glass_start failure. Optional deep: also spin up and tear down the default backend's headless display to verify it starts.

Drag with a button held from (x1,y1) to (x2,y2) — window-relative coordinates. Optional modifiers held during the action, e.g. ["ctrl"] or ["ctrl","shift"] for multi/range-select.

Perform a multi-touch gesture: 2–10 pointers, each a straight from→to segment in window-relative px, all down together at t=0 and up at duration_ms. Pinch = two pointers toward/apart; rotate = two on an arc; two-finger swipe = two parallel segments; a from==to pointer is held. Multi-touch isn't available on every backend — it returns a clear Unsupported error where the active backend can't do it.

Press a key chord like 'ctrl+s', 'Return', 'alt+F4'.

List the app's top-level windows: id, title, class, geometry, and which is active. Returns a JSON array. Window ids are not stable across calls — re-list after windows open/close instead of caching ids.

Read captured stdout/stderr log lines with a resumable cursor.

Move the pointer to window-relative coordinates.

Capture the app window (or an optional window-relative region) as a screenshot (lossless WebP image). A capture reaching off the display edge is clipped to the on-screen portion — the returned width/height are the actual captured size, so a frame smaller than the window/region means it was clipped; only a fully off-screen surface errors.

Scroll at window-relative coordinates by (dx,dy) wheel steps. Optional modifiers held during the action, e.g. ["ctrl"] or ["ctrl","shift"] for multi/range-select.

Scroll a container (any axis) until an accessibility element is on-screen, then return it (text-only, no image). Requires the element to be actually visible — not merely present in the a11y tree — so the returned id is usable with glass_click_element. Select by name (accessible-name substring) and/or role (e.g. "Button"); optional value_contains. direction: "up"/"down"/"left"/"right"; omit to infer it from the target's off-screen position (falls back to a vertical down→up sweep when the target isn't in the tree yet). It sweeps that way to the end, then reverses. Optional x,y aim the swipe at a specific container; by default it anchors on the target's own row/column so a container that isn't window-centered (e.g. a top toolbar) is still driven. step sets wheel notches per move (default 3). Returns {matched,elapsed_ms,element{id,role,name,bounds,states},scrolled{steps,reversed,direction}} — the id is usable with glass_click_element. Returns {matched:false} if it never becomes visible after sweeping both ends or timeout_ms (default 20000). Errors if the app exposes no accessibility tree.

Make a window active by id (from glass_list_windows). Subsequent screenshot/click/type/window ops target it; coordinates are relative to it.

Set an editable element's value directly via accessibility (instant, no keystrokes) — pick the element's #id from glass_a11y_snapshot. Errors if the element isn't editable, if it changed since the snapshot (re-snapshot), or if the app exposes no accessibility tree. Optional return: "snapshot" settles the UI then folds a fresh a11y tree into the result (and refreshes the snapshot cache); "settle" waits for the UI to stop changing (text-only); omit or "none" for no observe (default).

Build, launch, and locate a native GUI app; returns its window geometry. Choose a backend with the backend param (defaults to the host); pass a11y to enable the accessibility tools. Optional window_hint ({ title?, class? }) picks the right window when several appear, or locates one the launched process hands off to another process.

Stop the running app and end the session.

Type a string of text into the focused window.

Block until a UI element reaches a precise state, then return it as text (no image). Select by name (accessible-name substring) and/or role (e.g. "Button"); condition (default appears): appears|disappears|enabled|disabled|checked|unchecked|selected|unselected|expanded|collapsed|focused|visible|hidden; optional value_contains. Returns {matched,elapsed_ms,element{id,role,name,bounds,states}} — the id is usable with glass_click_element. On timeout returns {matched:false}. Errors if the app exposes no accessibility tree. Collapses screenshot poll-loops into one call.

Block until a log line containing contains (optionally on a given stream) appears, then return it as text. By default only lines emitted after this call count; pass a cursor from glass_logs to catch a line emitted just before. Returns {matched,line{seq,stream,text},cursor,elapsed_ms}; on timeout {matched:false}. Resume reading from the returned cursor.

Block until a visual region changes (diverges from a reference) or matches (converges to a saved baseline), then return text metrics (no image unless include_image:true). until: "changes" (default) or "matches" (needs baseline); optional window-relative region; mode perceptual|exact with threshold/tolerance. Returns {matched,changed_pct,bbox,elapsed_ms}. Use "matches" to confirm the UI reached an approved design without spending vision tokens.

Wait until the window stops changing, then return the settled frame. Optional stability_region watches only that sub-rectangle for settling (ignore unrelated motion); optional region crops the returned frame. Set include_image: false for a text-only {settled,width,height} result with no image (region ignored) — cheap before a text glass_diff.

Focus/resize/move the window or read its geometry. op: focus|resize|move|geometry.