Screenshot of the active window with a numbered box drawn on each interactable element (Set-of-Mark) — returns the annotated image plus a text legend (#<id> <Role> "<name>", or #<id> <Role> desc="<description>" for an element that has only a description). Pick an element visually, then click it with glass_click_element using its #id (same ids as glass_a11y_snapshot). Chips sit just outside each element so small icon buttons stay visible. The box is only as precise as the toolkit's accessibility geometry (it can drift ~10-20px), but the #id and the click are exact (click_element actuates via the native accessibility action when available, else clicks the element's center). Errors if no accessibility tree is available — use glass_screenshot then.

Capture the active window's accessibility tree (semantic elements: role, name, description, window-relative bounds) as compact text — deterministic, low-token element addressing alongside screenshots. Each line is #<id> <Role> "<name>" desc="<description>" (x,y wxh) [states]. desc carries a second label the platform exposes apart from the name, and appears only where one exists and differs from the name; it is display-only, since glass_wait_for_element and glass_scroll_to_element select on name, not description. Pass an #id to glass_click_element. Errors if the backend or app exposes no accessibility tree (e.g. a canvas/black-box app) — fall back to glass_screenshot then. Optional max_nodes: raise the element cap, or 0 to remove the element-count limit (default caps protect the token budget).

Save the current frame as a named visual baseline — a reference image glass_diff and glass_wait_for_region later compare against, so you can ask what changed without spending image tokens on a before-and-after pair. Captures the whole window at call time (not a saved region), so settle the UI first if something is still animating. Saving over an existing name replaces it silently; baselines live outside the app under a per-server directory and last until the server exits, surviving glass_stop. Use this plus glass_diff to detect change; use glass_screenshot when you actually need to look at the pixels.

Report which operations (input, multi-touch, clipboard, accessibility, window move/resize) can be performed right now on a backend, and any setup a blocked one needs — so you can check before acting instead of hitting an Unsupported error. Each operation reports a live status (supported, degraded — works now at reduced fidelity, note says what's lost; requires_setup — a setup step is missing, note says what; or unsupported — this backend never does it) plus the tools it gates, so a degraded or blocked operation names exactly which tool calls to expect trouble from. Pass backend to query a specific backend by name; omit for the active one. Static — no session required.

Click at window-relative coordinates. button: left|right|middle; count for multi-click. Optional modifiers held during the action, e.g. ["ctrl"] or ["ctrl","shift"] for multi/range-select.

Click an element by its #id from glass_a11y_snapshot (actuates via the platform's native accessibility action when the element exposes one — works even when it's occluded or scrolled off-screen — else falls back to a synthetic pointer click at the center of its bounds; the result's method field says which path ran, native_fallback says why when the pointer path was used, and actuated_id names the element actually clicked when a control's label is a separate element from the control itself). If the element actually renders in a popover owned by a different window than the active one (e.g. an open dropdown's option row), the click is automatically routed into that popover window and the previously-active window is restored afterward. Ids are only valid within the latest snapshot — re-run glass_a11y_snapshot if the UI changed. Optional return: "snapshot" settles the UI then folds a fresh a11y tree into the result (and refreshes the snapshot cache); "settle" waits for the UI to stop changing (text-only); omit or "none" for no observe (default).

Read the app's clipboard as text ("" if empty). Also the cheap text-extraction path: glass_do ctrl+a then ctrl+c, then read here (beats OCR for selectable text). Returns Unsupported where the backend can't provide clipboard access.

Write text to the app's clipboard so it can paste it. Returns Unsupported where the backend can't provide clipboard access.

Diff the current frame against a named baseline; returns change stats + bbox. Set include_image: true to also return the current frame cropped to the changed region (omitted when nothing changed).

Run an ordered sequence of input actions in ONE call (collapsing per-action round-trips), then optionally observe. actions is a list of {"action":"click|move|drag|scroll|type|key|settle", …same fields as the matching tool — except type's return observe, rejected here (use a settle action or then)}; settle waits for the screen to stop changing between steps. Optional then runs after all actions succeed: {settle?, diff?, screenshot?} (text-only unless screenshot/diff image). Fails fast: if an action errors it reports which index failed and how many ran. Use for KNOWN sequences (login, form-fill, menu→item); if you must see a result to choose the next action, don't batch that part.

Diagnose the glass environment and report per-check status + how to fix anything missing. Use this to self-diagnose a glass_start failure. Optional deep: also spin up and tear down the default backend's headless display to verify it starts. Returns report (the rendered text above) plus structured data: sections (each a {title, backend, checks: [{name, status, detail, remedy?, remedy_action?}]}, where backend is null for general checks that apply to every backend, and status is one of "ok"/"warn"/"fail"/"skip"; remedy and remedy_action are each omitted when absent, so a failing check may carry neither) and overall — the single verdict to branch on, since it already downgrades a non-default backend's failing check to a warning the way the rendered summary does.

Drag with a button held from (x1,y1) to (x2,y2) — window-relative coordinates, so 0,0 is the window's top-left, not the screen's. Presses at the start point, moves across in steps over duration_ms, and releases at the end; the button is left (button overrides) and optional modifiers are held throughout, e.g. ["ctrl"] or ["ctrl","shift"] for multi/range-select. Either endpoint outside the window is refused with an error giving the window size, so a drag never lands somewhere you did not aim. Use this for a single pointer — selecting text, moving an item, resizing a pane; glass_gesture is the multi-touch equivalent (2+ pointers, for pinch/rotate), and glass_click is the press-and-release-in-place case.

Perform a multi-touch gesture: 2–10 pointers, each a straight from→to segment in window-relative px, all down together at t=0 and up at duration_ms. Pinch = two pointers toward/apart; rotate = two on an arc; two-finger swipe = two parallel segments; a from==to pointer is held. Multi-touch isn't available on every backend — it returns a clear Unsupported error where the active backend can't do it.

Press a key chord like 'ctrl+s', 'Return', 'alt+F4'. One key with any number of modifiers, joined by '+': the last token is the key, every earlier one a modifier (ctrl, shift, alt, super — cmd, win and meta are accepted names for super, and all of them are case-insensitive). The key is a named key such as Return, Escape, Tab, Delete, an arrow or F1-F12, or a single printable ASCII character. An unrecognised modifier or key name is rejected with an error naming the token, so nothing is half-pressed; modifiers are released again when the chord completes. Use this for shortcuts and named keys — glass_type is for literal text and cannot express either.

List the app's top-level windows: id, title, class, geometry, and which is active. Returns a JSON array. Window ids are not stable across calls — re-list after windows open/close instead of caching ids.

Read captured stdout/stderr log lines with a resumable cursor. glass_start captures the app's output from launch; this returns what has accumulated and a cursor to pass back next time, so a loop reads each line once. Returns immediately with whatever is there, including nothing at all — it does not wait, so use glass_wait_for_log when you want to block until a line appears (starting up, finishing work). Filter server-side with stream and contains rather than reading everything and scanning it yourself. The buffer keeps the most recent lines and drops the oldest, so a chatty app can age out lines you never read; the lines are the app's own output and are returned marked as untrusted.

Move the pointer to window-relative coordinates.

Capture the app window (or an optional window-relative region) as a screenshot (lossless WebP image). A capture reaching off the display edge is clipped to the on-screen portion — the returned width/height are the actual captured size, so a frame smaller than the window/region means it was clipped; only a fully off-screen surface errors.

Scroll at window-relative coordinates by (dx,dy) wheel steps. Optional modifiers held during the action, e.g. ["ctrl"] or ["ctrl","shift"] for multi/range-select.

Scroll a container (any axis) until an accessibility element is on-screen, then return it (text-only, no image). Requires the element to be actually visible — not merely present in the a11y tree — so the returned id is usable with glass_click_element. Select by name (accessible-name substring) and/or role (e.g. "Button"); optional value_contains. direction: "up"/"down"/"left"/"right"; omit to infer it from the target's off-screen position (falls back to a vertical down→up sweep when the target isn't in the tree yet). It sweeps that way to the end, then reverses. Optional x,y aim the swipe at a specific container; by default it anchors on the target's own row/column so a container that isn't window-centered (e.g. a top toolbar) is still driven. step sets wheel notches per move (default 3). Returns {matched,elapsed_ms,element{id,role,name,bounds,states},scrolled{steps,reversed,direction}} — the id is usable with glass_click_element. Returns {matched:false} if it never becomes visible after sweeping both ends or timeout_ms (default 20000). Errors if the app exposes no accessibility tree.

Make a window active by id (from glass_list_windows). Subsequent screenshot/click/type/window ops target it; coordinates are relative to it.

Set an editable element's value — pick the element's #id from glass_a11y_snapshot. Where the platform can write the value directly this is instant and takes no keystrokes; where it has to be typed, glass taps the element, clears it and types, then reads the element back to confirm — up to three accessibility reads, since a field may commit a frame or two later. Errors if the element isn't editable, if it changed since the snapshot (re-snapshot), if the element does not hold the requested value afterwards, or if the app exposes no accessibility tree. That does-not-hold error names both what you asked for and what the element holds, and which one it is decides your next move: your text in another form means the element transformed it and writing again will not help; part of your text means a keystroke was dropped, so write again; what it held before means the write took no effect, and the error then closes with what this backend knows about that. A separate error says the text WAS typed but the write could not be confirmed — the read-back failed, or could not tell which element now holds it. Do NOT write again on that one: the keystrokes already went out, and re-snapshotting is how you see where they landed. Optional return: "snapshot" settles the UI then folds a fresh a11y tree into the result (and refreshes the snapshot cache); "settle" waits for the UI to stop changing (text-only); omit or "none" for no observe (default).

Build, launch, and locate a native GUI app; returns its window geometry. Choose a backend with the backend param (defaults to the host). The accessibility tools are enabled by default; pass a11y:false to skip the accessibility bus for canvas/pixel-only apps. Optional window_hint ({ title?, class? }) picks the right window when several appear, or locates one the launched process hands off to another process.

Stop the running app and end the session. The app is asked to close first, so it saves state and starts clean next time; one that will not close is terminated, which takes a moment longer. Ends everything session-scoped: captured logs and a11y element ids are gone afterwards, so read what you need first (saved baselines outlive it, until the server exits). There is no resume — only glass_start runs the app again, as a fresh session. Not needed between steps of a task; one session can be driven for as long as you need it, and errors if no session is running.

Type a string of text into the focused window. Does not focus anything itself — click the field first (glass_click_element, or glass_click), or the text goes wherever focus already was. Sent as individual keystrokes, not a paste, so per-key handlers, autocomplete and validation all run; a newline in text does not press Return, so send that as a separate glass_key. Prefer glass_set_value for a field the a11y tree exposes: it addresses the field directly and reports whether the value landed, where this types wherever the cursor already sits and cannot tell you what it hit. Optional return: "snapshot" settles the UI then folds a fresh a11y tree into the result (and refreshes the snapshot cache); "settle" waits for the UI to stop changing (text-only); omit or "none" for no observe (default).

Block until a UI element reaches a precise state, then return it as text (no image). Select by name (accessible-name substring) and/or role (e.g. "Button"); condition (default appears): appears|disappears|enabled|disabled|checked|unchecked|selected|unselected|expanded|collapsed|focused|visible|hidden; optional value_contains. Returns {matched,elapsed_ms,element{id,role,name,bounds,states}} — the id is usable with glass_click_element. On timeout returns {matched:false}. Waits through a just-launched app that has not published its accessibility tree yet, and errors if none appeared before the timeout. Collapses screenshot poll-loops into one call.

Block until a log line containing contains (optionally on a given stream) appears, then return it as text. By default only lines emitted after this call count; pass a cursor from glass_logs to catch a line emitted just before. Returns {matched,line{seq,stream,text},cursor,elapsed_ms}; on timeout {matched:false}. Resume reading from the returned cursor.

Block until a visual region changes (diverges from a reference) or matches (converges to a saved baseline), then return text metrics (no image unless include_image:true). until: "changes" (default) or "matches" (needs baseline); optional window-relative region; mode perceptual|exact with threshold/tolerance. Returns {matched,changed_pct,bbox,elapsed_ms}. Use "matches" to confirm the UI reached an approved design without spending vision tokens.

Wait until the window stops changing, then return the settled frame. Optional stability_region watches only that sub-rectangle for settling (ignore unrelated motion); optional region crops the returned frame. Set include_image: false for a text-only {settled,width,height} result with no image (region ignored) — cheap before a text glass_diff.

Focus/resize/move the window or read its geometry. op: focus|resize|move|geometry.