Launch a GUI app in its OWN new Xephyr display/window and return its instance id.

command is any shell command ('chromium', 'xterm', 'gimp', 'inkscape file.svg', …). Chromium automatically gets X11 flags, a per-instance profile, downloads routed to files//, and a CDP debug port (needed by upload_file). Other GTK apps get D-Bus isolation so they render on this display, not the host.

Returns {"ok", "instance", "display", "name", "files_dir"} — pass that instance id to every control tool. Fails (isError) if Xephyr can't start on the host display, the command doesn't exist, or the app exits immediately. Each instance has its own cursor, so separate agents can drive separate instances in parallel; never share one instance between agents. The window appears on GLOVEBOX_HOST_DISPLAY (default :0). Give the app a moment to draw (settle_ms/wait_ms) before the first screenshot.

List all instances (running app windows): id, display, name, command, up.

Returns {"ok", "count", "instances": [{instance, display, name, command, up}, …]}. Probes displays :1–:12 for sandboxes started outside this server (e.g. start-display.sh), so it also finds instances this process didn't launch (their name/command show as unknown). Use it to discover what you can drive and to verify an instance is up before acting.

Close an instance: kill its app + its Xephyr display, and verify it actually went down.

Returns {"ok", "instance", "detail"} only after the display stops responding. Fails (isError) if the instance isn't running, if it survives SIGTERM, or for instance 1 — the main start-display.sh sandbox is protected (stop it from the host with pkill Xephyr).

Screen size of an instance + the active vision backend and server version.

Returns {"ok", "instance", "width", "height", "vision", "version"}. Call it once before coordinate math — coordinates are absolute pixels, top-left origin, within THIS instance's display. "vision" tells you how to ground: 'basic'/'local' → parse_screen + click_element; 'none' → screenshot + reason about pixels yourself. Fails (isError) if the instance is down.

Screenshot of an instance's window (PNG image, full display).

The ONLY way to see the screen — always look before you act. For an element list with coordinates use parse_screen (basic/local vision). Prefer observe='screenshot' on action tools to act and see the result in one call. Fails (isError) if the instance is down.

Detect on-screen elements and return them with ids + pixel-centers (vision grounding).

Returns {"ok", "vision", "screen": [W,H], "annotated", "count", "truncated", "elements": [{id, type, label, interactive, center: [x,y]}, …]} — then click_element(id). Also saves a numbered Set-of-Mark image to /tmp/glovebox_annotated_.png. Backend = GLOVEBOX_VISION: 'local' (OmniParser, text + icons, ~2 s on GPU after a ~6 s first-call model load), 'basic' (tesseract, text only), 'none' (returns a note telling you to screenshot + reason about pixels yourself — this is normal, not an error). Elements are capped at 300 per call ("truncated": true when more were detected). Ids are only valid until the screen changes — re-parse after navigation.

Click at absolute pixel coords (top-left origin) in an instance.

Returns {"ok", "action", "instance", "detail"}. button: 1=left, 2=middle, 3=right. observe='screenshot'|'parse' additionally returns the resulting screen state in the SAME call (saves a round-trip); settle_ms (max 5000) waits for the UI to update first — use 400–1500 after anything that navigates or submits. Fails (isError) if the instance is down.

Drag (mouse-down → move → up) from (x1,y1) to (x2,y2) — for drawing, selecting, sliders.

Returns {"ok", "action", "instance", "detail"}. observe/settle_ms: see click(). Fails (isError) if the instance is down.

Double-click at absolute pixel coords in an instance (e.g. open a file icon, select a word).

Returns {"ok", "action", "instance", "detail"}. observe/settle_ms: see click(). Fails (isError) if the instance is down.

Click an element by id from that instance's most recent parse_screen (no coordinate guessing).

Returns {"ok", "action", "instance", "detail", "element", "center"}. Fails (isError) with the fix named if the id is unknown — run parse_screen(instance) first. Ids go stale when the screen changes: after navigation, re-parse before clicking. observe/settle_ms: see click().

Type into the focused field — unicode-safe. Click the field first to focus it.

Returns {"ok", "action", "instance", "detail", "method": "xdotool"|"clipboard"}. Pure ASCII is typed via xdotool; text with any non-ASCII char (č/š/ž …) is inserted via the clipboard + ctrl+v, because xdotool's synthetic unicode keystrokes get silently dropped by some toolkits (e.g. Inkscape's GTK canvas) even though they work in browsers. Fails (isError) rather than silently losing characters if xclip is missing for non-ASCII text. observe/settle_ms: see click().

Attach a local file to a page's via Chrome DevTools Protocol — use this for ALL browser uploads (logos, images, docs).

The sandbox's native GTK file picker is invisible to us AND hangs the renderer, so never click an upload button expecting a dialog — call this instead. Works on Chromium started by launch_app / start-display.sh (they open a per-instance --remote-debugging-port, 9222+N). selector targets the file input on the MAIN page (default = first file input). If the site only inserts the after you click its 'upload' control, click that first (it won't open a dialog we can see, but it wires up the input), then call this. Returns {"ok", "action", "instance", "detail"}; fails (isError) with the reason if the file is missing, no CDP target answers, or the selector matches nothing. NATIVE (non-browser) apps: use open_file() or drive the app's own Open dialog — it IS visible here. observe/settle_ms: see click().

Open a local file in an app ON an existing instance's display (NOT a new instance).

If app is given, runs <app> <filepath> there (e.g. app='gimp'); otherwise tries xdg-open. Relative paths resolve against the instance's staging folder files// (see list_files). Returns {"ok", "action", "instance", "opened", "app"}; fails (isError) if the file or the app doesn't exist. The app is started asynchronously — screenshot after a moment to confirm it drew. For BROWSER uploads use upload_file() instead. GTK apps get the same X11 + D-Bus handling as launch_app (so they render on this display, not the host).

The instance's staging folder (files// under the install dir) and its contents.

Returns {"ok", "instance", "dir", "count", "truncated", "files": […]} (capped at 200 entries, sorted). Drop files there (or reference any host path you can read) to open in apps via open_file() or an app's Open dialog; launch_app Chromium instances also download / 'save as' into it — call this after a download to confirm the file arrived. Browser uploads still go through upload_file().

Press a key or combo (xdotool syntax): 'Return', 'ctrl+a', 'Tab', 'ctrl+t', 'ctrl+l', 'F5'.

Returns {"ok", "action", "instance", "detail"}. Fails (isError) on an invalid keysym or a dead instance. Browser navigation: 'ctrl+l' → 'ctrl+a' → type_text(url) → 'Return'. observe/settle_ms: see click().

Move the mouse pointer to absolute pixel coords in an instance (no click).

Returns {"ok", "action", "instance", "detail"}. Useful for hover states (menus, tooltips) — follow with screenshot() to see the effect. Fails (isError) if the instance is down.

Scroll the wheel at the current mouse position: positive = up, negative = down.

Returns {"ok", "action", "instance", "detail"}. One unit ≈ one wheel notch (a few lines). Scrolling targets the window under the pointer — move_mouse/click there first. Fails (isError) on amount=0 (a no-op) or a dead instance. observe/settle_ms: see click().

Wait (e.g. for a page to load or an app to draw) — capped at 10000 ms.

Returns {"ok", "action", "requested_ms", "waited_ms", "clamped"} — "clamped": true means you asked for more than the cap and only waited_ms elapsed. Prefer settle_ms on the action itself when you also want to observe the result in the same call.

Server + sandbox status in one read-only call — run this first, and when filing bug reports.

Returns {"ok", "version", "vision", "host_display", "instances": [...same as list_instances], "deps": {xdotool, xclip, xephyr, wmctrl, tesseract, omniparser_weights: true|false}}. "vision" is the grounding backend (none|basic|local); "deps" shows whether each system tool is installed and (for 'local') whether the OmniParser weights are present. No side effects.