Launch a GUI app in its OWN new Xephyr display/window; returns its instance id. command is any shell command (e.g. 'chromium', 'xterm', 'gimp'). For Chromium, the X11 flags + a per-instance profile are added automatically. Control it with instance=.

List all instances (running app windows) with their display + name.

Close an instance (kills its app + Xephyr display). Refuses instance 1 (the main sandbox).

Screen size of an instance + active vision backend.

Screenshot of an instance's window.

Detect on-screen elements (id, type, label, interactive, pixel-center) for an instance (backend = GLOVEBOX_VISION). Saves a numbered image to /tmp/glovebox_annotated_.png.

Click at absolute coords. observe='screenshot'|'parse' returns the resulting screen state in the SAME call (saves a round-trip); settle_ms waits for the UI to update first. Default 'none'.

Drag (mouse-down → move → up) from (x1,y1) to (x2,y2) — for drawing shapes / selecting.

Double-click at absolute coords in an instance.

Click an element id from the most recent parse_screen. observe/settle_ms: see click().

Type at the focused field — unicode-safe. Pure ASCII is typed via xdotool; text with any non-ASCII char (č/š/ž …) is inserted via the clipboard + ctrl+v, because xdotool's synthetic unicode keystrokes get silently dropped by some toolkits (e.g. Inkscape's GTK canvas) even though they work in browsers. observe/settle_ms: see click().

Attach a local file to a page's via Chrome DevTools Protocol — use this for ALL browser uploads (logos, images, docs). The sandbox's native GTK file picker is invisible to us AND hangs the renderer, so never click an upload button expecting a dialog — call this instead. Works on Chromium started by launch_app / start-display.sh (they open a per-instance --remote-debugging-port). selector targets the file input on the MAIN page (default = first file input). If the site only inserts the after you click its 'upload' control, click that first (it won't open a dialog we can see, but it wires up the input), then call this. NATIVE (non-browser) apps: use open_file() or drive the app's own Open dialog — it IS visible here. observe/settle_ms: see click().

Open a local file in an app ON an existing instance's display (NOT a new instance). If app is given, runs <app> <filepath> there (e.g. app='gimp'); otherwise tries xdg-open. Relative paths resolve against the instance's staging folder files// (see list_files). For BROWSER uploads use upload_file() instead. GTK apps get the same X11 + D-Bus handling as launch_app (so they render on this display, not the host).

The instance's staging folder (files// under the install dir) and its contents. Drop files there (or reference any host path you can read) to open in apps via open_file() or an app's Open dialog; browser uploads still go through upload_file().

Press a key/combo (xdotool): 'Return','ctrl+a','Tab','ctrl+t'. observe/settle_ms: see click().

Move the mouse in an instance.

Scroll: positive = up, negative = down. observe/settle_ms: see click().

Wait (e.g. for a page to load). Capped at 10000 ms.