Skip to main content
Glama

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault
DISPLAYYesThe X11 display to control (e.g., ':1' for the sandbox). Required.
GLOVEBOX_VISIONNoVision backend: 'none', 'basic', or 'local'.local
GLOVEBOX_HOST_DISPLAYNoHost display for new instances.:0

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

Sandboxed computer use: drive real apps inside nested X11 windows. Call status() first; every tool returns JSON with ok/fix. Prefer labels over pixel guesses: find_elements('Save') -> click_element(id), wait_for_text('Done') instead of sleeping, fill(text, x, y) to set a form field and read it back, browser(action='text') for exact page text. click() reports the element it hit in browsers. See AGENTS.md in the repo.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": false
}
prompts
{
  "listChanged": false
}
resources
{
  "subscribe": false,
  "listChanged": false
}

Tools

Functions exposed to the LLM to take actions

NameDescription
launch_appA

Launch a GUI app in its OWN new Xephyr display/window and return its instance id.

command is any shell command ('chromium', 'xterm', 'gimp', 'inkscape file.svg', …). Chromium-family browsers (chromium, google-chrome, brave, microsoft-edge, vivaldi, opera) automatically get X11 flags, a per-instance profile, downloads routed to files//, and a CDP debug port (needed by upload_file and browser()). Other GTK apps get D-Bus isolation so they render on this display, not the host.

Returns {"ok", "instance", "display", "name", "files_dir", "width", "height", "warnings": [...], "keyboard": {method, host_layout, verified}, "clipboard_sync": bool} — pass that instance id to every control tool. "keyboard" says how the window got the host's keyboard layout ("host-keymap" = exact copy, verified against the host; Xephyr alone maps AltGr as plain Alt, so a person could not type @ { } [ ] on Slovenian/German/… layouts); "clipboard_sync" is true when the per-instance host bridge runs (it mirrors text between the host and instance clipboards and keeps the keymap equal to the host's; GLOVEBOX_CLIPBOARD_SYNC=0 disables it — note that whatever the person copies on the host then becomes readable inside the sandbox). Neither affects the agent's own type_text/clipboard. Fails (isError) BEFORE anything starts if the command is empty/unparsable or size is not WxH; after that if Xephyr can't start on the host display (its log tail is quoted), the command doesn't exist, or the app exits immediately (its stderr tail is quoted; app_log(instance) has more) — the display is closed again in those cases. "warnings" lists what degraded: window manager (openbox) missing (Xephyr then ignores pointer motion, so clicks would land in the screen center), keymap copy or clipboard bridge failed. Each instance has its own cursor, so separate agents can drive separate instances in parallel; display numbers are allocated under a cross-process lock, so parallel launches don't collide. Never share one instance between agents. The window appears on GLOVEBOX_HOST_DISPLAY (default :0). Give the app a moment to draw (wait_for_stable / settle_ms) before the first screenshot. Instances outlive this server process unless GLOVEBOX_CLEANUP_ON_EXIT=1.

list_instancesA

List all running instances (app windows): id, display, name, command, title, launched_here.

Returns {"ok", "count", "instances": [{instance, display, name, command, up, title, launched_here, clipboard_sync}, …]}. "title" is the active window's title (a browser's follows the page) — the way to tell instances apart that another agent's server launched (their name/command show as unknown, launched_here=false). Probes displays :1–:24 for sandboxes started outside this server (start-display.sh, other agents). Instances whose window was closed by hand disappear from the list and their number is reused.

close_instanceA

Close an instance: kill its app + its Xephyr display, and verify it actually went down.

Returns {"ok", "instance", "detail"} only after the display stops responding. With all_launched=true closes EVERY instance this server process launched (instance is ignored) and returns {"ok", "closed": [...], "failed": [...]} — "ok" is false if any survived. Fails (isError) if the instance isn't running, if it survives SIGTERM, or for instance 1 — the main start-display.sh sandbox is protected (stop it from the host with pkill Xephyr). Closing an instance another agent launched is allowed but visible to it (launched_here).

start_appA

Start ANOTHER app on an existing instance's display (same window/cursor) — e.g. a second tool next to the browser, or a restart of an app that quit. For a new, separate window use launch_app; to open a file in an app use open_file.

Returns {"ok", "instance", "command", "pid"}; fails (isError) if the instance is down, the command is unparsable/missing, or the app exits immediately (stderr tail quoted; app_log(instance) has more — logs of apps started here are appended to it). Browsers get the same X11/profile/CDP handling as launch_app. The app starts asynchronously: wait_for_stable() or screenshot() after a moment to confirm it drew.

get_screen_sizeA

Screen size of an instance + the active vision backend and server version.

Returns {"ok", "instance", "width", "height", "vision", "version"}. Call it once before coordinate math — coordinates are absolute pixels, top-left origin, within THIS instance's display (0 <= x < width, 0 <= y < height; out-of-range coordinates are rejected). "vision" is the EFFECTIVE grounding backend: 'basic'/'local' → parse_screen + click_element; 'none' → screenshot + reason about pixels yourself. Fails (isError) if the instance is down.

screenshotA

Screenshot of an instance's display — the ONLY way to see the screen; always look before you act.

Returns [json, image]: {"ok", "instance", "width", "height", "region", "scale", "format", "bytes"} plus the image. Options to cut payload or zoom: region="x,y,w,h" crop to a rectangle (zoom into small text/icons; coordinates stay absolute — add x,y back when you click), default = full display; scale=0.5 downscale (0.1–1.0) for cheap overview shots; format="jpeg" smaller than PNG for photos/video, quality 1–95 (default png, lossless). For an element list with coordinates use parse_screen (basic/local vision). Prefer observe='screenshot' on action tools to act and see the result in one call. Fails (isError) if the instance is down, the region is outside the display, or the format is unknown.

parse_screenA

Detect on-screen elements and return them with ids + pixel-centers (vision grounding).

Returns {"ok", "vision", "screen": [W,H], "annotated", "count", "truncated", "elements": [{id, type, label, interactive, center: [x,y]}, …]} — then click_element(id). Also saves a numbered Set-of-Mark image (path in "annotated"). Backend = the effective GLOVEBOX_VISION: 'local' (OmniParser on the GPU, ~2 s after a ~10 s first-call model load; served by ONE shared vision-service process per machine so parallel agents don't each load the models — "via": "service"|"in-process", "queued_ms" says how long the GPU queue was, "detector" which detector answered, "fallback_reason" why the service was not used. "type" is "text"|"icon" with OmniParser's own detector; with the optional ScreenParser detector (status().vision.omniparser.detector) it is one of text, button, link, tab, input, checkbox, radio, switch, select, slider, stepper, rating, icon, image, item, indicator and each element also carries "class", the detector's raw class name; buttons/tabs/ links carry their visible text as "label", icons/images a short caption. box_threshold raises the detector's confidence floor to cut noise), 'basic' (tesseract, text only; granularity='line' merges words on one line into one target like "Sign in" — default, split at wide gaps so a menu bar is one target per item — or 'word' for single words; the frame is OCR'd at 2× (GLOVEBOX_OCR_SCALE) because UI text is too small for tesseract at 1×; each element also carries "confidence" 0–100), 'none' (returns a note telling you to screenshot + reason about pixels yourself — this is normal, not an error; status() says why if 'auto' fell back to it). Fails (isError) with the fix named when an explicitly requested backend is not usable (OmniParser not installed, no/too-small GPU, GPU out of memory, the vision service busy, tesseract missing its language data). Elements are capped at 300 per call ("truncated": true when more were detected) — use find_elements to search instead of paging. Ids are only valid until the screen changes — re-parse after navigation.

find_elementsA

Parse the screen and return only the elements whose label contains query (case-insensitive substring; empty query = all), optionally filtered by type (button, link, text, icon, …). One call instead of parse_screen + scanning 300 elements. The ids are valid for click_element(id) right away (this IS the most recent parse).

Returns {"ok", "instance", "vision", "query", "type", "count": total matches, "elements": [{id, type, label, interactive, center}, …] (first limit, 1–100), "truncated"}. "count": 0 is an honest answer (the text is not on screen, or OCR did not read it — screenshot to check). Fails (isError) like parse_screen; in vision mode 'none' it fails with the fix (no grounding backend: screenshot and reason yourself).

wait_for_textA

Wait until text appears on screen (or, appear=false, disappears) — by OCR/parse polling. The honest replacement for "sleep and hope" after navigation or a submit: returns as soon as the text is (not) there, or reports "timed_out": true.

Returns {"ok", "instance", "found": bool, "satisfied": bool, "elapsed_ms", "polls", "element": {id, type, label, center} | null, "timed_out"}. "satisfied" is what you waited for; "timed_out": true with satisfied: false is reported, not hidden — screenshot to see what IS there. timeout_ms max 60000, interval_ms 300–5000. Fails (isError) if the instance is down or there is no grounding backend (vision mode 'none').

clickA

Click at absolute pixel coords (top-left origin) in an instance.

For anything with a label, find_elements('Save') -> click_element(id) beats guessing pixels. Returns {"ok", "action", "instance", "detail", "hit"?, "warnings"?}. "hit" (Chromium-family browsers only, read from the page's DOM just before the click) names what was under the point: {"role": "checkbox", "name": "Remember me", "tag", "checked_before", "disabled", "href", …} or {"role": "browser-ui"} for the toolbar/tab strip — compare it with what you meant to click; it is absent for native apps. "warnings" appears when the pointer was not where this server left it (a person or another agent is using the instance — re-screenshot). button: 1=left, 2=middle, 3=right. count: 1–3 clicks (2 = double-click, 3 = triple-click selects a line/paragraph). modifiers: keys held during the click — 'ctrl' (open in new tab / multi-select), 'shift' (range select), 'alt', 'super', combinable 'ctrl+shift'; released afterwards whatever happens. observe='screenshot'|'parse' additionally returns the resulting screen state in the SAME call (saves a round-trip); settle_ms (max 5000, "settle_clamped": true if you asked for more) waits for the UI to update first — use 400–1500 after anything that navigates or submits, or wait_for_stable() when you don't know how long. observe_region='x,y,w,h' crops that screenshot to the part you care about (implies observe='screenshot'; far fewer tokens than the full display; validated before the action). Fails (isError) if the instance is down, the coordinates are outside the display or a modifier name is unknown (nothing is clicked then). A failed observation after a successful click is reported as "observed": false inside an ok result — the click is not repeated.

dragA

Drag (mouse-down → move → up) from (x1,y1) to (x2,y2) — for drawing, selecting, sliders.

Returns {"ok", "action", "instance", "detail", "steps"}. steps (default 10, 1–100) is the number of intermediate motion events: canvases, sliders and drag-and-drop targets usually need a few to register the drag (1 = jump straight to the end point). The button is released even if a motion step fails (a stuck button would turn every later click into a drag). observe/settle_ms: see click(). Fails (isError) if the instance is down or either point is off-screen.

double_clickA

Double-click at absolute pixel coords in an instance (e.g. open a file icon, select a word). Same as click(count=2).

Returns {"ok", "action", "instance", "detail", "hit"?, "warnings"?} — see click(). observe/settle_ms/observe_region: see click(). Fails (isError) if the instance is down or the coordinates are off-screen.

click_elementA

Click an element by id from that instance's most recent parse_screen/find_elements (no coordinate guessing).

Returns {"ok", "action", "instance", "detail", "element", "center", "hit"?, "warnings"?} ("hit"/"warnings": see click()). Fails (isError) with the fix named if the id is unknown — run parse_screen(instance) first. Ids go stale when the screen changes: after navigation, re-parse before clicking. modifiers/observe/settle_ms: see click().

type_textA

Type into the focused field — unicode-safe. Click the field first to focus it.

To set a form field in one verified call use fill() instead (click + replace + read back). Returns {"ok", "action", "instance", "detail", "method": "xdotool"|"clipboard", "warnings"?}. clear=true selects the field's content first (ctrl+a) so the text REPLACES it; submit=true presses Return afterwards — one call for "fill and submit". Pure ASCII is typed via xdotool (timeout scales with length); text with any non-ASCII char (č/š/ž …) is inserted via the clipboard + ctrl+v, because xdotool's synthetic unicode keystrokes get silently dropped by some toolkits (e.g. Inkscape's GTK canvas) even though they work in browsers. The clipboard is put back as it was afterwards (previous text, or cleared), best-effort: an app that reads the selection late (> 0.3 s) can paste the restored text instead, and the host bridge may have mirrored the typed text in that window — see clipboard(). Terminals that paste with shift+Insert instead of ctrl+v need press_keys after clipboard(action='set'). Fails (isError) rather than silently losing characters if xclip is missing for non-ASCII text. observe/settle_ms: see click().

fillA

Set a form field in ONE verified call: click it, check the focus, replace its content with text, read the value back. Use it instead of click + type_text + screenshot for every field of a form.

Target the field with x,y (absolute pixels) or element (an id from parse_screen/ find_elements). Steps and what each can tell you:

  1. click, then a FOCUS CHECK before anything is typed — browsers through the page's DOM, native GTK/Qt/LibreOffice apps through the accessibility bus (AT-SPI). If the focus is not on an editable field, or is on a field whose rectangle does not contain the click (you clicked a label; the focus stayed where it was), the call FAILS with nothing typed and "focus" naming what has it. A password field is refused the same way unless allow_password=true — landing in one by accident is the case worth stopping.

  2. the old content is replaced (select-all, then type; text="" empties the field);

  3. the value is read back; submit=true presses Return only if nothing failed. Returns {"ok", "action", "instance", "detail", "method", "center", "focus_checked": "browser-dom"|"at-spi"|false, "focus"?: {role, name, rect}, "verified": true|false|null, "read_back": "browser-dom"|"clipboard"|"unreadable"|"skipped", "value"?, "click_effect"?: "changed"|"none", "suggest_element"?, "hit"?, "warnings"?}: focus_checked false — nobody could tell which widget had focus (canvas, game, Xlib-only app, iframe, or AT-SPI not installed: "focus_unchecked_reason"). The text was typed blind; "warnings" says so and "click_effect" reports whether the click changed any pixel near the target ("none" = probably a static label). Treat such a fill as UNCONFIRMED and screenshot. verified true — the focused field holds exactly text (trailing newline ignored); verified false — it holds something else ("value", first 200 chars): the field reformats or limits input (dates, phone masks, maxlength); verified null — the value could not be read (native password field, canvas). With element, a typed detector (ScreenParser) warns when the element is a text label and names the nearest input ("suggest_element"). Also fails (isError) when the field is still empty after typing; repeating fill() is always safe (it replaces). Native read-back goes through select-all + copy (the clipboard is restored). observe/settle_ms/ observe_region: see click().

clipboardA

Read or set the instance's clipboard — the cheapest way to get EXACT text out of an app.

action='get' returns {"ok", "instance", "text", "length", "empty", "truncated"}: select text in the app (click/drag, or press_keys('ctrl+a')) then press_keys('ctrl+c') and read it here — no OCR, no screenshot. "empty": true means nothing (or non-text data) is on the clipboard; text is capped at 100000 characters ("truncated": true). action='set' puts text on the clipboard (then press_keys('ctrl+v') or 'shift+Insert' in terminals) and returns {"ok", "instance", "length"}. Each instance has its own clipboard (separate X display); the bridge started by launch_app mirrors TEXT between it and the host's clipboard within ~1 s (so a person can copy/paste across — and whatever they copy on the host is readable here; GLOVEBOX_CLIPBOARD_SYNC=0 turns it off). Fails (isError) if xclip is missing, the instance is down, or xclip itself fails.

upload_fileA

Attach a local file to a page's via Chrome DevTools Protocol — use this for ALL browser uploads (logos, images, docs).

The sandbox's native GTK file picker is invisible to us AND hangs the renderer, so never click an upload button expecting a dialog — call this instead. Works on Chromium-family browsers started by launch_app / start-display.sh (they open a per-instance --remote-debugging-port, 9222+N). selector targets the file input on the MAIN page (default = first file input). With several tabs open the most recently active page target is used unless page_url_contains picks the tab by a URL fragment. If the site only inserts the after you click its 'upload' control, click that first (it won't open a dialog we can see, but it wires up the input), then call this. Relative paths resolve against files//. Returns {"ok", "action", "instance", "detail", "page"}; fails (isError) with the reason if the file is missing, no CDP target answers, or the selector matches nothing. NATIVE (non-browser) apps: use open_file() or drive the app's own Open dialog — it IS visible here. observe/settle_ms: see click().

browserA

Talk to the instance's Chromium-family browser over the Chrome DevTools Protocol — exact navigation and page text without keystrokes or OCR.

action='info' → {"ok", "url", "title", "pages": [{url, title}, …], "viewport"} of the active tab + all tabs. viewport = {dpr, left, top, css_width, css_height} maps the page's CSS pixels (getBoundingClientRect) to the pixels click()/screenshot() use: x = left + css_xdpr, y = top + css_ydpr (dpr is not 1 on a zoomed page; top is the toolbar height) action='navigate' → loads url in the active tab (waits for the load event, max 30 s); returns {"ok", "url", "title", "loaded": bool} action='text' → {"ok", "url", "title", "text", "length", "truncated"}: the rendered page text (document.body.innerText), capped at max_chars (default 20000, max 200000) action='eval' → {"ok", "result"}: the JSON value of the JavaScript expression js (e.g. "document.querySelector('h1').innerText"); a Promise is awaited. A result whose JSON is longer than max_chars (default 100000, max 1000000) comes back as {"ok", "truncated": true, "length", "result_json": the first max_chars characters} — raise max_chars or return less (slice/map in the script). Errors in the page script come back as a failure with the message timeout_s (1–600, default 30) bounds an eval or text call. On a timeout the call fails (isError, "timed_out": true) and the connection stays usable, but the script KEEPS RUNNING in the page — for long work store the outcome on window (js="fetchAll().then(r => { window.__r = r }); 'started'") and poll with short evals. page_url_contains picks a tab by URL fragment (default: the most recently active page). Works only for browsers started by launch_app/start-display.sh (CDP port 9222+N); fails (isError) with the fix otherwise, or when the tab, URL or script is bad. The page state you see in screenshots is this same tab.

open_fileA

Open a local file in an app ON an existing instance's display (NOT a new instance).

If app is given, runs <app> <filepath> there (e.g. app='gimp'); otherwise xdg-open picks the default handler. Relative paths resolve against the instance's staging folder files// (see list_files). Returns {"ok", "action", "instance", "opened", "app"}; fails (isError) if the file or the app doesn't exist, or if the handler exits immediately (e.g. xdg-open finds no application for that file type — then pass app=). The app is started asynchronously — wait_for_stable()/screenshot after a moment to confirm it drew. For BROWSER uploads use upload_file() instead. Both paths get the same X11 + D-Bus handling as launch_app (so the app renders on this display, not in an already-running window on the host).

list_filesA

The instance's staging folder (files// under the install dir) and its contents.

Returns {"ok", "instance", "dir", "count", "truncated", "files": [{name, bytes, modified, is_dir, partial}, …]} (capped at 200 entries, newest first). "partial": true marks an in-progress browser download (.crdownload/.part/.tmp) — wait for it to disappear before using the file. Drop files there (or reference any host path you can read) to open in apps via open_file() or an app's Open dialog; launch_app browser instances also download / 'save as' into it — call this after a download to confirm the file arrived. Browser uploads still go through upload_file(). Fails (isError) if the folder cannot be read.

put_fileA

Write a file into the instance's staging folder files// — the way to hand the sandbox a file you produced (a CSV to import, an SVG to open, a text to upload) without shell access. Then open_file(name) / upload_file(name) / an app's Open dialog can use it by that name.

name is a relative path inside the staging folder ('data.csv', 'in/logo.svg'; parent folders are created); absolute paths and '..' are refused — nothing outside files// can be written. content is text (written as UTF-8) or, with encoding='base64', binary data. Returns {"ok", "instance", "path", "name", "bytes", "overwritten"}. Fails (isError), writing nothing, if the file exists and overwrite is false, the name escapes the folder or names a directory, the base64 is invalid, or the content exceeds 10 MB. The instance does not have to be running (stage files first, launch after).

press_keysA

Press a key, a combo, or a SEQUENCE (xdotool keysym syntax, space-separated for sequences): 'Return', 'ctrl+a', 'Tab', 'ctrl+t', 'F5', 'ctrl+a Delete', 'alt+F4', 'shift+Insert'.

Returns {"ok", "action", "instance", "detail", "keys": [...]}. Every token is validated against the X keysym table BEFORE anything is pressed, so an invalid name fails with nothing pressed — also in the middle of a sequence. Fails (isError) on a dead instance or a frozen display. Browser navigation: 'ctrl+l' → type_text(url) → 'Return' (or browser(action= 'navigate')). observe/settle_ms: see click().

move_mouseA

Move the mouse pointer to absolute pixel coords in an instance (no click).

Returns {"ok", "action", "instance", "detail"}. Useful for hover states (menus, tooltips) — follow with screenshot() to see the effect. Fails (isError) if the instance is down or the coordinates are off-screen.

scrollA

Scroll the wheel. Say where: direction='down'|'up'|'left'|'right' with amount notches (sign ignored) — or the signed form: positive = up (or left), negative = down (or right).

Returns {"ok", "action", "instance", "detail", "applied", "clamped", "axis", "direction"}. One unit ≈ one wheel notch (a few lines); at most 100 notches per call ("clamped": true when you asked for more, "applied" says how many were sent). Scrolling targets the window under the pointer — pass x,y to move there first (both >= 0), otherwise the current pointer position is used. axis='horizontal' scrolls sideways (wheel buttons 6/7). Fails (isError) on amount=0 (a no-op), a dead instance, off-screen x,y or an unknown axis. observe/settle_ms: see click().

wait_msA

Wait (e.g. for a page to load or an app to draw) — capped at 10000 ms.

Returns {"ok", "action", "requested_ms", "waited_ms", "clamped"} — "clamped": true means you asked for more than the cap and only waited_ms elapsed. Prefer wait_for_stable() (waits until the screen stops changing), wait_for_text(), or settle_ms on the action itself when you also want to observe the result in the same call.

wait_for_stableA

Wait until the screen stops changing (page loaded, app drew, animation ended) — replaces guessing a settle_ms.

Samples the display every interval_ms (100–2000) and returns as soon as two consecutive frames differ by at most threshold (fraction of pixels, default 0.2%), or after timeout_ms (max 30000). Returns {"ok", "instance", "stable", "elapsed_ms", "frames", "change": last changed fraction, "timed_out"}. "stable": false with "timed_out": true means the screen was still changing (video, spinner, blinking cursor — raise the threshold to ~0.01 for those); it is reported, not hidden. observe='screenshot'|'parse' returns the final state in the same call. Fails (isError) if the instance is down.

window_infoA

The active window's title + geometry, the visible windows and the pointer position on an instance — a cheap, image-free way to confirm navigation (browser titles follow the page) or that an app opened.

Returns {"ok", "instance", "active": {id, title, x, y, width, height, pid} | null, "windows": [{id, title}, …] (visible, capped at 50, unnamed ones included with title ""), "count", "truncated", "cursor": [x, y]}. "active" is null when no window has focus yet (fresh instance) — that is reported, not an error. Window ids are what manage_window() takes. Fails (isError) if the instance is down or xdotool is missing/fails.

manage_windowA

Act on a window of an instance: action='activate' (focus/raise — switch between an app's windows or dialogs), 'close' (asks the app to close it — a save prompt may appear), 'minimize', 'maximize', 'restore' (un-maximize), 'move' (to x,y), 'resize' (to width x height). window_id=0 means the active window (window_info lists ids).

Returns {"ok", "action", "instance", "window", "detail"} plus observe/settle_ms like click(). Fails (isError) if the window id is unknown, the action is unknown, move/resize sizes are missing, or the instance is down. maximize/restore need wmctrl (status().deps).

app_logA

Tail of the stderr log of the apps launched on an instance (launch_app / start_app / open_file) — read it when an app crashed, drew nothing, or misbehaves.

Returns {"ok", "instance", "path", "lines": [...], "total_lines", "truncated"} (max 500 lines per call). The log is per instance: launch_app starts it fresh, start_app/open_file append. Instance 1 (start-display.sh) logs to /tmp/chromium-x1.log instead. Fails (isError) if no log exists for that instance yet.

statusA

Server + sandbox status in one read-only call — run this first, and when filing bug reports.

Returns {"ok", "version", "host_display", "tmp_dir", "cleanup_on_exit", "vision": {requested, effective, ready, reason, fallback, gpu: {present, name, vram_total_mb, vram_free_mb, enough, note}, omniparser: {code, weights, torch, ready, loadable, reason, detector: {requested, effective, screenparser_weights, reason}, loaded, service: {running, pid, loaded, detector, served, in_flight, idle_s, …} — the shared per-machine OmniParser process (one GPU load for all agents; starts on first use, idle-exits)}, tesseract: {binary, pytesseract, langs, lang, ready, reason}}, "instances": [...same as list_instances], "keyboard": {host_layout, override}, "clipboard_sync": {enabled, poll_s, note}, "deps": {xdotool, xclip, xephyr, openbox, wmctrl, xdpyinfo, xkbcomp, setxkbmap, dbus_run_session, tesseract, nvidia_smi, xprop, atspi: true|false}, "notes": [...], "tips": [...]} ("tips": the tools that save the most round-trips on this setup — read them once). "vision.effective" is the grounding backend actually in use (none|basic|local), re-resolved every 20 s (installing OmniParser takes effect without a restart); when GLOVEBOX_VISION=auto it is the best READY one and "reason" says why (e.g. OmniParser weights missing, GPU too small). "notes" lists things worth knowing (e.g. the service runs a different detector than this server would pick). No side effects.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription

No resources

TDQS

A3.9/5.0

Scored across 30 tools

Disambiguation4/5

Most tools target clearly distinct operations; launch_app/start_app/open_file and click/click_element/double_click are separated by precise descriptions. Minor overlap remains between fill and type_text, parse_screen and find_elements, and double_click as click(count=2), but the descriptions make the intended choice clear.

Naming Consistency4/5

All tool names use snake_case with no camelCase mixing, and most follow a readable verb_noun pattern. A few bare nouns (status, browser, clipboard, screenshot) and mixed get_/info patterns are minor deviations rather than inconsistency.

Tool Count2/5

At 30 tools the surface is heavier than ideal, and several conveniences such as double_click, find_elements, and wait_ms could be folded into existing tools. The broad GUI/browser/vision/file scope explains many tools, but the count is still well above the recommended range.

Completeness4/5

The set covers the full lifecycle well: launch/start/close/list instances, mouse and keyboard control, screenshots and vision parsing, browser CDP actions, window management, clipboard, and file staging/upload. Minor gaps exist around direct file deletion/rename and some file-management operations, but agents can generally work around them via overwrite or app dialogs.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues