glovebox-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| DISPLAY | Yes | The X11 display to control (e.g., ':1' for the sandbox). Required. | |
| GLOVEBOX_VISION | No | Vision backend: 'none', 'basic', or 'local'. | local |
| GLOVEBOX_HOST_DISPLAY | No | Host display for new instances. | :0 |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
Sandboxed computer use: drive real apps inside nested X11 windows. Call status() first; every tool returns JSON with ok/fix. Prefer labels over pixel guesses: find_elements('Save') -> click_element(id), wait_for_text('Done') instead of sleeping, fill(text, x, y) to set a form field and read it back, browser(action='text') for exact page text. click() reports the element it hit in browsers. See AGENTS.md in the repo.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| launch_appA | Launch a GUI app in its OWN new Xephyr display/window and return its instance id.
Returns {"ok", "instance", "display", "name", "files_dir", "width", "height", "warnings": [...],
"keyboard": {method, host_layout, verified}, "clipboard_sync": bool} — pass that instance id to every
control tool. "keyboard" says how the window got the host's keyboard layout ("host-keymap" =
exact copy, verified against the host; Xephyr alone maps AltGr as plain Alt, so a person could
not type @ { } [ ] on Slovenian/German/… layouts); "clipboard_sync" is true when the per-instance
host bridge runs (it mirrors text between the host and instance clipboards and keeps the
keymap equal to the host's; GLOVEBOX_CLIPBOARD_SYNC=0 disables it — note that whatever the
person copies on the host then becomes readable inside the sandbox). Neither affects the
agent's own type_text/clipboard. Fails (isError) BEFORE anything starts if the command is
empty/unparsable or |
| list_instancesA | List all running instances (app windows): id, display, name, command, title, launched_here. Returns {"ok", "count", "instances": [{instance, display, name, command, up, title, launched_here, clipboard_sync}, …]}. "title" is the active window's title (a browser's follows the page) — the way to tell instances apart that another agent's server launched (their name/command show as unknown, launched_here=false). Probes displays :1–:24 for sandboxes started outside this server (start-display.sh, other agents). Instances whose window was closed by hand disappear from the list and their number is reused. |
| close_instanceA | Close an instance: kill its app + its Xephyr display, and verify it actually went down. Returns {"ok", "instance", "detail"} only after the display stops responding. With
all_launched=true closes EVERY instance this server process launched (instance is ignored)
and returns {"ok", "closed": [...], "failed": [...]} — "ok" is false if any survived.
Fails (isError) if the instance isn't running, if it survives SIGTERM, or for instance 1 —
the main start-display.sh sandbox is protected (stop it from the host with |
| start_appA | Start ANOTHER app on an existing instance's display (same window/cursor) — e.g. a second tool next to the browser, or a restart of an app that quit. For a new, separate window use launch_app; to open a file in an app use open_file. Returns {"ok", "instance", "command", "pid"}; fails (isError) if the instance is down, the command is unparsable/missing, or the app exits immediately (stderr tail quoted; app_log(instance) has more — logs of apps started here are appended to it). Browsers get the same X11/profile/CDP handling as launch_app. The app starts asynchronously: wait_for_stable() or screenshot() after a moment to confirm it drew. |
| get_screen_sizeA | Screen size of an instance + the active vision backend and server version. Returns {"ok", "instance", "width", "height", "vision", "version"}. Call it once before coordinate math — coordinates are absolute pixels, top-left origin, within THIS instance's display (0 <= x < width, 0 <= y < height; out-of-range coordinates are rejected). "vision" is the EFFECTIVE grounding backend: 'basic'/'local' → parse_screen + click_element; 'none' → screenshot + reason about pixels yourself. Fails (isError) if the instance is down. |
| screenshotA | Screenshot of an instance's display — the ONLY way to see the screen; always look before you act. Returns [json, image]: {"ok", "instance", "width", "height", "region", "scale", "format", "bytes"} plus the image. Options to cut payload or zoom: region="x,y,w,h" crop to a rectangle (zoom into small text/icons; coordinates stay absolute — add x,y back when you click), default = full display; scale=0.5 downscale (0.1–1.0) for cheap overview shots; format="jpeg" smaller than PNG for photos/video, quality 1–95 (default png, lossless). For an element list with coordinates use parse_screen (basic/local vision). Prefer observe='screenshot' on action tools to act and see the result in one call. Fails (isError) if the instance is down, the region is outside the display, or the format is unknown. |
| parse_screenA | Detect on-screen elements and return them with ids + pixel-centers (vision grounding). Returns {"ok", "vision", "screen": [W,H], "annotated", "count", "truncated",
"elements": [{id, type, label, interactive, center: [x,y]}, …]} — then click_element(id).
Also saves a numbered Set-of-Mark image (path in "annotated").
Backend = the effective GLOVEBOX_VISION: 'local' (OmniParser on the GPU, ~2 s after a ~10 s
first-call model load; served by ONE shared vision-service process per machine so parallel
agents don't each load the models — "via": "service"|"in-process", "queued_ms" says how long
the GPU queue was, "detector" which detector answered, "fallback_reason" why the service was
not used. "type" is "text"|"icon" with OmniParser's own detector; with the optional
ScreenParser detector (status().vision.omniparser.detector) it is one of text, button, link,
tab, input, checkbox, radio, switch, select, slider, stepper, rating, icon, image, item,
indicator and each element also carries "class", the detector's raw class name; buttons/tabs/
links carry their visible text as "label", icons/images a short caption. |
| find_elementsA | Parse the screen and return only the elements whose label contains Returns {"ok", "instance", "vision", "query", "type", "count": total matches, "elements":
[{id, type, label, interactive, center}, …] (first |
| wait_for_textA | Wait until Returns {"ok", "instance", "found": bool, "satisfied": bool, "elapsed_ms", "polls", "element": {id, type, label, center} | null, "timed_out"}. "satisfied" is what you waited for; "timed_out": true with satisfied: false is reported, not hidden — screenshot to see what IS there. timeout_ms max 60000, interval_ms 300–5000. Fails (isError) if the instance is down or there is no grounding backend (vision mode 'none'). |
| clickA | Click at absolute pixel coords (top-left origin) in an instance. For anything with a label, find_elements('Save') -> click_element(id) beats guessing pixels.
Returns {"ok", "action", "instance", "detail", "hit"?, "warnings"?}. "hit" (Chromium-family
browsers only, read from the page's DOM just before the click) names what was under the
point: {"role": "checkbox", "name": "Remember me", "tag", "checked_before", "disabled",
"href", …} or {"role": "browser-ui"} for the toolbar/tab strip — compare it with what you
meant to click; it is absent for native apps. "warnings" appears when the pointer was not
where this server left it (a person or another agent is using the instance — re-screenshot).
button: 1=left, 2=middle, 3=right.
count: 1–3 clicks (2 = double-click, 3 = triple-click selects a line/paragraph).
modifiers: keys held during the click — 'ctrl' (open in new tab / multi-select), 'shift'
(range select), 'alt', 'super', combinable 'ctrl+shift'; released afterwards whatever happens.
|
| dragA | Drag (mouse-down → move → up) from (x1,y1) to (x2,y2) — for drawing, selecting, sliders. Returns {"ok", "action", "instance", "detail", "steps"}. |
| double_clickA | Double-click at absolute pixel coords in an instance (e.g. open a file icon, select a word). Same as click(count=2). Returns {"ok", "action", "instance", "detail", "hit"?, "warnings"?} — see click().
|
| click_elementA | Click an element by id from that instance's most recent parse_screen/find_elements (no coordinate guessing). Returns {"ok", "action", "instance", "detail", "element", "center", "hit"?, "warnings"?}
("hit"/"warnings": see click()). Fails (isError) with the
fix named if the id is unknown — run parse_screen(instance) first. Ids go stale when the
screen changes: after navigation, re-parse before clicking. |
| type_textA | Type into the focused field — unicode-safe. Click the field first to focus it. To set a form field in one verified call use fill() instead (click + replace + read back).
Returns {"ok", "action", "instance", "detail", "method": "xdotool"|"clipboard", "warnings"?}.
clear=true selects the field's content first (ctrl+a) so the text REPLACES it;
submit=true presses Return afterwards — one call for "fill and submit".
Pure ASCII is typed via xdotool (timeout scales with length); text with any non-ASCII char
(č/š/ž …) is inserted via the clipboard + ctrl+v, because xdotool's synthetic unicode
keystrokes get silently dropped by some toolkits (e.g. Inkscape's GTK canvas) even though
they work in browsers. The clipboard is put back as it was afterwards (previous text, or
cleared), best-effort: an app that reads the selection late (> 0.3 s) can paste the restored
text instead, and the host bridge may have mirrored the typed text in that window — see
clipboard(). Terminals that paste with shift+Insert instead of ctrl+v need press_keys after
clipboard(action='set'). Fails (isError) rather than silently losing characters if xclip is
missing for non-ASCII text. |
| fillA | Set a form field in ONE verified call: click it, check the focus, replace its content
with Target the field with x,y (absolute pixels) or
|
| clipboardA | Read or set the instance's clipboard — the cheapest way to get EXACT text out of an app. action='get' returns {"ok", "instance", "text", "length", "empty", "truncated"}: select text
in the app (click/drag, or press_keys('ctrl+a')) then press_keys('ctrl+c') and read it here
— no OCR, no screenshot. "empty": true means nothing (or non-text data) is on the clipboard;
text is capped at 100000 characters ("truncated": true). action='set' puts |
| upload_fileA | Attach a local file to a page's via Chrome DevTools Protocol — use this for ALL browser uploads (logos, images, docs). The sandbox's native GTK file picker is invisible to us AND hangs the renderer, so never click
an upload button expecting a dialog — call this instead. Works on Chromium-family browsers
started by launch_app / start-display.sh (they open a per-instance --remote-debugging-port,
9222+N). |
| browserA | Talk to the instance's Chromium-family browser over the Chrome DevTools Protocol — exact navigation and page text without keystrokes or OCR. action='info' → {"ok", "url", "title", "pages": [{url, title}, …], "viewport"} of the
active tab + all tabs. |
| open_fileA | Open a local file in an app ON an existing instance's display (NOT a new instance). If |
| list_filesA | The instance's staging folder (files// under the install dir) and its contents. Returns {"ok", "instance", "dir", "count", "truncated", "files": [{name, bytes, modified, is_dir, partial}, …]} (capped at 200 entries, newest first). "partial": true marks an in-progress browser download (.crdownload/.part/.tmp) — wait for it to disappear before using the file. Drop files there (or reference any host path you can read) to open in apps via open_file() or an app's Open dialog; launch_app browser instances also download / 'save as' into it — call this after a download to confirm the file arrived. Browser uploads still go through upload_file(). Fails (isError) if the folder cannot be read. |
| put_fileA | Write a file into the instance's staging folder files// — the way to hand the sandbox a file you produced (a CSV to import, an SVG to open, a text to upload) without shell access. Then open_file(name) / upload_file(name) / an app's Open dialog can use it by that name.
|
| press_keysA | Press a key, a combo, or a SEQUENCE (xdotool keysym syntax, space-separated for sequences): 'Return', 'ctrl+a', 'Tab', 'ctrl+t', 'F5', 'ctrl+a Delete', 'alt+F4', 'shift+Insert'. Returns {"ok", "action", "instance", "detail", "keys": [...]}. Every token is validated
against the X keysym table BEFORE anything is pressed, so an invalid name fails with nothing
pressed — also in the middle of a sequence. Fails (isError) on a dead instance or a frozen
display. Browser navigation: 'ctrl+l' → type_text(url) → 'Return' (or browser(action=
'navigate')). |
| move_mouseA | Move the mouse pointer to absolute pixel coords in an instance (no click). Returns {"ok", "action", "instance", "detail"}. Useful for hover states (menus, tooltips) — follow with screenshot() to see the effect. Fails (isError) if the instance is down or the coordinates are off-screen. |
| scrollA | Scroll the wheel. Say where: direction='down'|'up'|'left'|'right' with Returns {"ok", "action", "instance", "detail", "applied", "clamped", "axis", "direction"}. One unit ≈ one wheel
notch (a few lines); at most 100 notches per call ("clamped": true when you asked for more,
"applied" says how many were sent). Scrolling targets the window under the pointer — pass
x,y to move there first (both >= 0), otherwise the current pointer position is used.
axis='horizontal' scrolls sideways (wheel buttons 6/7). Fails (isError) on amount=0 (a
no-op), a dead instance, off-screen x,y or an unknown axis. |
| wait_msA | Wait (e.g. for a page to load or an app to draw) — capped at 10000 ms. Returns {"ok", "action", "requested_ms", "waited_ms", "clamped"} — "clamped": true means you asked for more than the cap and only waited_ms elapsed. Prefer wait_for_stable() (waits until the screen stops changing), wait_for_text(), or settle_ms on the action itself when you also want to observe the result in the same call. |
| wait_for_stableA | Wait until the screen stops changing (page loaded, app drew, animation ended) — replaces guessing a settle_ms. Samples the display every |
| window_infoA | The active window's title + geometry, the visible windows and the pointer position on an instance — a cheap, image-free way to confirm navigation (browser titles follow the page) or that an app opened. Returns {"ok", "instance", "active": {id, title, x, y, width, height, pid} | null, "windows": [{id, title}, …] (visible, capped at 50, unnamed ones included with title ""), "count", "truncated", "cursor": [x, y]}. "active" is null when no window has focus yet (fresh instance) — that is reported, not an error. Window ids are what manage_window() takes. Fails (isError) if the instance is down or xdotool is missing/fails. |
| manage_windowA | Act on a window of an instance: action='activate' (focus/raise — switch between an app's windows or dialogs), 'close' (asks the app to close it — a save prompt may appear), 'minimize', 'maximize', 'restore' (un-maximize), 'move' (to x,y), 'resize' (to width x height). window_id=0 means the active window (window_info lists ids). Returns {"ok", "action", "instance", "window", "detail"} plus |
| app_logA | Tail of the stderr log of the apps launched on an instance (launch_app / start_app / open_file) — read it when an app crashed, drew nothing, or misbehaves. Returns {"ok", "instance", "path", "lines": [...], "total_lines", "truncated"} (max 500 lines per call). The log is per instance: launch_app starts it fresh, start_app/open_file append. Instance 1 (start-display.sh) logs to /tmp/chromium-x1.log instead. Fails (isError) if no log exists for that instance yet. |
| statusA | Server + sandbox status in one read-only call — run this first, and when filing bug reports. Returns {"ok", "version", "host_display", "tmp_dir", "cleanup_on_exit", "vision": {requested, effective, ready, reason, fallback, gpu: {present, name, vram_total_mb, vram_free_mb, enough, note}, omniparser: {code, weights, torch, ready, loadable, reason, detector: {requested, effective, screenparser_weights, reason}, loaded, service: {running, pid, loaded, detector, served, in_flight, idle_s, …} — the shared per-machine OmniParser process (one GPU load for all agents; starts on first use, idle-exits)}, tesseract: {binary, pytesseract, langs, lang, ready, reason}}, "instances": [...same as list_instances], "keyboard": {host_layout, override}, "clipboard_sync": {enabled, poll_s, note}, "deps": {xdotool, xclip, xephyr, openbox, wmctrl, xdpyinfo, xkbcomp, setxkbmap, dbus_run_session, tesseract, nvidia_smi, xprop, atspi: true|false}, "notes": [...], "tips": [...]} ("tips": the tools that save the most round-trips on this setup — read them once). "vision.effective" is the grounding backend actually in use (none|basic|local), re-resolved every 20 s (installing OmniParser takes effect without a restart); when GLOVEBOX_VISION=auto it is the best READY one and "reason" says why (e.g. OmniParser weights missing, GPU too small). "notes" lists things worth knowing (e.g. the service runs a different detector than this server would pick). No side effects. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 30 tools
Most tools target clearly distinct operations; launch_app/start_app/open_file and click/click_element/double_click are separated by precise descriptions. Minor overlap remains between fill and type_text, parse_screen and find_elements, and double_click as click(count=2), but the descriptions make the intended choice clear.
All tool names use snake_case with no camelCase mixing, and most follow a readable verb_noun pattern. A few bare nouns (status, browser, clipboard, screenshot) and mixed get_/info patterns are minor deviations rather than inconsistency.
At 30 tools the surface is heavier than ideal, and several conveniences such as double_click, find_elements, and wait_ms could be folded into existing tools. The broad GUI/browser/vision/file scope explains many tools, but the count is still well above the recommended range.
The set covers the full lifecycle well: launch/start/close/list instances, mouse and keyboard control, screenshots and vision parsing, browser CDP actions, window management, clipboard, and file staging/upload. Minor gaps exist around direct file deletion/rename and some file-management operations, but agents can generally work around them via overwrite or app dialogs.