glovebox-mcp
This MCP server lets an AI agent safely control GUI applications and browsers inside isolated nested X11/Xephyr sandboxes.
Launch, list, inspect, and close sandbox app instances/windows.
Capture screenshots and use vision backends to detect elements, find text, and wait for UI changes.
Drive the GUI with clicks, double-clicks, drags, scrolling, mouse movement, keyboard shortcuts, typing, and verified form filling.
Control Chromium-family browsers via CDP: navigate, read page text, evaluate JavaScript, inspect tabs/viewport, and upload files.
Read/write the instance clipboard and manage windows: activate, close, minimize, maximize, move, resize.
Stage files in per-instance folders, list downloads, open files in native apps, and handle browser uploads.
Run multiple instances in parallel, each with its own display/cursor, and inspect app logs or server/GPU/vision status.
Keep agent input and vision confined to the sandbox window, while still letting a human watch and interact with the same sandbox.
Supports launching and automating Brave inside the sandbox as a Chromium-family browser, including isolated per-instance profiles, downloads to the instance staging folder, and CDP-based file uploads.
Supports launching and automating Opera inside the sandbox as a Chromium-family browser, including isolated per-instance profiles, downloads to the instance staging folder, and CDP-based file uploads.
Supports launching and automating Vivaldi inside the sandbox as a Chromium-family browser, including isolated per-instance profiles, downloads to the instance staging folder, and CDP-based file uploads.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@glovebox-mcp Open example.com and take a screenshot"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
glovebox-mcp
A sandboxed computer-use MCP server — let an AI agent drive a real browser and desktop apps (mouse, keyboard, screenshots, clipboard, vision grounding), confined to a nested X11 window so its input and vision never touch your real screen or the windows of your other apps.
Like a lab glovebox: the agent reaches in and manipulates real applications, sealed off from your desktop. Bring the sandbox up, log into whatever sites or apps you want to automate inside that window, and the agent operates only there — you can watch it live and close it instantly.
Speaks the Model Context Protocol, so it works with MCP clients like Claude Code. Your host can run Wayland; the sandbox gives the agent a real X server to drive.

An agent driving a real browser in the sandbox — gliding the cursor, inserting a unicode name (Nadja Kovačič), typing, and submitting. All confined to a nested X11 window.
Why a nested X11 sandbox?
Most desktop automation (xdotool, PyAutoGUI) is X11-only, but many modern desktops run Wayland.
Xephyr provides a real X server inside a single window (
DISPLAY :1). Everything the agent does — clicks, typing, screenshots — is confined to that window, not your real desktop.You stay in control: watch it live,
pkill Xephyrto close everything.
The isolation is display-level (see Safety): the apps themselves run as your user with your filesystem. For OS-level isolation run glovebox inside a VM or container.
Related MCP server: Secure Browser MCP
Requirements
Linux — the sandbox nests a real X server (Xephyr), so it works even on Wayland hosts (via Xwayland). Not macOS/Windows. Developed on Ubuntu; any modern Linux with the packages below.
Python 3.10+ and
uv(used for the virtualenv).System packages —
xserver-xephyr(Xephyr),openbox,scrot,x11-utils,x11-xkb-utils(xkbcomp: gives the sandbox your keyboard layout),xdotool,wmctrl,xclip, optionalat-spi2-core+gir1.2-atspi-2.0+python3-gi(focus check in native apps) (+tesseract-ocrforbasic). On Debian/Ubuntu the installer auto-installs them viaapt(sudo); on Fedora/Arch it prints the matchingdnf/pacmancommand. The MCP server itself is distro-agnostic — any Linux with these tools works.openboxis not optional in practice: without a window manager Xephyr ignores synthetic pointer motion, so clicks would land in the screen center (launch_appwarns when it is missing).A browser in the sandbox (Chromium, Chrome, Brave, Edge, Vivaldi or Opera).
NVIDIA GPU (≥6 GB VRAM) — only for the
localvision mode. The server checks this itself (status()→vision.gpu) andautofalls back tobasic/nonewhen it is not met.
Install
One command. The installer probes the GPU, installs the system packages (auto via apt on
Debian/Ubuntu; it prints the line for dnf/pacman), creates the Python env from pyproject.toml,
downloads the vision models the machine can run, writes mcp-config.json and .mcp.json
(Claude Code) with your paths, and installs the Claude Code skill:
git clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.shWhat auto picks: an NVIDIA GPU with ≥ 6 GB VRAM → local (OmniParser + ScreenParser, typed
elements; ~4 GB of downloads, --no-screenparser skips the 153 MB detector); otherwise
basic (Tesseract OCR, CPU). Tesseract is installed in every mode — the server's own auto
backend selection falls back to it when the GPU is busy. Force a mode with ./install.sh none|basic|local;
./install.sh skill installs only the skill; --no-skill, --dev (pytest) are the other flags.
Re-running is safe: it only downloads what is missing.
The requested mode is written to .vision-mode (auto unless you forced one; override per run
with the GLOVEBOX_VISION env var). python vision.py check prints the same diagnosis status()
returns (GPU, weights, detector, tesseract).
Claude Code skill
The installer copies skills/glovebox/SKILL.md into ~/.claude/skills/glovebox/ when ~/.claude
exists (./install.sh skill does only that) — a skill that teaches Claude Code the observe → act → verify loop, grounding per vision mode, the
upload and navigation gotchas, and when to stop. Claude Code loads it automatically when an agent
is asked to drive a GUI app (or explicitly with /glovebox). AGENTS.md remains the long form for
any other agent/harness.
Works with any MCP client / harness
Claude Code, Cursor, Codex, or your own agent — it's a standard MCP server, not tied to any one host.
Two compatibility notes: basic/local return element coordinates as text, so they work even
with text-only agents; none relies on the client passing the tool's screenshots to a
multimodal model (fine for Claude Code, Cursor, and other image-capable MCP clients). In any mode,
clipboard() gets exact text out of an app without vision at all.
Quickstart
Start the sandbox (leave it running):
./start-display.sh # 1440×900 Xephyr window with a browser ./start-display.sh 1920x1080 # …or pass a screen size (or set $RES)Log into any sites or apps you want to automate in that window.
Register the server with your MCP client.
install.shalready wrotemcp-config.jsonwith your real install path — copy itsgloveboxblock into your client's MCP config:{ "mcpServers": { "glovebox": { "command": "/abs/path/to/glovebox-mcp/.venv/bin/python", // filled in by install.sh "args": ["/abs/path/to/glovebox-mcp/server.py"], "env": { "DISPLAY": ":1" } } } }Restart the client so it loads the server.
Ask the agent to screenshot / click / type — it operates only on the
:1window.
Driving it with an AI agent? Paste
AGENTS.mdinto the agent's system prompt — it teaches the observe → act → verify loop, grounding, the upload/unicode gotchas, and when to stop.
Tools (30)
Tool | What |
| Server + sandbox status in one read-only call: version, the vision diagnosis (requested vs. effective backend and why, GPU name/VRAM, OmniParser code/weights, tesseract languages), live instances, and which system deps are present. Run it first — and paste it into bug reports. |
|
|
| Vision grounding → JSON of detected elements (id, type, label, interactive, pixel-center; capped at 300 per call, flagged via |
| Parse + filter in one call: the elements whose label contains |
| Poll OCR/parse until a text appears (or disappears) — the honest replacement for "sleep and hope" after navigation; |
| Set a form field in one verified call: click it, check the focus before typing, replace its content, read the value back. The focus check runs through the page's DOM in browsers and through the accessibility bus (AT-SPI) in native GTK/Qt/LibreOffice apps: a click that landed on a label (the focus stayed elsewhere), a non-editable widget, or a password field you did not announce ( |
| Click an element from the last |
| Pointer ops. In Chromium-family browsers a click returns |
| Unicode-safe typing (ASCII via xdotool; anything with č/š/ž… is inserted via the clipboard, because xdotool's synthetic unicode is silently dropped by some GTK apps; the clipboard is put back afterwards). |
| Keys, combos, and space-separated sequences (xdotool syntax). Every token is validated against the X keysym table before anything is pressed — an invalid name anywhere in a sequence errors with nothing pressed. |
| The instance's Chromium over CDP: current URL/title and tabs, exact navigation (waits for load), the rendered page text without OCR, or a JavaScript expression's value (Promises awaited; |
| activate / close / minimize / maximize / restore / move / resize a window (ids from |
| Start another app on an existing instance (same window/cursor) — a second tool, or a restart. |
| Read the instance's clipboard (select + |
| Wait until the screen stops changing (page loaded, app drew) instead of guessing a delay. Reports |
| Active window title + geometry, the visible windows (ids for |
| Attach a local file to a page's |
| Open a local file in a native app on the instance's display (e.g. |
| Write a file (text, or binary as base64; max 10 MB) into the instance's staging folder — hand the sandbox something you produced, then |
| The instance's staging folder |
| Tail of the launched app's stderr — read it when an app crashed or drew nothing. |
| Multi-instance control (see below). |
| Timing / sandbox size + effective vision backend. |
Every control tool takes instance=N and optional observe / settle_ms /
observe_region="x,y,w,h" (see below; the region crops the returned screenshot to the part you
care about — a fraction of the tokens of a full frame). Action tools add warnings when the
pointer is not where the server left it: a person or another agent is using the same instance.
In local mode OmniParser is lazy-loaded on first parse_screen (~6 s once, then ~2 s/parse),
after a GPU preflight (VRAM present and free).
Structured, honest responses
Every tool returns JSON: {"ok": true, "action": "click", "instance": 1, "detail": "clicked (10,20) button 1", …}.
Failures come back with the MCP isError flag set and the same JSON shape embedded in the error
text — {"ok": false, "error": "…", "fix": "…"} — where fix names the call that unblocks you
(e.g. a click on a dead instance says to run list_instances() / launch_app(); an unknown element
id says to re-run parse_screen(); a GPU that is too small says which mode to switch to). Silent
no-ops are treated as failures too: an invalid keysym, a zero scroll, an off-screen click, closing an
instance that isn't running, an app that exits on launch, or non-ASCII typing without xclip all
error instead of pretending success. Clamped waits are flagged (clamped / settle_clamped).
Image-returning calls (screenshot(), observe="screenshot"|"parse") return [json, image].
No subprocess can hang a call: xdotool/xdpyinfo/xclip run under timeouts and a frozen display is
reported as such; a missing binary is a structured error, never a raw exception.
Two more rules of the contract: an action that happened is never reported as failed because the
observation after it failed (click(observe="parse") whose parse hits a GPU error returns
ok: true, observed: false, observe_error: … — the agent must not click again), and the host
display is never driven: instance=0 (or whatever GLOVEBOX_HOST_DISPLAY is) is refused by every
tool. Calls that touch one instance's cursor, keyboard or clipboard are serialized per instance, so
two overlapping calls from one agent cannot interleave a mousemove with another call's click;
different instances run in parallel.
Vision backend (selectable, self-diagnosing)
GLOVEBOX_VISION env var, or the .vision-mode file, or default auto:
Mode |
| Needs | When |
| the best ready backend below ( | — | default |
| disabled (returns a note) — use | nothing (mcp, mss, xdotool) | lightest; let the agent's own vision do grounding |
| Tesseract OCR → text lines + coords (+ confidence) |
| no GPU; text-only grounding |
| OmniParser → text + icons + coords; with the optional ScreenParser detector → typed elements (button, link, tab, input, checkbox, …) | torch + CUDA + OmniParser weights + NVIDIA GPU ≥ 6 GB | best grounding |
In local mode the models are loaded by one shared vision-service process per machine
(vision_service.py), not by every agent's server: the first parse_screen starts it, parallel
agents queue on it (queued_ms in the result), and it exits after 15 idle minutes so a forgotten
service never pins the GPU. If it can't be reached the server loads the models in-process instead —
the service is an optimization, never a new failure mode. status() shows it under
vision.omniparser.service; python vision_service.py status|stop from the shell.
An explicitly requested basic/local that is not usable fails parse_screen with the fix named
(OmniParser not installed, no/too-small GPU, GPU out of memory, missing tesseract language data) —
it never silently falls back; only auto does, and it says so. Tesseract languages default to
slv+eng filtered to what is installed (GLOVEBOX_TESS_LANG overrides). Switch anytime with
./install.sh <mode> (installs only what that mode needs).
Multi-instance (a fleet of app windows)
Every control tool takes instance=N (default 1 = the start-display.sh sandbox). Spin up more —
each its own Xephyr display/window on the host desktop:
launch_app(command, name?, size?)→ starts the next free:Nrunning any GUI app (chromium,gimp,inkscape,xterm, …). Chromium-family browsers auto-get X11 flags, a per-instance profile, a remote-debugging port, and downloads routed tofiles/<N>/; other GTK apps get D-Bus isolation. Returns the instance id, size and any warnings.list_instances()·close_instance(n)·app_log(n).
A person can use the window too. Two things do not cross the nested-X-server boundary on their
own, and launch_app (and start-display.sh) fix both:
Keyboard layout — Xephyr copies the host's keysyms but files the AltGr key under plain
Alt, so on Slovenian, Croatian, German, … layouts AltGr+V arrives as Alt+v and@ { } [ ] | ~ €cannot be typed by hand (the agent'stype_textis unaffected — xdotool addresses keysyms). The host's complete keymap is copied into every instance (xkbcomp), verified, and re-copied by the bridge within 5 s if Xephyr reloads its own map or you switch the host layout. The launch result reportskeyboard: {method: "host-keymap", host_layout, verified}.GLOVEBOX_KEYBOARD=deforces a layout instead.Clipboard — each X server has its own; the per-instance bridge process mirrors text between the host and the instance clipboard (both directions, ~0.5 s,
CLIPBOARDonly, no images). On Wayland desktops the compositor already bridges the Xwayland clipboard to native apps, so copy on the host → paste in the sandbox and back just works.type_textrestores the previous clipboard after its unicode paste so agent text does not linger on your host clipboard. Disable withGLOVEBOX_CLIPBOARD_SYNC=0;list_instancesshowsclipboard_syncper instance. The bridge exits by itself when the instance closes.
Because each display has its own cursor, multiple agents can drive different instances in parallel —
one window each. Each agent's server process allocates display numbers under a cross-process lock,
so simultaneous launches don't collide. The GPU is shared through the vision service (one model
load, requests queue; see above). The host display for new windows is GLOVEBOX_HOST_DISPLAY
(default :0); XAUTHORITY is auto-discovered. Instances outlive the server process by default
(another agent or session can pick them up via list_instances); set GLOVEBOX_CLEANUP_ON_EXIT=1 to
close the ones a server launched when it exits.
One-call action + observe
click · click_element · type_text · press_keys · scroll · drag · double_click ·
upload_file · wait_for_stable take observe (none default · screenshot · parse) and
most take settle_ms. With observe="screenshot" the action returns its result and the
resulting screen in a single call — no separate screenshot round-trip. Use settle_ms when you know
the delay, wait_for_stable(observe="screenshot") when you don't. Default none keeps routine steps
cheap; opt into screenshot/parse on the steps that change the page (navigations, submits).
Files & uploads
Each instance gets a staging folder files/<N>/ inside the install dir — a stable place to drop
files for that instance (readable by native apps and, since it's under $HOME, by snap Chromium too).
list_files(instance) shows the folder and its contents, with sizes and a partial flag for
in-progress downloads.
Browser
<input type=file>→upload_file(path, instance)(via CDP;page_url_containspicks the tab when several are open). The nested file picker is invisible to automation and hangs snap Chromium, so never click an upload button expecting a dialog.Native apps (GIMP, Inkscape, editors) →
open_file(path, instance, app="gimp"), or just drive the app's own Open dialog — unlike the browser's, it's a real visible window you can type a path into (Ctrl+Lin a GTK file chooser).Saving / downloads → apps run as your user, so they can save anywhere you can write.
launch_appbrowser instances are pre-configured to download and "save as" intofiles/<N>/; point native apps' Save dialogs there too, thenlist_files(instance)to see the results.
Environment variables
Variable | Default | Meaning |
|
|
|
|
| host display where new Xephyr windows appear |
| off |
|
|
| per-instance host bridge (clipboard mirroring + keymap kept equal to the host); |
|
| seconds between clipboard polls in the bridge |
| unset | force a layout for new instances ( |
|
| tesseract language string for |
|
|
|
|
| the service's Unix socket |
|
| the service exits after this many idle minutes |
|
| upscale factor before OCR in |
|
| detector inside |
|
| instance 1's display (set by |
Maintenance (local mode)
The OmniParser lifecycle lives in vision.py (the server owns it; install.sh local just calls
it): python vision.py install-omniparser clones OmniParser,
downloads the v2 weights into OmniParser/weights/, and makes PaddleOCR optional (this uses easyocr).
python vision.py check reports readiness. transformers is pinned to 4.49.0 — newer releases
break Florence-2's remote config. If you upgrade OmniParser manually, re-run install-omniparser to
re-apply the patch.
Optional detector: ScreenParser. OmniParser's own detector only knows "interactable icon", so
every detected box is type: "icon". ScreenParser
(docling, ICML 2026, Apache-2.0, 153 MB) is a YOLO11-L trained on 1.45M screenshots with 55 element
classes; dropped into the same pipeline (easyocr text + Florence-2 captions) it gives parse_screen
typed elements — button, link, tab, input, checkbox, radio, switch, select,
slider, image, icon, … plus the raw class — and the paper reports fewer missed targets
(ScreenSpot action accuracy 77.8 → 80.1 with a fixed agent). Install it and it is used automatically:
python vision.py install-screenparser # downloads best.pt next to the OmniParser weights
python vision_service.py stop # a running vision service keeps the old detector until restartedMeasured on an RTX A5000 Laptop, 1440×900 Google results page: 110 elements vs. 93, same 1.7 s per
parse (OCR and captions dominate; the detector itself is 60 ms), 10 s model load. Layout containers
(window, toolbar, side bar, table, …) are not reported — they would swallow the text inside them.
GLOVEBOX_DETECTOR=omniparser|screenparser forces one; status() shows
vision.omniparser.detector.
Stop
pkill Xephyr # closes the sandbox (browser + WM + display)Safety
The agent's input and vision are scoped to the sandbox display — it does not see or control your real desktop.
The server process and the apps it launches run as your user (shell/file access, like any MCP server); only the GUI control is sandboxed to the Xephyr window. Files the agent saves, opens or uploads are your real files. For OS-level isolation from your files, run it inside a VM or container.
You can watch everything live and close it instantly with
pkill Xephyr.Automate responsibly — only sites and services you are authorized to use.
Files
server.py— the MCP server (all 30 tools).vision.py— vision backends: selection rules, GPU preflight, tesseract grouping, OmniParser lifecycle (check/install-omniparserCLI).vision_service.py— the shared OmniParser process (Unix socket, idle-exit,status/stopCLI)._paths.py— shared helpers: the 0700 per-user temp dir, display naming, the liveness probe.skills/glovebox/SKILL.md— a Claude Code skill that teaches an agent the workflow (./install.sh skillinstalls it).Launched apps are told to draw on the instance's X display (
XDG_SESSION_TYPE=x11,QT_QPA_PLATFORM=xcb), LibreOffice gets a per-instance profile (profiles/libreoffice-<N>) so it cannot hand a document to a LibreOffice on the host, andclose_instanceends the whole process group it started.a11y_focus.py— AT-SPI helper forfill()'s focus check; runs under the systempython3(needspython3-gi,gir1.2-atspi-2.0,at-spi2-core—install.shadds them; without themfillstill works and reportsfocus_checked: false).host_bridge.py— per-instance host bridge: host keymap copy (AltGr fix) + clipboard mirroring (keymap/serve/spawn/statusCLI).install.sh— mode-aware installer (none/basic/local).start-display.sh— launches the Xephyr sandbox (display + window manager + browser).AGENTS.md— drop-in tool-usage instructions for the AI agent (paste into its system prompt).ROADMAP.md— what is planned next (MCP SDK 2.x migration, …).mcp-config.json— a ready-to-paste MCP client config snippet.
Credits
local vision mode uses Microsoft's OmniParser
(cloned and weights downloaded by vision.py, under its own license). Screen capture uses
mss; input is driven with
xdotool. Not affiliated with Microsoft.
Contributing
Shipped as-is under MIT. Issues and PRs are welcome, but this is maintained by one person — no support or response time is guaranteed. If it's useful to you, a ⭐ helps.
License
MIT — see LICENSE.
Available Tools
30 toolsapp_logA
Tail of the stderr log of the apps launched on an instance (launch_app / start_app / open_file) — read it when an app crashed, drew nothing, or misbehaves.
Returns {"ok", "instance", "path", "lines": [...], "total_lines", "truncated"} (max 500 lines per call). The log is per instance: launch_app starts it fresh, start_app/open_file append. Instance 1 (start-display.sh) logs to /tmp/chromium-x1.log instead. Fails (isError) if no log exists for that instance yet.
Inputs: lines, instance. Returns: result
browserA
Talk to the instance's Chromium-family browser over the Chrome DevTools Protocol — exact navigation and page text without keystrokes or OCR.
action='info' → {"ok", "url", "title", "pages": [{url, title}, …], "viewport"} of the
active tab + all tabs. viewport = {dpr, left, top, css_width,
css_height} maps the page's CSS pixels (getBoundingClientRect) to the
pixels click()/screenshot() use: x = left + css_xdpr, y = top + css_ydpr
(dpr is not 1 on a zoomed page; top is the toolbar height)
action='navigate' → loads url in the active tab (waits for the load event, max 30 s);
returns {"ok", "url", "title", "loaded": bool}
action='text' → {"ok", "url", "title", "text", "length", "truncated"}: the rendered page
text (document.body.innerText), capped at max_chars (default 20000,
max 200000)
action='eval' → {"ok", "result"}: the JSON value of the JavaScript expression js
(e.g. "document.querySelector('h1').innerText"); a Promise is awaited.
A result whose JSON is longer than max_chars (default 100000, max
1000000) comes back as {"ok", "truncated": true, "length", "result_json":
the first max_chars characters} — raise max_chars or return less
(slice/map in the script). Errors in the page script come back as a
failure with the message
timeout_s (1–600, default 30) bounds an eval or text call. On a timeout the call fails
(isError, "timed_out": true) and the connection stays usable, but the script KEEPS RUNNING
in the page — for long work store the outcome on window (js="fetchAll().then(r => {
window.__r = r }); 'started'") and poll with short evals.
page_url_contains picks a tab by URL fragment (default: the most recently active page).
Works only for browsers started by launch_app/start-display.sh (CDP port 9222+N); fails
(isError) with the fix otherwise, or when the tab, URL or script is bad. The page state you
see in screenshots is this same tab.
Inputs: js, url, action, instance, +3 more. Returns: result
clickA
Click at absolute pixel coords (top-left origin) in an instance.
For anything with a label, find_elements('Save') -> click_element(id) beats guessing pixels.
Returns {"ok", "action", "instance", "detail", "hit"?, "warnings"?}. "hit" (Chromium-family
browsers only, read from the page's DOM just before the click) names what was under the
point: {"role": "checkbox", "name": "Remember me", "tag", "checked_before", "disabled",
"href", …} or {"role": "browser-ui"} for the toolbar/tab strip — compare it with what you
meant to click; it is absent for native apps. "warnings" appears when the pointer was not
where this server left it (a person or another agent is using the instance — re-screenshot).
button: 1=left, 2=middle, 3=right.
count: 1–3 clicks (2 = double-click, 3 = triple-click selects a line/paragraph).
modifiers: keys held during the click — 'ctrl' (open in new tab / multi-select), 'shift'
(range select), 'alt', 'super', combinable 'ctrl+shift'; released afterwards whatever happens.
observe='screenshot'|'parse' additionally returns the resulting screen state in the SAME
call (saves a round-trip); settle_ms (max 5000, "settle_clamped": true if you asked for
more) waits for the UI to update first — use 400–1500 after anything that navigates or
submits, or wait_for_stable() when you don't know how long. observe_region='x,y,w,h' crops
that screenshot to the part you care about (implies observe='screenshot'; far fewer tokens
than the full display; validated before the action). Fails (isError) if the instance
is down, the coordinates are outside the display or a modifier name is unknown (nothing is
clicked then). A failed observation after a successful click is reported as "observed":
false inside an ok result — the click is not repeated.
Inputs: x (required), y (required), count, button, +5 more.
click_elementA
Click an element by id from that instance's most recent parse_screen/find_elements (no coordinate guessing).
Returns {"ok", "action", "instance", "detail", "element", "center", "hit"?, "warnings"?}
("hit"/"warnings": see click()). Fails (isError) with the
fix named if the id is unknown — run parse_screen(instance) first. Ids go stale when the
screen changes: after navigation, re-parse before clicking. modifiers/observe/settle_ms:
see click().
Inputs: id (required), button, observe, instance, +3 more.
clipboardA
Read or set the instance's clipboard — the cheapest way to get EXACT text out of an app.
action='get' returns {"ok", "instance", "text", "length", "empty", "truncated"}: select text
in the app (click/drag, or press_keys('ctrl+a')) then press_keys('ctrl+c') and read it here
— no OCR, no screenshot. "empty": true means nothing (or non-text data) is on the clipboard;
text is capped at 100000 characters ("truncated": true). action='set' puts text on the
clipboard (then press_keys('ctrl+v') or 'shift+Insert' in terminals) and returns {"ok",
"instance", "length"}. Each instance has its own clipboard (separate X display); the bridge
started by launch_app mirrors TEXT between it and the host's clipboard within ~1 s (so a
person can copy/paste across — and whatever they copy on the host is readable here;
GLOVEBOX_CLIPBOARD_SYNC=0 turns it off). Fails (isError) if xclip is missing, the instance
is down, or xclip itself fails.
Inputs: text, action, instance. Returns: result
close_instanceA
Close an instance: kill its app + its Xephyr display, and verify it actually went down.
Returns {"ok", "instance", "detail"} only after the display stops responding. With
all_launched=true closes EVERY instance this server process launched (instance is ignored)
and returns {"ok", "closed": [...], "failed": [...]} — "ok" is false if any survived.
Fails (isError) if the instance isn't running, if it survives SIGTERM, or for instance 1 —
the main start-display.sh sandbox is protected (stop it from the host with pkill Xephyr).
Closing an instance another agent launched is allowed but visible to it (launched_here).
Inputs: instance, all_launched. Returns: result
double_clickA
Double-click at absolute pixel coords in an instance (e.g. open a file icon, select a word). Same as click(count=2).
Returns {"ok", "action", "instance", "detail", "hit"?, "warnings"?} — see click().
observe/settle_ms/observe_region: see click(). Fails (isError) if the instance is down
or the coordinates are off-screen.
Inputs: x (required), y (required), observe, instance, +2 more.
dragA
Drag (mouse-down → move → up) from (x1,y1) to (x2,y2) — for drawing, selecting, sliders.
Returns {"ok", "action", "instance", "detail", "steps"}. steps (default 10, 1–100) is the
number of intermediate motion events: canvases, sliders and drag-and-drop targets usually
need a few to register the drag (1 = jump straight to the end point). The button is released
even if a motion step fails (a stuck button would turn every later click into a drag).
observe/settle_ms: see click(). Fails (isError) if the instance is down or either point
is off-screen.
Inputs: x1 (required), x2 (required), y1 (required), y2 (required), +6 more.
fillA
Set a form field in ONE verified call: click it, check the focus, replace its content
with text, read the value back. Use it instead of click + type_text + screenshot for every
field of a form.
Target the field with x,y (absolute pixels) or element (an id from parse_screen/
find_elements). Steps and what each can tell you:
click, then a FOCUS CHECK before anything is typed — browsers through the page's DOM, native GTK/Qt/LibreOffice apps through the accessibility bus (AT-SPI). If the focus is not on an editable field, or is on a field whose rectangle does not contain the click (you clicked a label; the focus stayed where it was), the call FAILS with nothing typed and "focus" naming what has it. A password field is refused the same way unless allow_password=true — landing in one by accident is the case worth stopping.
the old content is replaced (select-all, then type; text="" empties the field);
the value is read back; submit=true presses Return only if nothing failed. Returns {"ok", "action", "instance", "detail", "method", "center", "focus_checked": "browser-dom"|"at-spi"|false, "focus"?: {role, name, rect}, "verified": true|false|null, "read_back": "browser-dom"|"clipboard"|"unreadable"|"skipped", "value"?, "click_effect"?: "changed"|"none", "suggest_element"?, "hit"?, "warnings"?}: focus_checked false — nobody could tell which widget had focus (canvas, game, Xlib-only app, iframe, or AT-SPI not installed: "focus_unchecked_reason"). The text was typed blind; "warnings" says so and "click_effect" reports whether the click changed any pixel near the target ("none" = probably a static label). Treat such a fill as UNCONFIRMED and screenshot. verified true — the focused field holds exactly
text(trailing newline ignored); verified false — it holds something else ("value", first 200 chars): the field reformats or limits input (dates, phone masks, maxlength); verified null — the value could not be read (native password field, canvas). Withelement, a typed detector (ScreenParser) warns when the element is a text label and names the nearest input ("suggest_element"). Also fails (isError) when the field is still empty after typing; repeating fill() is always safe (it replaces). Native read-back goes through select-all + copy (the clipboard is restored).observe/settle_ms/observe_region: see click().
Inputs: text (required), x, y, submit, +7 more.
find_elementsA
Parse the screen and return only the elements whose label contains query (case-insensitive
substring; empty query = all), optionally filtered by type (button, link, text, icon, …).
One call instead of parse_screen + scanning 300 elements. The ids are valid for
click_element(id) right away (this IS the most recent parse).
Returns {"ok", "instance", "vision", "query", "type", "count": total matches, "elements":
[{id, type, label, interactive, center}, …] (first limit, 1–100), "truncated"}. "count": 0
is an honest answer (the text is not on screen, or OCR did not read it — screenshot to check).
Fails (isError) like parse_screen; in vision mode 'none' it fails with the fix (no grounding
backend: screenshot and reason yourself).
Inputs: query (required), type, limit, instance. Returns: result
get_screen_sizeA
Screen size of an instance + the active vision backend and server version.
Returns {"ok", "instance", "width", "height", "vision", "version"}. Call it once before coordinate math — coordinates are absolute pixels, top-left origin, within THIS instance's display (0 <= x < width, 0 <= y < height; out-of-range coordinates are rejected). "vision" is the EFFECTIVE grounding backend: 'basic'/'local' → parse_screen + click_element; 'none' → screenshot + reason about pixels yourself. Fails (isError) if the instance is down.
Inputs: instance. Returns: result
launch_appA
Launch a GUI app in its OWN new Xephyr display/window and return its instance id.
command is any shell command ('chromium', 'xterm', 'gimp', 'inkscape file.svg', …).
Chromium-family browsers (chromium, google-chrome, brave, microsoft-edge, vivaldi, opera)
automatically get X11 flags, a per-instance profile, downloads routed to files//, and a CDP
debug port (needed by upload_file and browser()). Other GTK apps get D-Bus isolation so they
render on this display, not the host.
Returns {"ok", "instance", "display", "name", "files_dir", "width", "height", "warnings": [...],
"keyboard": {method, host_layout, verified}, "clipboard_sync": bool} — pass that instance id to every
control tool. "keyboard" says how the window got the host's keyboard layout ("host-keymap" =
exact copy, verified against the host; Xephyr alone maps AltGr as plain Alt, so a person could
not type @ { } [ ] on Slovenian/German/… layouts); "clipboard_sync" is true when the per-instance
host bridge runs (it mirrors text between the host and instance clipboards and keeps the
keymap equal to the host's; GLOVEBOX_CLIPBOARD_SYNC=0 disables it — note that whatever the
person copies on the host then becomes readable inside the sandbox). Neither affects the
agent's own type_text/clipboard. Fails (isError) BEFORE anything starts if the command is
empty/unparsable or size is not WxH; after that if Xephyr can't start on the host display
(its log tail is quoted), the command doesn't exist, or the app exits immediately (its stderr
tail is quoted; app_log(instance) has more) — the display is closed again in those cases.
"warnings" lists what degraded: window manager (openbox) missing (Xephyr then ignores pointer
motion, so clicks would land in the screen center), keymap copy or clipboard bridge failed.
Each instance has its own cursor, so separate agents can drive separate instances in parallel;
display numbers are allocated under a cross-process lock, so parallel launches don't collide.
Never share one instance between agents. The window appears on GLOVEBOX_HOST_DISPLAY
(default :0). Give the app a moment to draw (wait_for_stable / settle_ms) before the first
screenshot. Instances outlive this server process unless GLOVEBOX_CLEANUP_ON_EXIT=1.
Inputs: command (required), name, size. Returns: result
list_filesA
The instance's staging folder (files// under the install dir) and its contents.
Returns {"ok", "instance", "dir", "count", "truncated", "files": [{name, bytes, modified, is_dir, partial}, …]} (capped at 200 entries, newest first). "partial": true marks an in-progress browser download (.crdownload/.part/.tmp) — wait for it to disappear before using the file. Drop files there (or reference any host path you can read) to open in apps via open_file() or an app's Open dialog; launch_app browser instances also download / 'save as' into it — call this after a download to confirm the file arrived. Browser uploads still go through upload_file(). Fails (isError) if the folder cannot be read.
Inputs: instance. Returns: result
list_instancesA
List all running instances (app windows): id, display, name, command, title, launched_here.
Returns {"ok", "count", "instances": [{instance, display, name, command, up, title, launched_here, clipboard_sync}, …]}. "title" is the active window's title (a browser's follows the page) — the way to tell instances apart that another agent's server launched (their name/command show as unknown, launched_here=false). Probes displays :1–:24 for sandboxes started outside this server (start-display.sh, other agents). Instances whose window was closed by hand disappear from the list and their number is reused.
Inputs: no named parameters. Returns: result
manage_windowA
Act on a window of an instance: action='activate' (focus/raise — switch between an app's windows or dialogs), 'close' (asks the app to close it — a save prompt may appear), 'minimize', 'maximize', 'restore' (un-maximize), 'move' (to x,y), 'resize' (to width x height). window_id=0 means the active window (window_info lists ids).
Returns {"ok", "action", "instance", "window", "detail"} plus observe/settle_ms like
click(). Fails (isError) if the window id is unknown, the action is unknown, move/resize
sizes are missing, or the instance is down. maximize/restore need wmctrl (status().deps).
Inputs: action (required), x, y, width, +6 more.
move_mouseA
Move the mouse pointer to absolute pixel coords in an instance (no click).
Returns {"ok", "action", "instance", "detail"}. Useful for hover states (menus, tooltips) — follow with screenshot() to see the effect. Fails (isError) if the instance is down or the coordinates are off-screen.
Inputs: x (required), y (required), instance. Returns: result
open_fileA
Open a local file in an app ON an existing instance's display (NOT a new instance).
If app is given, runs <app> <filepath> there (e.g. app='gimp'); otherwise xdg-open picks
the default handler. Relative paths resolve against the instance's staging folder files//
(see list_files). Returns {"ok", "action", "instance", "opened", "app"}; fails (isError) if
the file or the app doesn't exist, or if the handler exits immediately (e.g. xdg-open finds
no application for that file type — then pass app=). The app is started asynchronously —
wait_for_stable()/screenshot after a moment to confirm it drew. For BROWSER
uploads use upload_file() instead. Both paths get the same X11 + D-Bus handling as launch_app
(so the app renders on this display, not in an already-running window on the host).
Inputs: filepath (required), app, instance. Returns: result
parse_screenA
Detect on-screen elements and return them with ids + pixel-centers (vision grounding).
Returns {"ok", "vision", "screen": [W,H], "annotated", "count", "truncated",
"elements": [{id, type, label, interactive, center: [x,y]}, …]} — then click_element(id).
Also saves a numbered Set-of-Mark image (path in "annotated").
Backend = the effective GLOVEBOX_VISION: 'local' (OmniParser on the GPU, ~2 s after a ~10 s
first-call model load; served by ONE shared vision-service process per machine so parallel
agents don't each load the models — "via": "service"|"in-process", "queued_ms" says how long
the GPU queue was, "detector" which detector answered, "fallback_reason" why the service was
not used. "type" is "text"|"icon" with OmniParser's own detector; with the optional
ScreenParser detector (status().vision.omniparser.detector) it is one of text, button, link,
tab, input, checkbox, radio, switch, select, slider, stepper, rating, icon, image, item,
indicator and each element also carries "class", the detector's raw class name; buttons/tabs/
links carry their visible text as "label", icons/images a short caption. box_threshold
raises the detector's confidence floor to cut noise), 'basic' (tesseract, text only;
granularity='line' merges words on one line into one target like "Sign in" — default,
split at wide gaps so a menu bar is one target per item — or 'word' for single words; the
frame is OCR'd at 2× (GLOVEBOX_OCR_SCALE) because UI text is too small for tesseract at 1×;
each element also carries "confidence" 0–100), 'none' (returns a note telling you to
screenshot + reason about pixels yourself — this is normal, not an error; status() says why
if 'auto' fell back to it). Fails (isError) with the fix named when an explicitly requested
backend is not usable (OmniParser not installed, no/too-small GPU, GPU out of memory, the
vision service busy, tesseract missing its language data). Elements are capped at 300 per
call ("truncated": true when more were detected) — use find_elements to search instead of
paging. Ids are only valid until the screen changes — re-parse after navigation.
Inputs: instance, granularity, box_threshold. Returns: result
press_keysA
Press a key, a combo, or a SEQUENCE (xdotool keysym syntax, space-separated for sequences): 'Return', 'ctrl+a', 'Tab', 'ctrl+t', 'F5', 'ctrl+a Delete', 'alt+F4', 'shift+Insert'.
Returns {"ok", "action", "instance", "detail", "keys": [...]}. Every token is validated
against the X keysym table BEFORE anything is pressed, so an invalid name fails with nothing
pressed — also in the middle of a sequence. Fails (isError) on a dead instance or a frozen
display. Browser navigation: 'ctrl+l' → type_text(url) → 'Return' (or browser(action=
'navigate')). observe/settle_ms: see click().
Inputs: keys (required), observe, instance, settle_ms, +1 more.
put_fileA
Write a file into the instance's staging folder files// — the way to hand the sandbox a file you produced (a CSV to import, an SVG to open, a text to upload) without shell access. Then open_file(name) / upload_file(name) / an app's Open dialog can use it by that name.
name is a relative path inside the staging folder ('data.csv', 'in/logo.svg'; parent
folders are created); absolute paths and '..' are refused — nothing outside files// can be
written. content is text (written as UTF-8) or, with encoding='base64', binary data.
Returns {"ok", "instance", "path", "name", "bytes", "overwritten"}. Fails (isError), writing
nothing, if the file exists and overwrite is false, the name escapes the folder or names a
directory, the base64 is invalid, or the content exceeds 10 MB. The instance does not have
to be running (stage files first, launch after).
Inputs: name (required), content (required), encoding, instance, +1 more. Returns: result
screenshotA
Screenshot of an instance's display — the ONLY way to see the screen; always look before you act.
Returns [json, image]: {"ok", "instance", "width", "height", "region", "scale", "format", "bytes"} plus the image. Options to cut payload or zoom: region="x,y,w,h" crop to a rectangle (zoom into small text/icons; coordinates stay absolute — add x,y back when you click), default = full display; scale=0.5 downscale (0.1–1.0) for cheap overview shots; format="jpeg" smaller than PNG for photos/video, quality 1–95 (default png, lossless). For an element list with coordinates use parse_screen (basic/local vision). Prefer observe='screenshot' on action tools to act and see the result in one call. Fails (isError) if the instance is down, the region is outside the display, or the format is unknown.
Inputs: scale, format, region, quality, +1 more.
scrollA
Scroll the wheel. Say where: direction='down'|'up'|'left'|'right' with amount notches
(sign ignored) — or the signed form: positive = up (or left), negative = down (or right).
Returns {"ok", "action", "instance", "detail", "applied", "clamped", "axis", "direction"}. One unit ≈ one wheel
notch (a few lines); at most 100 notches per call ("clamped": true when you asked for more,
"applied" says how many were sent). Scrolling targets the window under the pointer — pass
x,y to move there first (both >= 0), otherwise the current pointer position is used.
axis='horizontal' scrolls sideways (wheel buttons 6/7). Fails (isError) on amount=0 (a
no-op), a dead instance, off-screen x,y or an unknown axis. observe/settle_ms: see click().
Inputs: x, y, axis, amount, +5 more.
start_appA
Start ANOTHER app on an existing instance's display (same window/cursor) — e.g. a second tool next to the browser, or a restart of an app that quit. For a new, separate window use launch_app; to open a file in an app use open_file.
Returns {"ok", "instance", "command", "pid"}; fails (isError) if the instance is down, the command is unparsable/missing, or the app exits immediately (stderr tail quoted; app_log(instance) has more — logs of apps started here are appended to it). Browsers get the same X11/profile/CDP handling as launch_app. The app starts asynchronously: wait_for_stable() or screenshot() after a moment to confirm it drew.
Inputs: command (required), instance. Returns: result
statusA
Server + sandbox status in one read-only call — run this first, and when filing bug reports.
Returns {"ok", "version", "host_display", "tmp_dir", "cleanup_on_exit", "vision": {requested, effective, ready, reason, fallback, gpu: {present, name, vram_total_mb, vram_free_mb, enough, note}, omniparser: {code, weights, torch, ready, loadable, reason, detector: {requested, effective, screenparser_weights, reason}, loaded, service: {running, pid, loaded, detector, served, in_flight, idle_s, …} — the shared per-machine OmniParser process (one GPU load for all agents; starts on first use, idle-exits)}, tesseract: {binary, pytesseract, langs, lang, ready, reason}}, "instances": [...same as list_instances], "keyboard": {host_layout, override}, "clipboard_sync": {enabled, poll_s, note}, "deps": {xdotool, xclip, xephyr, openbox, wmctrl, xdpyinfo, xkbcomp, setxkbmap, dbus_run_session, tesseract, nvidia_smi, xprop, atspi: true|false}, "notes": [...], "tips": [...]} ("tips": the tools that save the most round-trips on this setup — read them once). "vision.effective" is the grounding backend actually in use (none|basic|local), re-resolved every 20 s (installing OmniParser takes effect without a restart); when GLOVEBOX_VISION=auto it is the best READY one and "reason" says why (e.g. OmniParser weights missing, GPU too small). "notes" lists things worth knowing (e.g. the service runs a different detector than this server would pick). No side effects.
Inputs: no named parameters. Returns: result
type_textA
Type into the focused field — unicode-safe. Click the field first to focus it.
To set a form field in one verified call use fill() instead (click + replace + read back).
Returns {"ok", "action", "instance", "detail", "method": "xdotool"|"clipboard", "warnings"?}.
clear=true selects the field's content first (ctrl+a) so the text REPLACES it;
submit=true presses Return afterwards — one call for "fill and submit".
Pure ASCII is typed via xdotool (timeout scales with length); text with any non-ASCII char
(č/š/ž …) is inserted via the clipboard + ctrl+v, because xdotool's synthetic unicode
keystrokes get silently dropped by some toolkits (e.g. Inkscape's GTK canvas) even though
they work in browsers. The clipboard is put back as it was afterwards (previous text, or
cleared), best-effort: an app that reads the selection late (> 0.3 s) can paste the restored
text instead, and the host bridge may have mirrored the typed text in that window — see
clipboard(). Terminals that paste with shift+Insert instead of ctrl+v need press_keys after
clipboard(action='set'). Fails (isError) rather than silently losing characters if xclip is
missing for non-ASCII text. observe/settle_ms: see click().
Inputs: text (required), clear, submit, observe, +3 more.
upload_fileA
Attach a local file to a page's via Chrome DevTools Protocol — use this for ALL browser uploads (logos, images, docs).
The sandbox's native GTK file picker is invisible to us AND hangs the renderer, so never click
an upload button expecting a dialog — call this instead. Works on Chromium-family browsers
started by launch_app / start-display.sh (they open a per-instance --remote-debugging-port,
9222+N). selector targets the file input on the MAIN page (default = first file input).
With several tabs open the most recently active page target is used unless page_url_contains
picks the tab by a URL fragment. If the site only inserts the after you click its
'upload' control, click that first (it won't open a dialog we can see, but it wires up the
input), then call this. Relative paths resolve against files//.
Returns {"ok", "action", "instance", "detail", "page"}; fails (isError) with the reason if the
file is missing, no CDP target answers, or the selector matches nothing.
NATIVE (non-browser) apps: use open_file() or drive the app's own Open dialog — it IS visible
here. observe/settle_ms: see click().
Inputs: filepath (required), observe, instance, selector, +3 more.
wait_for_stableA
Wait until the screen stops changing (page loaded, app drew, animation ended) — replaces guessing a settle_ms.
Samples the display every interval_ms (100–2000) and returns as soon as two consecutive
frames differ by at most threshold (fraction of pixels, default 0.2%), or after timeout_ms
(max 30000). Returns {"ok", "instance", "stable", "elapsed_ms", "frames", "change": last
changed fraction, "timed_out"}. "stable": false with "timed_out": true means the screen was
still changing (video, spinner, blinking cursor — raise the threshold to ~0.01 for those); it
is reported, not hidden. observe='screenshot'|'parse' returns the final state in the same
call. Fails (isError) if the instance is down.
Inputs: observe, instance, threshold, timeout_ms, +2 more.
wait_for_textA
Wait until text appears on screen (or, appear=false, disappears) — by OCR/parse polling.
The honest replacement for "sleep and hope" after navigation or a submit: returns as soon as
the text is (not) there, or reports "timed_out": true.
Returns {"ok", "instance", "found": bool, "satisfied": bool, "elapsed_ms", "polls", "element": {id, type, label, center} | null, "timed_out"}. "satisfied" is what you waited for; "timed_out": true with satisfied: false is reported, not hidden — screenshot to see what IS there. timeout_ms max 60000, interval_ms 300–5000. Fails (isError) if the instance is down or there is no grounding backend (vision mode 'none').
Inputs: text (required), appear, instance, timeout_ms, +1 more. Returns: result
wait_msA
Wait (e.g. for a page to load or an app to draw) — capped at 10000 ms.
Returns {"ok", "action", "requested_ms", "waited_ms", "clamped"} — "clamped": true means you asked for more than the cap and only waited_ms elapsed. Prefer wait_for_stable() (waits until the screen stops changing), wait_for_text(), or settle_ms on the action itself when you also want to observe the result in the same call.
Inputs: ms (required). Returns: result
window_infoA
The active window's title + geometry, the visible windows and the pointer position on an instance — a cheap, image-free way to confirm navigation (browser titles follow the page) or that an app opened.
Returns {"ok", "instance", "active": {id, title, x, y, width, height, pid} | null, "windows": [{id, title}, …] (visible, capped at 50, unnamed ones included with title ""), "count", "truncated", "cursor": [x, y]}. "active" is null when no window has focus yet (fresh instance) — that is reported, not an error. Window ids are what manage_window() takes. Fails (isError) if the instance is down or xdotool is missing/fails.
Inputs: instance. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 30 tools
Most tools target clearly distinct operations; launch_app/start_app/open_file and click/click_element/double_click are separated by precise descriptions. Minor overlap remains between fill and type_text, parse_screen and find_elements, and double_click as click(count=2), but the descriptions make the intended choice clear.
All tool names use snake_case with no camelCase mixing, and most follow a readable verb_noun pattern. A few bare nouns (status, browser, clipboard, screenshot) and mixed get_/info patterns are minor deviations rather than inconsistency.
At 30 tools the surface is heavier than ideal, and several conveniences such as double_click, find_elements, and wait_ms could be folded into existing tools. The broad GUI/browser/vision/file scope explains many tools, but the count is still well above the recommended range.
The set covers the full lifecycle well: launch/start/close/list instances, mouse and keyboard control, screenshots and vision parsing, browser CDP actions, window management, clipboard, and file staging/upload. Minor gaps exist around direct file deletion/rename and some file-management operations, but agents can generally work around them via overwrite or app dialogs.
Maintenance
Related MCP Connectors
Secure Linux sandboxes for AI agents, as MCP tools.
Remote MCP server exposing your Node Command fleet (files, shell, processes, browser) to AI hosts.
Official Browserless MCP server: browser automation, scraping, screenshots and PDFs for AI agents.
MCP server for your apps' tools and custom tools, plus hosted AI agents and approval-gated workflows
Related MCP Servers
- AlicenseBqualityDmaintenanceAn MCP server that provides AI agents with a full Ubuntu desktop environment inside Docker, enabling them to perform complex computer tasks like browsing, coding, testing, and GUI automation.3613MIT
- FlicenseNot gradedqualityDmaintenanceAn MCP server that gives AI clients controlled access to a real headless browser with domain allowlisting, SSRF protection, per-session isolation, audit logging, and persistent cookies/state across restarts.-
- AlicenseAqualityDmaintenanceMCP server enabling AI agents to control a real Linux browser with live view, human takeover, and safety guardrails.131MIT
- AlicenseAqualityBmaintenanceAn MCP server that gives AI agents local Linux desktop control through tools for screenshots, accessibility, mouse, keyboard, and reliable text input.36MIT