Skip to main content
Glama

glovebox-mcp

A sandboxed computer-use MCP server — let an AI agent drive a real browser and desktop apps (mouse, keyboard, screenshots, clipboard, vision grounding), confined to a nested X11 window so its input and vision never touch your real screen or the windows of your other apps.

Like a lab glovebox: the agent reaches in and manipulates real applications, sealed off from your desktop. Bring the sandbox up, log into whatever sites or apps you want to automate inside that window, and the agent operates only there — you can watch it live and close it instantly.

Speaks the Model Context Protocol, so it works with MCP clients like Claude Code. Your host can run Wayland; the sandbox gives the agent a real X server to drive.

glovebox-mcp — an agent filling a sign-up form inside the sandbox

An agent driving a real browser in the sandbox — gliding the cursor, inserting a unicode name (Nadja Kovačič), typing, and submitting. All confined to a nested X11 window.

Why a nested X11 sandbox?

  • Most desktop automation (xdotool, PyAutoGUI) is X11-only, but many modern desktops run Wayland.

  • Xephyr provides a real X server inside a single window (DISPLAY :1). Everything the agent does — clicks, typing, screenshots — is confined to that window, not your real desktop.

  • You stay in control: watch it live, pkill Xephyr to close everything.

The isolation is display-level (see Safety): the apps themselves run as your user with your filesystem. For OS-level isolation run glovebox inside a VM or container.

Related MCP server: Secure Browser MCP

Requirements

  • Linux — the sandbox nests a real X server (Xephyr), so it works even on Wayland hosts (via Xwayland). Not macOS/Windows. Developed on Ubuntu; any modern Linux with the packages below.

  • Python 3.10+ and uv (used for the virtualenv).

  • System packages — xserver-xephyr (Xephyr), openbox, scrot, x11-utils, x11-xkb-utils (xkbcomp: gives the sandbox your keyboard layout), xdotool, wmctrl, xclip, optional at-spi2-core + gir1.2-atspi-2.0 + python3-gi (focus check in native apps) (+ tesseract-ocr for basic). On Debian/Ubuntu the installer auto-installs them via apt (sudo); on Fedora/Arch it prints the matching dnf/pacman command. The MCP server itself is distro-agnostic — any Linux with these tools works. openbox is not optional in practice: without a window manager Xephyr ignores synthetic pointer motion, so clicks would land in the screen center (launch_app warns when it is missing).

  • A browser in the sandbox (Chromium, Chrome, Brave, Edge, Vivaldi or Opera).

  • NVIDIA GPU (≥6 GB VRAM) — only for the local vision mode. The server checks this itself (status() → vision.gpu) and auto falls back to basic/none when it is not met.

Install

One command. The installer probes the GPU, installs the system packages (auto via apt on Debian/Ubuntu; it prints the line for dnf/pacman), creates the Python env from pyproject.toml, downloads the vision models the machine can run, writes mcp-config.json and .mcp.json (Claude Code) with your paths, and installs the Claude Code skill:

git clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh

What auto picks: an NVIDIA GPU with ≥ 6 GB VRAM → local (OmniParser + ScreenParser, typed elements; ~4 GB of downloads, --no-screenparser skips the 153 MB detector); otherwise basic (Tesseract OCR, CPU). Tesseract is installed in every mode — the server's own auto backend selection falls back to it when the GPU is busy. Force a mode with ./install.sh none|basic|local; ./install.sh skill installs only the skill; --no-skill, --dev (pytest) are the other flags. Re-running is safe: it only downloads what is missing.

The requested mode is written to .vision-mode (auto unless you forced one; override per run with the GLOVEBOX_VISION env var). python vision.py check prints the same diagnosis status() returns (GPU, weights, detector, tesseract).

Claude Code skill

The installer copies skills/glovebox/SKILL.md into ~/.claude/skills/glovebox/ when ~/.claude exists (./install.sh skill does only that) — a skill that teaches Claude Code the observe → act → verify loop, grounding per vision mode, the upload and navigation gotchas, and when to stop. Claude Code loads it automatically when an agent is asked to drive a GUI app (or explicitly with /glovebox). AGENTS.md remains the long form for any other agent/harness.

Works with any MCP client / harness

Claude Code, Cursor, Codex, or your own agent — it's a standard MCP server, not tied to any one host. Two compatibility notes: basic/local return element coordinates as text, so they work even with text-only agents; none relies on the client passing the tool's screenshots to a multimodal model (fine for Claude Code, Cursor, and other image-capable MCP clients). In any mode, clipboard() gets exact text out of an app without vision at all.

Quickstart

  1. Start the sandbox (leave it running):

    ./start-display.sh              # 1440×900 Xephyr window with a browser
    ./start-display.sh 1920x1080    # …or pass a screen size (or set $RES)

    Log into any sites or apps you want to automate in that window.

  2. Register the server with your MCP client. install.sh already wrote mcp-config.json with your real install path — copy its glovebox block into your client's MCP config:

    { "mcpServers": { "glovebox": {
        "command": "/abs/path/to/glovebox-mcp/.venv/bin/python",   // filled in by install.sh
        "args":    ["/abs/path/to/glovebox-mcp/server.py"],
        "env":     { "DISPLAY": ":1" }
    } } }

    Restart the client so it loads the server.

  3. Ask the agent to screenshot / click / type — it operates only on the :1 window.

Driving it with an AI agent? Paste AGENTS.md into the agent's system prompt — it teaches the observe → act → verify loop, grounding, the upload/unicode gotchas, and when to stop.

Tools (30)

Tool

What

status()

Server + sandbox status in one read-only call: version, the vision diagnosis (requested vs. effective backend and why, GPU name/VRAM, OmniParser code/weights, tesseract languages), live instances, and which system deps are present. Run it first — and paste it into bug reports.

screenshot(instance, region?, scale?, format?)

[json, image]. region="x,y,w,h" zooms into a rectangle, scale=0.5 shrinks, format="jpeg" compresses — cut payload or read small text.

parse_screen(instance, granularity?)

Vision grounding → JSON of detected elements (id, type, label, interactive, pixel-center; capped at 300 per call, flagged via "truncated") + a numbered Set-of-Mark image. basic merges OCR words into lines ("Sign in" = one target; granularity="word" for single words).

find_elements(query, type?)

Parse + filter in one call: the elements whose label contains query (and optionally of one type), ids ready for click_element. count: 0 is an honest "not on screen".

wait_for_text(text, timeout_ms?, appear?)

Poll OCR/parse until a text appears (or disappears) — the honest replacement for "sleep and hope" after navigation; timed_out is reported, never hidden.

fill(text, x, y | element=id, submit?, verify?, allow_password?)

Set a form field in one verified call: click it, check the focus before typing, replace its content, read the value back. The focus check runs through the page's DOM in browsers and through the accessibility bus (AT-SPI) in native GTK/Qt/LibreOffice apps: a click that landed on a label (the focus stayed elsewhere), a non-editable widget, or a password field you did not announce (allow_password=true) fails with nothing typed and focus naming what has it. verified: true = the field holds exactly the text; false + value = it reformatted or cut it; null = unreadable. Where nobody can tell the focus (canvas, games, Xlib-only apps) the reply says focus_checked: false, a warning, and click_effect (did any pixel near the click change) — never a clean success.

click_element(id, modifiers?)

Click an element from the last parse_screen/find_elements (no coordinate guessing).

click(x,y, count?, modifiers?) · double_click · move_mouse · scroll(amount, direction?, x?, y?, axis?) · drag(…, steps?)

Pointer ops. In Chromium-family browsers a click returns hit — the element under the point, read from the DOM just before the click ({"role": "checkbox", "name": "Remember me", "checked_before": false}), so a misclick is visible at once. scroll(3, direction="down") or the signed form (negative = down). Coordinates outside the display are rejected (xdotool would silently clamp them). count=2/3 double/triple-clicks, modifiers="ctrl+shift" holds keys (released whatever happens). scroll can move to x,y first, scroll horizontal, and reports applied/clamped (max 100 notches). drag emits intermediate motion events so canvases and sliders register it, and releases the button even if a step fails.

type_text(text, clear?, submit?)

Unicode-safe typing (ASCII via xdotool; anything with č/š/ž… is inserted via the clipboard, because xdotool's synthetic unicode is silently dropped by some GTK apps; the clipboard is put back afterwards). clear=true replaces the field's content, submit=true presses Return — one call to fill and submit.

press_keys("ctrl+a" / "Return" / "ctrl+a Delete")

Keys, combos, and space-separated sequences (xdotool syntax). Every token is validated against the X keysym table before anything is pressed — an invalid name anywhere in a sequence errors with nothing pressed.

browser(action="info"|"navigate"|"text"|"eval")

The instance's Chromium over CDP: current URL/title and tabs, exact navigation (waits for load), the rendered page text without OCR, or a JavaScript expression's value (Promises awaited; timeout_s 1–600, default 30; results over max_chars come back truncated and flagged). info also returns viewport (dpr, left, top) to turn the page's CSS pixels into click coordinates.

manage_window(action, window_id?)

activate / close / minimize / maximize / restore / move / resize a window (ids from window_info; 0 = the active one) — switch between an app's windows and dialogs.

start_app(command)

Start another app on an existing instance (same window/cursor) — a second tool, or a restart.

clipboard(action="get"|"set", text?)

Read the instance's clipboard (select + ctrl+c, then read exact text — no OCR) or set it (then ctrl+v). Per-instance; text is mirrored to/from the host clipboard by the per-instance bridge, so a person can copy-paste across.

wait_for_stable(instance, timeout_ms?, threshold?)

Wait until the screen stops changing (page loaded, app drew) instead of guessing a delay. Reports stable/timed_out honestly.

window_info(instance)

Active window title + geometry, the visible windows (ids for manage_window) and the pointer position — confirm navigation or that an app opened, without an image.

upload_file(filepath, selector?, page_url_contains?)

Attach a local file to a page's <input type=file> via the Chrome DevTools Protocol. The nested X11 file picker is invisible to automation and hangs the renderer, so use this for all uploads — never click an upload button expecting a dialog. Works on Chromium-family browsers started by launch_app/start-display.sh (per-instance --remote-debugging-port, 9222+N). Browser file inputs only — for native apps see open_file.

open_file(filepath, app?)

Open a local file in a native app on the instance's display (e.g. app="gimp") or via xdg-open. Fails honestly if the handler exits immediately (no app registered for that type).

put_file(name, content, encoding?, overwrite?)

Write a file (text, or binary as base64; max 10 MB) into the instance's staging folder — hand the sandbox something you produced, then open_file/upload_file it by name. Only inside files/<N>/: absolute paths, .. and symlinks out are refused; an existing file needs overwrite=true.

list_files()

The instance's staging folder files/<N>/ + entries with size, mtime and a partial flag for in-progress downloads.

app_log(instance, lines?)

Tail of the launched app's stderr — read it when an app crashed or drew nothing.

launch_app(command, name?, size?) · list_instances() · close_instance(n | all_launched=true)

Multi-instance control (see below). launch_app also makes the window usable by a person: the host keyboard layout is copied in (keyboard) and the clipboard bridge started (clipboard_sync). list_instances shows each window's title; instances closed by hand disappear and their number is reused.

wait_ms(ms) · get_screen_size()

Timing / sandbox size + effective vision backend.

Every control tool takes instance=N and optional observe / settle_ms / observe_region="x,y,w,h" (see below; the region crops the returned screenshot to the part you care about — a fraction of the tokens of a full frame). Action tools add warnings when the pointer is not where the server left it: a person or another agent is using the same instance. In local mode OmniParser is lazy-loaded on first parse_screen (~6 s once, then ~2 s/parse), after a GPU preflight (VRAM present and free).

Structured, honest responses

Every tool returns JSON: {"ok": true, "action": "click", "instance": 1, "detail": "clicked (10,20) button 1", …}. Failures come back with the MCP isError flag set and the same JSON shape embedded in the error text — {"ok": false, "error": "…", "fix": "…"} — where fix names the call that unblocks you (e.g. a click on a dead instance says to run list_instances() / launch_app(); an unknown element id says to re-run parse_screen(); a GPU that is too small says which mode to switch to). Silent no-ops are treated as failures too: an invalid keysym, a zero scroll, an off-screen click, closing an instance that isn't running, an app that exits on launch, or non-ASCII typing without xclip all error instead of pretending success. Clamped waits are flagged (clamped / settle_clamped). Image-returning calls (screenshot(), observe="screenshot"|"parse") return [json, image]. No subprocess can hang a call: xdotool/xdpyinfo/xclip run under timeouts and a frozen display is reported as such; a missing binary is a structured error, never a raw exception.

Two more rules of the contract: an action that happened is never reported as failed because the observation after it failed (click(observe="parse") whose parse hits a GPU error returns ok: true, observed: false, observe_error: … — the agent must not click again), and the host display is never driven: instance=0 (or whatever GLOVEBOX_HOST_DISPLAY is) is refused by every tool. Calls that touch one instance's cursor, keyboard or clipboard are serialized per instance, so two overlapping calls from one agent cannot interleave a mousemove with another call's click; different instances run in parallel.

Vision backend (selectable, self-diagnosing)

GLOVEBOX_VISION env var, or the .vision-mode file, or default auto:

Mode

parse_screen

Needs

When

auto

the best ready backend below (local → basic → none); status() says which and why

—

default

none

disabled (returns a note) — use screenshot() + reason, clipboard() for exact text

nothing (mcp, mss, xdotool)

lightest; let the agent's own vision do grounding

basic

Tesseract OCR → text lines + coords (+ confidence)

tesseract-ocr + pytesseract

no GPU; text-only grounding

local

OmniParser → text + icons + coords; with the optional ScreenParser detector → typed elements (button, link, tab, input, checkbox, …)

torch + CUDA + OmniParser weights + NVIDIA GPU ≥ 6 GB

best grounding

In local mode the models are loaded by one shared vision-service process per machine (vision_service.py), not by every agent's server: the first parse_screen starts it, parallel agents queue on it (queued_ms in the result), and it exits after 15 idle minutes so a forgotten service never pins the GPU. If it can't be reached the server loads the models in-process instead — the service is an optimization, never a new failure mode. status() shows it under vision.omniparser.service; python vision_service.py status|stop from the shell.

An explicitly requested basic/local that is not usable fails parse_screen with the fix named (OmniParser not installed, no/too-small GPU, GPU out of memory, missing tesseract language data) — it never silently falls back; only auto does, and it says so. Tesseract languages default to slv+eng filtered to what is installed (GLOVEBOX_TESS_LANG overrides). Switch anytime with ./install.sh <mode> (installs only what that mode needs).

Multi-instance (a fleet of app windows)

Every control tool takes instance=N (default 1 = the start-display.sh sandbox). Spin up more — each its own Xephyr display/window on the host desktop:

  • launch_app(command, name?, size?) → starts the next free :N running any GUI app (chromium, gimp, inkscape, xterm, …). Chromium-family browsers auto-get X11 flags, a per-instance profile, a remote-debugging port, and downloads routed to files/<N>/; other GTK apps get D-Bus isolation. Returns the instance id, size and any warnings.

  • list_instances() · close_instance(n) · app_log(n).

A person can use the window too. Two things do not cross the nested-X-server boundary on their own, and launch_app (and start-display.sh) fix both:

  • Keyboard layout — Xephyr copies the host's keysyms but files the AltGr key under plain Alt, so on Slovenian, Croatian, German, … layouts AltGr+V arrives as Alt+v and @ { } [ ] | ~ € cannot be typed by hand (the agent's type_text is unaffected — xdotool addresses keysyms). The host's complete keymap is copied into every instance (xkbcomp), verified, and re-copied by the bridge within 5 s if Xephyr reloads its own map or you switch the host layout. The launch result reports keyboard: {method: "host-keymap", host_layout, verified}. GLOVEBOX_KEYBOARD=de forces a layout instead.

  • Clipboard — each X server has its own; the per-instance bridge process mirrors text between the host and the instance clipboard (both directions, ~0.5 s, CLIPBOARD only, no images). On Wayland desktops the compositor already bridges the Xwayland clipboard to native apps, so copy on the host → paste in the sandbox and back just works. type_text restores the previous clipboard after its unicode paste so agent text does not linger on your host clipboard. Disable with GLOVEBOX_CLIPBOARD_SYNC=0; list_instances shows clipboard_sync per instance. The bridge exits by itself when the instance closes.

Because each display has its own cursor, multiple agents can drive different instances in parallel — one window each. Each agent's server process allocates display numbers under a cross-process lock, so simultaneous launches don't collide. The GPU is shared through the vision service (one model load, requests queue; see above). The host display for new windows is GLOVEBOX_HOST_DISPLAY (default :0); XAUTHORITY is auto-discovered. Instances outlive the server process by default (another agent or session can pick them up via list_instances); set GLOVEBOX_CLEANUP_ON_EXIT=1 to close the ones a server launched when it exits.

One-call action + observe

click · click_element · type_text · press_keys · scroll · drag · double_click · upload_file · wait_for_stable take observe (none default · screenshot · parse) and most take settle_ms. With observe="screenshot" the action returns its result and the resulting screen in a single call — no separate screenshot round-trip. Use settle_ms when you know the delay, wait_for_stable(observe="screenshot") when you don't. Default none keeps routine steps cheap; opt into screenshot/parse on the steps that change the page (navigations, submits).

Files & uploads

Each instance gets a staging folder files/<N>/ inside the install dir — a stable place to drop files for that instance (readable by native apps and, since it's under $HOME, by snap Chromium too). list_files(instance) shows the folder and its contents, with sizes and a partial flag for in-progress downloads.

  • Browser <input type=file> → upload_file(path, instance) (via CDP; page_url_contains picks the tab when several are open). The nested file picker is invisible to automation and hangs snap Chromium, so never click an upload button expecting a dialog.

  • Native apps (GIMP, Inkscape, editors) → open_file(path, instance, app="gimp"), or just drive the app's own Open dialog — unlike the browser's, it's a real visible window you can type a path into (Ctrl+L in a GTK file chooser).

  • Saving / downloads → apps run as your user, so they can save anywhere you can write. launch_app browser instances are pre-configured to download and "save as" into files/<N>/; point native apps' Save dialogs there too, then list_files(instance) to see the results.

Environment variables

Variable

Default

Meaning

GLOVEBOX_VISION

auto (or .vision-mode)

auto · none · basic · local

GLOVEBOX_HOST_DISPLAY

:0

host display where new Xephyr windows appear

GLOVEBOX_CLEANUP_ON_EXIT

off

1 → close the instances this server launched when it exits

GLOVEBOX_CLIPBOARD_SYNC

1

per-instance host bridge (clipboard mirroring + keymap kept equal to the host); 0 = off

GLOVEBOX_CLIPBOARD_POLL

0.5

seconds between clipboard polls in the bridge

GLOVEBOX_KEYBOARD

unset

force a layout for new instances (si, de(nodeadkeys)) instead of copying the host keymap

GLOVEBOX_TESS_LANG

slv+eng ∩ installed

tesseract language string for basic

GLOVEBOX_VISION_SERVICE

1

local mode parses through the shared per-machine service; 0 = load models in every server process

GLOVEBOX_VISION_SOCKET

$XDG_RUNTIME_DIR/glovebox-vision.sock

the service's Unix socket

GLOVEBOX_VISION_IDLE_MIN

15

the service exits after this many idle minutes

GLOVEBOX_OCR_SCALE

2

upscale factor before OCR in basic (1–4); UI text is too small for tesseract at 1x

GLOVEBOX_DETECTOR

auto

detector inside local: auto = ScreenParser when installed, else OmniParser's; omniparser · screenparser force

DISPLAY

:1

instance 1's display (set by mcp-config.json)

Maintenance (local mode)

The OmniParser lifecycle lives in vision.py (the server owns it; install.sh local just calls it): python vision.py install-omniparser clones OmniParser, downloads the v2 weights into OmniParser/weights/, and makes PaddleOCR optional (this uses easyocr). python vision.py check reports readiness. transformers is pinned to 4.49.0 — newer releases break Florence-2's remote config. If you upgrade OmniParser manually, re-run install-omniparser to re-apply the patch.

Optional detector: ScreenParser. OmniParser's own detector only knows "interactable icon", so every detected box is type: "icon". ScreenParser (docling, ICML 2026, Apache-2.0, 153 MB) is a YOLO11-L trained on 1.45M screenshots with 55 element classes; dropped into the same pipeline (easyocr text + Florence-2 captions) it gives parse_screen typed elements — button, link, tab, input, checkbox, radio, switch, select, slider, image, icon, … plus the raw class — and the paper reports fewer missed targets (ScreenSpot action accuracy 77.8 → 80.1 with a fixed agent). Install it and it is used automatically:

python vision.py install-screenparser     # downloads best.pt next to the OmniParser weights
python vision_service.py stop             # a running vision service keeps the old detector until restarted

Measured on an RTX A5000 Laptop, 1440×900 Google results page: 110 elements vs. 93, same 1.7 s per parse (OCR and captions dominate; the detector itself is 60 ms), 10 s model load. Layout containers (window, toolbar, side bar, table, …) are not reported — they would swallow the text inside them. GLOVEBOX_DETECTOR=omniparser|screenparser forces one; status() shows vision.omniparser.detector.

Stop

pkill Xephyr      # closes the sandbox (browser + WM + display)

Safety

  • The agent's input and vision are scoped to the sandbox display — it does not see or control your real desktop.

  • The server process and the apps it launches run as your user (shell/file access, like any MCP server); only the GUI control is sandboxed to the Xephyr window. Files the agent saves, opens or uploads are your real files. For OS-level isolation from your files, run it inside a VM or container.

  • You can watch everything live and close it instantly with pkill Xephyr.

  • Automate responsibly — only sites and services you are authorized to use.

Files

  • server.py — the MCP server (all 30 tools).

  • vision.py — vision backends: selection rules, GPU preflight, tesseract grouping, OmniParser lifecycle (check / install-omniparser CLI).

  • vision_service.py — the shared OmniParser process (Unix socket, idle-exit, status / stop CLI).

  • _paths.py — shared helpers: the 0700 per-user temp dir, display naming, the liveness probe.

  • skills/glovebox/SKILL.md — a Claude Code skill that teaches an agent the workflow (./install.sh skill installs it).

  • Launched apps are told to draw on the instance's X display (XDG_SESSION_TYPE=x11, QT_QPA_PLATFORM=xcb), LibreOffice gets a per-instance profile (profiles/libreoffice-<N>) so it cannot hand a document to a LibreOffice on the host, and close_instance ends the whole process group it started.

  • a11y_focus.py — AT-SPI helper for fill()'s focus check; runs under the system python3 (needs python3-gi, gir1.2-atspi-2.0, at-spi2-core — install.sh adds them; without them fill still works and reports focus_checked: false).

  • host_bridge.py — per-instance host bridge: host keymap copy (AltGr fix) + clipboard mirroring (keymap / serve / spawn / status CLI).

  • install.sh — mode-aware installer (none / basic / local).

  • start-display.sh — launches the Xephyr sandbox (display + window manager + browser).

  • AGENTS.md — drop-in tool-usage instructions for the AI agent (paste into its system prompt).

  • ROADMAP.md — what is planned next (MCP SDK 2.x migration, …).

  • mcp-config.json — a ready-to-paste MCP client config snippet.

Credits

local vision mode uses Microsoft's OmniParser (cloned and weights downloaded by vision.py, under its own license). Screen capture uses mss; input is driven with xdotool. Not affiliated with Microsoft.

Contributing

Shipped as-is under MIT. Issues and PRs are welcome, but this is maintained by one person — no support or response time is guaranteed. If it's useful to you, a ⭐ helps.

License

MIT — see LICENSE.

Available Tools

30 tools
app_logA

Tail of the stderr log of the apps launched on an instance (launch_app / start_app / open_file) — read it when an app crashed, drew nothing, or misbehaves.

Returns {"ok", "instance", "path", "lines": [...], "total_lines", "truncated"} (max 500 lines per call). The log is per instance: launch_app starts it fresh, start_app/open_file append. Instance 1 (start-display.sh) logs to /tmp/chromium-x1.log instead. Fails (isError) if no log exists for that instance yet.

Inputs: lines, instance. Returns: result

browserA

Talk to the instance's Chromium-family browser over the Chrome DevTools Protocol — exact navigation and page text without keystrokes or OCR.

action='info' → {"ok", "url", "title", "pages": [{url, title}, …], "viewport"} of the active tab + all tabs. viewport = {dpr, left, top, css_width, css_height} maps the page's CSS pixels (getBoundingClientRect) to the pixels click()/screenshot() use: x = left + css_xdpr, y = top + css_ydpr (dpr is not 1 on a zoomed page; top is the toolbar height) action='navigate' → loads url in the active tab (waits for the load event, max 30 s); returns {"ok", "url", "title", "loaded": bool} action='text' → {"ok", "url", "title", "text", "length", "truncated"}: the rendered page text (document.body.innerText), capped at max_chars (default 20000, max 200000) action='eval' → {"ok", "result"}: the JSON value of the JavaScript expression js (e.g. "document.querySelector('h1').innerText"); a Promise is awaited. A result whose JSON is longer than max_chars (default 100000, max 1000000) comes back as {"ok", "truncated": true, "length", "result_json": the first max_chars characters} — raise max_chars or return less (slice/map in the script). Errors in the page script come back as a failure with the message timeout_s (1–600, default 30) bounds an eval or text call. On a timeout the call fails (isError, "timed_out": true) and the connection stays usable, but the script KEEPS RUNNING in the page — for long work store the outcome on window (js="fetchAll().then(r => { window.__r = r }); 'started'") and poll with short evals. page_url_contains picks a tab by URL fragment (default: the most recently active page). Works only for browsers started by launch_app/start-display.sh (CDP port 9222+N); fails (isError) with the fix otherwise, or when the tab, URL or script is bad. The page state you see in screenshots is this same tab.

Inputs: js, url, action, instance, +3 more. Returns: result

clickA

Click at absolute pixel coords (top-left origin) in an instance.

For anything with a label, find_elements('Save') -> click_element(id) beats guessing pixels. Returns {"ok", "action", "instance", "detail", "hit"?, "warnings"?}. "hit" (Chromium-family browsers only, read from the page's DOM just before the click) names what was under the point: {"role": "checkbox", "name": "Remember me", "tag", "checked_before", "disabled", "href", …} or {"role": "browser-ui"} for the toolbar/tab strip — compare it with what you meant to click; it is absent for native apps. "warnings" appears when the pointer was not where this server left it (a person or another agent is using the instance — re-screenshot). button: 1=left, 2=middle, 3=right. count: 1–3 clicks (2 = double-click, 3 = triple-click selects a line/paragraph). modifiers: keys held during the click — 'ctrl' (open in new tab / multi-select), 'shift' (range select), 'alt', 'super', combinable 'ctrl+shift'; released afterwards whatever happens. observe='screenshot'|'parse' additionally returns the resulting screen state in the SAME call (saves a round-trip); settle_ms (max 5000, "settle_clamped": true if you asked for more) waits for the UI to update first — use 400–1500 after anything that navigates or submits, or wait_for_stable() when you don't know how long. observe_region='x,y,w,h' crops that screenshot to the part you care about (implies observe='screenshot'; far fewer tokens than the full display; validated before the action). Fails (isError) if the instance is down, the coordinates are outside the display or a modifier name is unknown (nothing is clicked then). A failed observation after a successful click is reported as "observed": false inside an ok result — the click is not repeated.

Inputs: x (required), y (required), count, button, +5 more.

click_elementA

Click an element by id from that instance's most recent parse_screen/find_elements (no coordinate guessing).

Returns {"ok", "action", "instance", "detail", "element", "center", "hit"?, "warnings"?} ("hit"/"warnings": see click()). Fails (isError) with the fix named if the id is unknown — run parse_screen(instance) first. Ids go stale when the screen changes: after navigation, re-parse before clicking. modifiers/observe/settle_ms: see click().

Inputs: id (required), button, observe, instance, +3 more.

clipboardA

Read or set the instance's clipboard — the cheapest way to get EXACT text out of an app.

action='get' returns {"ok", "instance", "text", "length", "empty", "truncated"}: select text in the app (click/drag, or press_keys('ctrl+a')) then press_keys('ctrl+c') and read it here — no OCR, no screenshot. "empty": true means nothing (or non-text data) is on the clipboard; text is capped at 100000 characters ("truncated": true). action='set' puts text on the clipboard (then press_keys('ctrl+v') or 'shift+Insert' in terminals) and returns {"ok", "instance", "length"}. Each instance has its own clipboard (separate X display); the bridge started by launch_app mirrors TEXT between it and the host's clipboard within ~1 s (so a person can copy/paste across — and whatever they copy on the host is readable here; GLOVEBOX_CLIPBOARD_SYNC=0 turns it off). Fails (isError) if xclip is missing, the instance is down, or xclip itself fails.

Inputs: text, action, instance. Returns: result

close_instanceA

Close an instance: kill its app + its Xephyr display, and verify it actually went down.

Returns {"ok", "instance", "detail"} only after the display stops responding. With all_launched=true closes EVERY instance this server process launched (instance is ignored) and returns {"ok", "closed": [...], "failed": [...]} — "ok" is false if any survived. Fails (isError) if the instance isn't running, if it survives SIGTERM, or for instance 1 — the main start-display.sh sandbox is protected (stop it from the host with pkill Xephyr). Closing an instance another agent launched is allowed but visible to it (launched_here).

Inputs: instance, all_launched. Returns: result

double_clickA

Double-click at absolute pixel coords in an instance (e.g. open a file icon, select a word). Same as click(count=2).

Returns {"ok", "action", "instance", "detail", "hit"?, "warnings"?} — see click(). observe/settle_ms/observe_region: see click(). Fails (isError) if the instance is down or the coordinates are off-screen.

Inputs: x (required), y (required), observe, instance, +2 more.

dragA

Drag (mouse-down → move → up) from (x1,y1) to (x2,y2) — for drawing, selecting, sliders.

Returns {"ok", "action", "instance", "detail", "steps"}. steps (default 10, 1–100) is the number of intermediate motion events: canvases, sliders and drag-and-drop targets usually need a few to register the drag (1 = jump straight to the end point). The button is released even if a motion step fails (a stuck button would turn every later click into a drag). observe/settle_ms: see click(). Fails (isError) if the instance is down or either point is off-screen.

Inputs: x1 (required), x2 (required), y1 (required), y2 (required), +6 more.

fillA

Set a form field in ONE verified call: click it, check the focus, replace its content with text, read the value back. Use it instead of click + type_text + screenshot for every field of a form.

Target the field with x,y (absolute pixels) or element (an id from parse_screen/ find_elements). Steps and what each can tell you:

  1. click, then a FOCUS CHECK before anything is typed — browsers through the page's DOM, native GTK/Qt/LibreOffice apps through the accessibility bus (AT-SPI). If the focus is not on an editable field, or is on a field whose rectangle does not contain the click (you clicked a label; the focus stayed where it was), the call FAILS with nothing typed and "focus" naming what has it. A password field is refused the same way unless allow_password=true — landing in one by accident is the case worth stopping.

  2. the old content is replaced (select-all, then type; text="" empties the field);

  3. the value is read back; submit=true presses Return only if nothing failed. Returns {"ok", "action", "instance", "detail", "method", "center", "focus_checked": "browser-dom"|"at-spi"|false, "focus"?: {role, name, rect}, "verified": true|false|null, "read_back": "browser-dom"|"clipboard"|"unreadable"|"skipped", "value"?, "click_effect"?: "changed"|"none", "suggest_element"?, "hit"?, "warnings"?}: focus_checked false — nobody could tell which widget had focus (canvas, game, Xlib-only app, iframe, or AT-SPI not installed: "focus_unchecked_reason"). The text was typed blind; "warnings" says so and "click_effect" reports whether the click changed any pixel near the target ("none" = probably a static label). Treat such a fill as UNCONFIRMED and screenshot. verified true — the focused field holds exactly text (trailing newline ignored); verified false — it holds something else ("value", first 200 chars): the field reformats or limits input (dates, phone masks, maxlength); verified null — the value could not be read (native password field, canvas). With element, a typed detector (ScreenParser) warns when the element is a text label and names the nearest input ("suggest_element"). Also fails (isError) when the field is still empty after typing; repeating fill() is always safe (it replaces). Native read-back goes through select-all + copy (the clipboard is restored). observe/settle_ms/ observe_region: see click().

Inputs: text (required), x, y, submit, +7 more.

find_elementsA

Parse the screen and return only the elements whose label contains query (case-insensitive substring; empty query = all), optionally filtered by type (button, link, text, icon, …). One call instead of parse_screen + scanning 300 elements. The ids are valid for click_element(id) right away (this IS the most recent parse).

Returns {"ok", "instance", "vision", "query", "type", "count": total matches, "elements": [{id, type, label, interactive, center}, …] (first limit, 1–100), "truncated"}. "count": 0 is an honest answer (the text is not on screen, or OCR did not read it — screenshot to check). Fails (isError) like parse_screen; in vision mode 'none' it fails with the fix (no grounding backend: screenshot and reason yourself).

Inputs: query (required), type, limit, instance. Returns: result

get_screen_sizeA

Screen size of an instance + the active vision backend and server version.

Returns {"ok", "instance", "width", "height", "vision", "version"}. Call it once before coordinate math — coordinates are absolute pixels, top-left origin, within THIS instance's display (0 <= x < width, 0 <= y < height; out-of-range coordinates are rejected). "vision" is the EFFECTIVE grounding backend: 'basic'/'local' → parse_screen + click_element; 'none' → screenshot + reason about pixels yourself. Fails (isError) if the instance is down.

Inputs: instance. Returns: result

launch_appA

Launch a GUI app in its OWN new Xephyr display/window and return its instance id.

command is any shell command ('chromium', 'xterm', 'gimp', 'inkscape file.svg', …). Chromium-family browsers (chromium, google-chrome, brave, microsoft-edge, vivaldi, opera) automatically get X11 flags, a per-instance profile, downloads routed to files//, and a CDP debug port (needed by upload_file and browser()). Other GTK apps get D-Bus isolation so they render on this display, not the host.

Returns {"ok", "instance", "display", "name", "files_dir", "width", "height", "warnings": [...], "keyboard": {method, host_layout, verified}, "clipboard_sync": bool} — pass that instance id to every control tool. "keyboard" says how the window got the host's keyboard layout ("host-keymap" = exact copy, verified against the host; Xephyr alone maps AltGr as plain Alt, so a person could not type @ { } [ ] on Slovenian/German/… layouts); "clipboard_sync" is true when the per-instance host bridge runs (it mirrors text between the host and instance clipboards and keeps the keymap equal to the host's; GLOVEBOX_CLIPBOARD_SYNC=0 disables it — note that whatever the person copies on the host then becomes readable inside the sandbox). Neither affects the agent's own type_text/clipboard. Fails (isError) BEFORE anything starts if the command is empty/unparsable or size is not WxH; after that if Xephyr can't start on the host display (its log tail is quoted), the command doesn't exist, or the app exits immediately (its stderr tail is quoted; app_log(instance) has more) — the display is closed again in those cases. "warnings" lists what degraded: window manager (openbox) missing (Xephyr then ignores pointer motion, so clicks would land in the screen center), keymap copy or clipboard bridge failed. Each instance has its own cursor, so separate agents can drive separate instances in parallel; display numbers are allocated under a cross-process lock, so parallel launches don't collide. Never share one instance between agents. The window appears on GLOVEBOX_HOST_DISPLAY (default :0). Give the app a moment to draw (wait_for_stable / settle_ms) before the first screenshot. Instances outlive this server process unless GLOVEBOX_CLEANUP_ON_EXIT=1.

Inputs: command (required), name, size. Returns: result

list_filesA

The instance's staging folder (files// under the install dir) and its contents.

Returns {"ok", "instance", "dir", "count", "truncated", "files": [{name, bytes, modified, is_dir, partial}, …]} (capped at 200 entries, newest first). "partial": true marks an in-progress browser download (.crdownload/.part/.tmp) — wait for it to disappear before using the file. Drop files there (or reference any host path you can read) to open in apps via open_file() or an app's Open dialog; launch_app browser instances also download / 'save as' into it — call this after a download to confirm the file arrived. Browser uploads still go through upload_file(). Fails (isError) if the folder cannot be read.

Inputs: instance. Returns: result

list_instancesA

List all running instances (app windows): id, display, name, command, title, launched_here.

Returns {"ok", "count", "instances": [{instance, display, name, command, up, title, launched_here, clipboard_sync}, …]}. "title" is the active window's title (a browser's follows the page) — the way to tell instances apart that another agent's server launched (their name/command show as unknown, launched_here=false). Probes displays :1–:24 for sandboxes started outside this server (start-display.sh, other agents). Instances whose window was closed by hand disappear from the list and their number is reused.

Inputs: no named parameters. Returns: result

manage_windowA

Act on a window of an instance: action='activate' (focus/raise — switch between an app's windows or dialogs), 'close' (asks the app to close it — a save prompt may appear), 'minimize', 'maximize', 'restore' (un-maximize), 'move' (to x,y), 'resize' (to width x height). window_id=0 means the active window (window_info lists ids).

Returns {"ok", "action", "instance", "window", "detail"} plus observe/settle_ms like click(). Fails (isError) if the window id is unknown, the action is unknown, move/resize sizes are missing, or the instance is down. maximize/restore need wmctrl (status().deps).

Inputs: action (required), x, y, width, +6 more.

move_mouseA

Move the mouse pointer to absolute pixel coords in an instance (no click).

Returns {"ok", "action", "instance", "detail"}. Useful for hover states (menus, tooltips) — follow with screenshot() to see the effect. Fails (isError) if the instance is down or the coordinates are off-screen.

Inputs: x (required), y (required), instance. Returns: result

open_fileA

Open a local file in an app ON an existing instance's display (NOT a new instance).

If app is given, runs <app> <filepath> there (e.g. app='gimp'); otherwise xdg-open picks the default handler. Relative paths resolve against the instance's staging folder files// (see list_files). Returns {"ok", "action", "instance", "opened", "app"}; fails (isError) if the file or the app doesn't exist, or if the handler exits immediately (e.g. xdg-open finds no application for that file type — then pass app=). The app is started asynchronously — wait_for_stable()/screenshot after a moment to confirm it drew. For BROWSER uploads use upload_file() instead. Both paths get the same X11 + D-Bus handling as launch_app (so the app renders on this display, not in an already-running window on the host).

Inputs: filepath (required), app, instance. Returns: result

parse_screenA

Detect on-screen elements and return them with ids + pixel-centers (vision grounding).

Returns {"ok", "vision", "screen": [W,H], "annotated", "count", "truncated", "elements": [{id, type, label, interactive, center: [x,y]}, …]} — then click_element(id). Also saves a numbered Set-of-Mark image (path in "annotated"). Backend = the effective GLOVEBOX_VISION: 'local' (OmniParser on the GPU, ~2 s after a ~10 s first-call model load; served by ONE shared vision-service process per machine so parallel agents don't each load the models — "via": "service"|"in-process", "queued_ms" says how long the GPU queue was, "detector" which detector answered, "fallback_reason" why the service was not used. "type" is "text"|"icon" with OmniParser's own detector; with the optional ScreenParser detector (status().vision.omniparser.detector) it is one of text, button, link, tab, input, checkbox, radio, switch, select, slider, stepper, rating, icon, image, item, indicator and each element also carries "class", the detector's raw class name; buttons/tabs/ links carry their visible text as "label", icons/images a short caption. box_threshold raises the detector's confidence floor to cut noise), 'basic' (tesseract, text only; granularity='line' merges words on one line into one target like "Sign in" — default, split at wide gaps so a menu bar is one target per item — or 'word' for single words; the frame is OCR'd at 2× (GLOVEBOX_OCR_SCALE) because UI text is too small for tesseract at 1×; each element also carries "confidence" 0–100), 'none' (returns a note telling you to screenshot + reason about pixels yourself — this is normal, not an error; status() says why if 'auto' fell back to it). Fails (isError) with the fix named when an explicitly requested backend is not usable (OmniParser not installed, no/too-small GPU, GPU out of memory, the vision service busy, tesseract missing its language data). Elements are capped at 300 per call ("truncated": true when more were detected) — use find_elements to search instead of paging. Ids are only valid until the screen changes — re-parse after navigation.

Inputs: instance, granularity, box_threshold. Returns: result

press_keysA

Press a key, a combo, or a SEQUENCE (xdotool keysym syntax, space-separated for sequences): 'Return', 'ctrl+a', 'Tab', 'ctrl+t', 'F5', 'ctrl+a Delete', 'alt+F4', 'shift+Insert'.

Returns {"ok", "action", "instance", "detail", "keys": [...]}. Every token is validated against the X keysym table BEFORE anything is pressed, so an invalid name fails with nothing pressed — also in the middle of a sequence. Fails (isError) on a dead instance or a frozen display. Browser navigation: 'ctrl+l' → type_text(url) → 'Return' (or browser(action= 'navigate')). observe/settle_ms: see click().

Inputs: keys (required), observe, instance, settle_ms, +1 more.

put_fileA

Write a file into the instance's staging folder files// — the way to hand the sandbox a file you produced (a CSV to import, an SVG to open, a text to upload) without shell access. Then open_file(name) / upload_file(name) / an app's Open dialog can use it by that name.

name is a relative path inside the staging folder ('data.csv', 'in/logo.svg'; parent folders are created); absolute paths and '..' are refused — nothing outside files// can be written. content is text (written as UTF-8) or, with encoding='base64', binary data. Returns {"ok", "instance", "path", "name", "bytes", "overwritten"}. Fails (isError), writing nothing, if the file exists and overwrite is false, the name escapes the folder or names a directory, the base64 is invalid, or the content exceeds 10 MB. The instance does not have to be running (stage files first, launch after).

Inputs: name (required), content (required), encoding, instance, +1 more. Returns: result

screenshotA

Screenshot of an instance's display — the ONLY way to see the screen; always look before you act.

Returns [json, image]: {"ok", "instance", "width", "height", "region", "scale", "format", "bytes"} plus the image. Options to cut payload or zoom: region="x,y,w,h" crop to a rectangle (zoom into small text/icons; coordinates stay absolute — add x,y back when you click), default = full display; scale=0.5 downscale (0.1–1.0) for cheap overview shots; format="jpeg" smaller than PNG for photos/video, quality 1–95 (default png, lossless). For an element list with coordinates use parse_screen (basic/local vision). Prefer observe='screenshot' on action tools to act and see the result in one call. Fails (isError) if the instance is down, the region is outside the display, or the format is unknown.

Inputs: scale, format, region, quality, +1 more.

scrollA

Scroll the wheel. Say where: direction='down'|'up'|'left'|'right' with amount notches (sign ignored) — or the signed form: positive = up (or left), negative = down (or right).

Returns {"ok", "action", "instance", "detail", "applied", "clamped", "axis", "direction"}. One unit ≈ one wheel notch (a few lines); at most 100 notches per call ("clamped": true when you asked for more, "applied" says how many were sent). Scrolling targets the window under the pointer — pass x,y to move there first (both >= 0), otherwise the current pointer position is used. axis='horizontal' scrolls sideways (wheel buttons 6/7). Fails (isError) on amount=0 (a no-op), a dead instance, off-screen x,y or an unknown axis. observe/settle_ms: see click().

Inputs: x, y, axis, amount, +5 more.

start_appA

Start ANOTHER app on an existing instance's display (same window/cursor) — e.g. a second tool next to the browser, or a restart of an app that quit. For a new, separate window use launch_app; to open a file in an app use open_file.

Returns {"ok", "instance", "command", "pid"}; fails (isError) if the instance is down, the command is unparsable/missing, or the app exits immediately (stderr tail quoted; app_log(instance) has more — logs of apps started here are appended to it). Browsers get the same X11/profile/CDP handling as launch_app. The app starts asynchronously: wait_for_stable() or screenshot() after a moment to confirm it drew.

Inputs: command (required), instance. Returns: result

statusA

Server + sandbox status in one read-only call — run this first, and when filing bug reports.

Returns {"ok", "version", "host_display", "tmp_dir", "cleanup_on_exit", "vision": {requested, effective, ready, reason, fallback, gpu: {present, name, vram_total_mb, vram_free_mb, enough, note}, omniparser: {code, weights, torch, ready, loadable, reason, detector: {requested, effective, screenparser_weights, reason}, loaded, service: {running, pid, loaded, detector, served, in_flight, idle_s, …} — the shared per-machine OmniParser process (one GPU load for all agents; starts on first use, idle-exits)}, tesseract: {binary, pytesseract, langs, lang, ready, reason}}, "instances": [...same as list_instances], "keyboard": {host_layout, override}, "clipboard_sync": {enabled, poll_s, note}, "deps": {xdotool, xclip, xephyr, openbox, wmctrl, xdpyinfo, xkbcomp, setxkbmap, dbus_run_session, tesseract, nvidia_smi, xprop, atspi: true|false}, "notes": [...], "tips": [...]} ("tips": the tools that save the most round-trips on this setup — read them once). "vision.effective" is the grounding backend actually in use (none|basic|local), re-resolved every 20 s (installing OmniParser takes effect without a restart); when GLOVEBOX_VISION=auto it is the best READY one and "reason" says why (e.g. OmniParser weights missing, GPU too small). "notes" lists things worth knowing (e.g. the service runs a different detector than this server would pick). No side effects.

Inputs: no named parameters. Returns: result

type_textA

Type into the focused field — unicode-safe. Click the field first to focus it.

To set a form field in one verified call use fill() instead (click + replace + read back). Returns {"ok", "action", "instance", "detail", "method": "xdotool"|"clipboard", "warnings"?}. clear=true selects the field's content first (ctrl+a) so the text REPLACES it; submit=true presses Return afterwards — one call for "fill and submit". Pure ASCII is typed via xdotool (timeout scales with length); text with any non-ASCII char (č/š/ž …) is inserted via the clipboard + ctrl+v, because xdotool's synthetic unicode keystrokes get silently dropped by some toolkits (e.g. Inkscape's GTK canvas) even though they work in browsers. The clipboard is put back as it was afterwards (previous text, or cleared), best-effort: an app that reads the selection late (> 0.3 s) can paste the restored text instead, and the host bridge may have mirrored the typed text in that window — see clipboard(). Terminals that paste with shift+Insert instead of ctrl+v need press_keys after clipboard(action='set'). Fails (isError) rather than silently losing characters if xclip is missing for non-ASCII text. observe/settle_ms: see click().

Inputs: text (required), clear, submit, observe, +3 more.

upload_fileA

Attach a local file to a page's via Chrome DevTools Protocol — use this for ALL browser uploads (logos, images, docs).

The sandbox's native GTK file picker is invisible to us AND hangs the renderer, so never click an upload button expecting a dialog — call this instead. Works on Chromium-family browsers started by launch_app / start-display.sh (they open a per-instance --remote-debugging-port, 9222+N). selector targets the file input on the MAIN page (default = first file input). With several tabs open the most recently active page target is used unless page_url_contains picks the tab by a URL fragment. If the site only inserts the after you click its 'upload' control, click that first (it won't open a dialog we can see, but it wires up the input), then call this. Relative paths resolve against files//. Returns {"ok", "action", "instance", "detail", "page"}; fails (isError) with the reason if the file is missing, no CDP target answers, or the selector matches nothing. NATIVE (non-browser) apps: use open_file() or drive the app's own Open dialog — it IS visible here. observe/settle_ms: see click().

Inputs: filepath (required), observe, instance, selector, +3 more.

wait_for_stableA

Wait until the screen stops changing (page loaded, app drew, animation ended) — replaces guessing a settle_ms.

Samples the display every interval_ms (100–2000) and returns as soon as two consecutive frames differ by at most threshold (fraction of pixels, default 0.2%), or after timeout_ms (max 30000). Returns {"ok", "instance", "stable", "elapsed_ms", "frames", "change": last changed fraction, "timed_out"}. "stable": false with "timed_out": true means the screen was still changing (video, spinner, blinking cursor — raise the threshold to ~0.01 for those); it is reported, not hidden. observe='screenshot'|'parse' returns the final state in the same call. Fails (isError) if the instance is down.

Inputs: observe, instance, threshold, timeout_ms, +2 more.

wait_for_textA

Wait until text appears on screen (or, appear=false, disappears) — by OCR/parse polling. The honest replacement for "sleep and hope" after navigation or a submit: returns as soon as the text is (not) there, or reports "timed_out": true.

Returns {"ok", "instance", "found": bool, "satisfied": bool, "elapsed_ms", "polls", "element": {id, type, label, center} | null, "timed_out"}. "satisfied" is what you waited for; "timed_out": true with satisfied: false is reported, not hidden — screenshot to see what IS there. timeout_ms max 60000, interval_ms 300–5000. Fails (isError) if the instance is down or there is no grounding backend (vision mode 'none').

Inputs: text (required), appear, instance, timeout_ms, +1 more. Returns: result

wait_msA

Wait (e.g. for a page to load or an app to draw) — capped at 10000 ms.

Returns {"ok", "action", "requested_ms", "waited_ms", "clamped"} — "clamped": true means you asked for more than the cap and only waited_ms elapsed. Prefer wait_for_stable() (waits until the screen stops changing), wait_for_text(), or settle_ms on the action itself when you also want to observe the result in the same call.

Inputs: ms (required). Returns: result

window_infoA

The active window's title + geometry, the visible windows and the pointer position on an instance — a cheap, image-free way to confirm navigation (browser titles follow the page) or that an app opened.

Returns {"ok", "instance", "active": {id, title, x, y, width, height, pid} | null, "windows": [{id, title}, …] (visible, capped at 50, unnamed ones included with title ""), "count", "truncated", "cursor": [x, y]}. "active" is null when no window has focus yet (fresh instance) — that is reported, not an error. Window ids are what manage_window() takes. Fails (isError) if the instance is down or xdotool is missing/fails.

Inputs: instance. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 30 tools

    Disambiguation4/5

    Most tools target clearly distinct operations; launch_app/start_app/open_file and click/click_element/double_click are separated by precise descriptions. Minor overlap remains between fill and type_text, parse_screen and find_elements, and double_click as click(count=2), but the descriptions make the intended choice clear.

    Naming Consistency4/5

    All tool names use snake_case with no camelCase mixing, and most follow a readable verb_noun pattern. A few bare nouns (status, browser, clipboard, screenshot) and mixed get_/info patterns are minor deviations rather than inconsistency.

    Tool Count2/5

    At 30 tools the surface is heavier than ideal, and several conveniences such as double_click, find_elements, and wait_ms could be folded into existing tools. The broad GUI/browser/vision/file scope explains many tools, but the count is still well above the recommended range.

    Completeness4/5

    The set covers the full lifecycle well: launch/start/close/list instances, mouse and keyboard control, screenshots and vision parsing, browser CDP actions, window management, clipboard, and file staging/upload. Minor gaps exist around direct file deletion/rename and some file-management operations, but agents can generally work around them via overwrite or app dialogs.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers