Skip to main content
Glama

glovebox-mcp

A sandboxed computer-use MCP server — let an AI agent drive a real browser and desktop apps (mouse, keyboard, screenshots, vision grounding), confined to a nested X11 window so it can never touch your real screen, files, or other apps.

Like a lab glovebox: the agent reaches in and manipulates real applications, sealed off from everything else. Bring the sandbox up, log into whatever sites or apps you want to automate inside that window, and the agent operates only there — you can watch it live and close it instantly.

Speaks the Model Context Protocol, so it works with MCP clients like Claude Code. Your host can run Wayland; the sandbox gives the agent a real X server to drive.

glovebox-mcp — an agent filling a sign-up form inside the sandbox

An agent driving a real browser in the sandbox — gliding the cursor, inserting a unicode name (Nadja Kovačič), typing, and submitting. All confined to a nested X11 window.

Why a nested X11 sandbox?

  • Most desktop automation (xdotool, PyAutoGUI) is X11-only, but many modern desktops run Wayland.

  • Xephyr provides a real X server inside a single window (DISPLAY :1). Everything the agent does — clicks, typing, screenshots — is confined to that window, not your real desktop.

  • You stay in control: watch it live, pkill Xephyr to close everything.

Related MCP server: openowl

Requirements

  • Linux — the sandbox nests a real X server (Xephyr), so it works even on Wayland hosts (via Xwayland). Not macOS/Windows. Developed on Ubuntu; any modern Linux with the packages below.

  • Python 3.10+ and uv (used for the virtualenv).

  • System packagesxserver-xephyr (Xephyr), openbox, scrot, x11-utils, xdotool, wmctrl, xclip (+ tesseract-ocr for basic). On Debian/Ubuntu the installer auto-installs them via apt (sudo); on Fedora/Arch it prints the matching dnf/pacman command. The MCP server itself is distro-agnostic — any Linux with these tools works.

  • A browser in the sandbox (Chromium or Chrome).

  • NVIDIA GPU (≥6 GB VRAM) — only for the local vision mode.

Install

Pick a vision backend and run its one-liner (clone → install). Each one installs the system packages (auto via apt on Debian/Ubuntu) and the Python deps for that mode, and writes a ready-to-paste mcp-config.json with your paths.

none — no local models; your agent reads screenshots itself (lightest, instant):

git clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh none

basic — Tesseract OCR grounding (parse_screen → text + coordinates, CPU-only):

git clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh basic

local — OmniParser on an NVIDIA GPU (parse_screen → text + icons, pixel-precise; ~4 GB weights, ≥6 GB VRAM):

git clone https://github.com/segentic-lab/glovebox-mcp && cd glovebox-mcp && ./install.sh local

Your choice is written to .vision-mode (override per run with the GLOVEBOX_VISION env var).

Works with any MCP client / harness

Claude Code, Cursor, Codex, or your own agent — it's a standard MCP server, not tied to any one host. Two compatibility notes: basic/local return element coordinates as text, so they work even with text-only agents; none relies on the client passing the tool's screenshots to a multimodal model (fine for Claude Code, Cursor, and other image-capable MCP clients).

Quickstart

  1. Start the sandbox (leave it running):

    ./start-display.sh              # 1440×900 Xephyr window with a browser
    ./start-display.sh 1920x1080    # …or pass a screen size (or set $RES)

    Log into any sites or apps you want to automate in that window.

  2. Register the server with your MCP client. install.sh already wrote mcp-config.json with your real install path — copy its glovebox block into your client's MCP config:

    { "mcpServers": { "glovebox": {
        "command": "/abs/path/to/glovebox-mcp/.venv/bin/python",   // filled in by install.sh
        "args":    ["/abs/path/to/glovebox-mcp/server.py"],
        "env":     { "DISPLAY": ":1" }
    } } }

    Restart the client so it loads the server.

  3. Ask the agent to screenshot / click / type — it operates only on the :1 window.

Driving it with an AI agent? Paste AGENTS.md into the agent's system prompt — it teaches the observe → act → verify loop, grounding, the upload/unicode gotchas, and when to stop.

Tools

Tool

What

status()

Server + sandbox status in one read-only call: version, vision backend, host display, live instances, and which system deps (xdotool, xclip, Xephyr, tesseract, OmniParser weights) are present. Run it first — and paste it into bug reports.

parse_screen()

Vision grounding → JSON of detected elements (id, type, label, interactive, pixel-center; capped at 300 per call, flagged via "truncated") + a numbered Set-of-Mark image at /tmp/glovebox_annotated_<N>.png. (local mode: OmniParser on GPU, ~2 s.)

click_element(id)

Click an element from the last parse_screen (no coordinate guessing).

screenshot()

Screenshot of an instance.

click(x,y) · move_mouse · scroll · drag · double_click

Pointer ops.

type_text(text)

Unicode-safe typing (ASCII via xdotool; anything with č/š/ž… is inserted via the clipboard, because xdotool's synthetic unicode is silently dropped by some GTK apps).

press_keys("ctrl+a"/"Return"/…)

Keys/combos (xdotool syntax).

upload_file(filepath, selector?)

Attach a local file to a page's <input type=file> via the Chrome DevTools Protocol. The nested X11 file picker is invisible to automation and hangs the renderer, so use this for all uploads — never click an upload button expecting a dialog. Works on Chromium started by launch_app/start-display.sh (they open a per-instance --remote-debugging-port, 9222+N). Browser file inputs only — for native apps see open_file.

open_file(filepath, app?)

Open a local file in a native app on the instance's display (e.g. app="gimp") or via xdg-open. GTK apps get the same X11/D-Bus handling as launch_app.

list_files()

The instance's staging folder files/<N>/ (under the install dir) + its contents.

launch_app(command, name?, size?) · list_instances() · close_instance(n)

Multi-instance control (see below).

wait_ms(ms) · get_screen_size()

Timing / sandbox size.

Every control tool takes instance=N and optional observe / settle_ms (see below). In local mode OmniParser is lazy-loaded on first parse_screen (~6 s once, then ~2 s/parse).

Structured, honest responses

Every tool returns JSON: {"ok": true, "action": "click", "instance": 1, "detail": "clicked (10,20) button 1", …}. Failures come back with the MCP isError flag set and the same JSON shape embedded in the error text — {"ok": false, "error": "…", "fix": "…"} — where fix names the call that unblocks you (e.g. a click on a dead instance says to run list_instances() / launch_app(); an unknown element id says to re-run parse_screen()). Silent no-ops are treated as failures too: an invalid keysym, a zero scroll, closing an instance that isn't running, or non-ASCII typing without xclip all error instead of pretending success. screenshot() returns a PNG image; observe="screenshot"|"parse" returns [json, image] in one call.

Vision backend (selectable)

GLOVEBOX_VISION env var, or the .vision-mode file, or default local:

Mode

parse_screen

Needs

When

none

disabled (returns a note) — use screenshot() + reason

nothing (mcp, mss, xdotool)

lightest; let the agent's own vision do grounding

basic

Tesseract OCR → text elements + coords

tesseract-ocr + pytesseract

no GPU; text-only grounding

local

OmniParser → text + icons + coords

torch + CUDA + OmniParser weights

best grounding

Switch anytime with ./install.sh <mode> (installs only what that mode needs).

Multi-instance (a fleet of app windows)

Every control tool takes instance=N (default 1 = the start-display.sh sandbox). Spin up more — each its own Xephyr display/window on the host desktop:

  • launch_app(command, name?, size?) → starts the next free :N running any GUI app (chromium, gimp, inkscape, xterm, …). Chromium auto-gets X11 flags, a per-instance profile, a remote-debugging port, and D-Bus isolation. Returns the instance id.

  • list_instances() · close_instance(n).

Because each display has its own cursor, multiple agents can drive different instances in parallel — one window each. The only shared resource is the GPU for local-mode parse_screen (it just queues). The host display for new windows is GLOVEBOX_HOST_DISPLAY (default :0); XAUTHORITY is auto-discovered.

One-call action + observe

click · click_element · type_text · press_keys · scroll · drag · double_click take observe (none default · screenshot · parse) and settle_ms. With observe="screenshot" the action returns its result and the resulting screen in a single call (with settle_ms to let the page update first) — no separate screenshot round-trip. Default none keeps routine steps cheap; opt into screenshot/parse on the steps that change the page (navigations, submits).

Files & uploads

Each instance gets a staging folder files/<N>/ inside the install dir — a stable place to drop files for that instance (readable by native apps and, since it's under $HOME, by snap Chromium too). list_files(instance) shows the folder and its contents.

  • Browser <input type=file>upload_file(path, instance) (via CDP). The nested file picker is invisible to automation and hangs snap Chromium, so never click an upload button expecting a dialog.

  • Native apps (GIMP, Inkscape, editors) → open_file(path, instance, app="gimp"), or just drive the app's own Open dialog — unlike the browser's, it's a real visible window you can type a path into (Ctrl+L in a GTK file chooser).

  • Saving / downloads → apps run as your user, so they can save anywhere you can write. launch_app Chromium instances are pre-configured to download and "save as" into files/<N>/; point native apps' Save dialogs there too, then list_files(instance) to see the results.

Maintenance (local mode)

install.sh clones OmniParser, downloads the v2 weights, and applies two patches automatically:

  • PaddleOCR made optional (this uses easyocr): OmniParser/util/utils.py's from paddleocr import PaddleOCR is wrapped in try/except and the module-level paddle_ocr = PaddleOCR(...) is guarded with … if PaddleOCR is not None else None.

  • transformers is pinned to 4.49.0 — newer releases break Florence-2's remote config.

If you upgrade OmniParser manually, re-apply the PaddleOCR patch. Weights live in OmniParser/weights/.

Stop

pkill Xephyr      # closes the sandbox (browser + WM + display)

Safety

  • The agent's input and vision are scoped to the sandbox display — it does not see or control your real desktop.

  • The server process runs as your user (shell/file access, like any MCP server); only its GUI control is sandboxed to the Xephyr window. For OS-level isolation from your files, run it inside a VM or container.

  • You can watch everything live and close it instantly with pkill Xephyr.

  • Automate responsibly — only sites and services you are authorized to use.

Files

  • server.py — the MCP server (all tools).

  • install.sh — mode-aware installer (none / basic / local).

  • start-display.sh — launches the Xephyr sandbox (display + window manager + browser).

  • AGENTS.md — drop-in tool-usage instructions for the AI agent (paste into its system prompt).

  • mcp-config.json — a ready-to-paste MCP client config snippet.

Credits

local vision mode uses Microsoft's OmniParser (cloned and weights downloaded by install.sh, under its own license). Screen capture uses mss; input is driven with xdotool. Not affiliated with Microsoft.

Contributing

Shipped as-is under MIT. Issues and PRs are welcome, but this is maintained by one person — no support or response time is guaranteed. If it's useful to you, a ⭐ helps.

License

MIT — see LICENSE.

Available Tools

19 tools
clickA

Click at absolute pixel coords (top-left origin) in an instance.

Returns {"ok", "action", "instance", "detail"}. button: 1=left, 2=middle, 3=right. observe='screenshot'|'parse' additionally returns the resulting screen state in the SAME call (saves a round-trip); settle_ms (max 5000) waits for the UI to update first — use 400–1500 after anything that navigates or submits. Fails (isError) if the instance is down.

ParametersJSON Schema
NameRequiredDescriptionDefault
xYes
yYes
buttonNo
observeNonone
instanceNo
settle_msNo

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description fully covers behavior: it describes the coordinate system (top-left origin), button mapping (1=left, 2=middle, 3=right), observe behavior (returns state in same call), settle_ms waiting (up to 5000ms), and failure condition (isError if instance down). This exceeds transparency expectations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is moderately concise with three sentences packing essential info. It front-loads the main action and parameters, but the second sentence about return values could be slightly more streamlined. Overall, every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 6 parameters, 0% schema coverage, and no output schema, the description is remarkably complete. It covers all parameters, return values, and failure semantics. No significant gaps remain for an AI agent to invoke correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema coverage, the description compensates fully: it explains x,y as absolute pixel coords, button as integer with meaning, observe as 'screenshot'/'parse' for state, settle_ms as wait time with suggested range, and instance as targeted instance. This adds significant meaning beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states 'Click at absolute pixel coords (top-left origin) in an instance', providing a specific verb ('click') and resource ('instance'). It distinguishes from sibling tools like 'click_element' (which clicks UI elements) and 'double_click', making the purpose clear.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives usage context: 'use 400–1500 after anything that navigates or submits' for settle_ms, and explains that observe values 'screenshot'/'parse' save a round-trip. However, it does not explicitly state when not to use this tool versus alternatives like 'click_element', though sibling names imply the distinction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

click_elementA

Click an element by id from that instance's most recent parse_screen (no coordinate guessing).

Returns {"ok", "action", "instance", "detail", "element", "center"}. Fails (isError) with the fix named if the id is unknown — run parse_screen(instance) first. Ids go stale when the screen changes: after navigation, re-parse before clicking. observe/settle_ms: see click().

ParametersJSON Schema
NameRequiredDescriptionDefault
idYes
buttonNo
observeNonone
instanceNo
settle_msNo

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses failure mode (isError with fix named), data staleness, and referential behavior for observe/settle_ms. With no annotations provided, the description fully bears the transparency burden and meets it well.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences front-load the core purpose and essential guidelines. However, it could benefit from a brief parameter explanation, but overall efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 5 parameters, no output schema, and no annotations, the description covers usage context and failure behavior but lacks parameter details and full return semantics. Adequate but could be more complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, yet the description does not explain the individual parameters (button, instance, observe, settle_ms). Only id usage is implied, and others are merely referenced to click(). This is insufficient.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states specific action (click), resource (element by id), and context (from parse_screen, no coordinate guessing). It effectively distinguishes from sibling 'click' which likely uses coordinates.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly instructs to run parse_screen first, explains staleness of ids, and provides guidance on failure recovery (fix named). Also references click() for observe/settle_ms parameters.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

close_instanceA

Close an instance: kill its app + its Xephyr display, and verify it actually went down.

Returns {"ok", "instance", "detail"} only after the display stops responding. Fails (isError) if the instance isn't running, if it survives SIGTERM, or for instance 1 — the main start-display.sh sandbox is protected (stop it from the host with pkill Xephyr).

ParametersJSON Schema
NameRequiredDescriptionDefault
instanceYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description fully discloses behavioral traits: it waits for the display to stop responding before returning, returns a structured response, and fails under specific conditions. It also highlights the protected nature of instance 1, providing critical behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured: a single sentence for the main action and return, followed by a sentence listing failure conditions. Every sentence adds essential information without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (one parameter, explicit output schema implied), the description covers the tool's purpose, return value, failure modes, and a special case. It is sufficiently complete for an AI agent to understand and use the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, and the description does not explicitly define the 'instance' parameter beyond mentioning instance 1's special behavior. While it adds context via an edge case, it lacks a clear definition of valid instance numbers or how to obtain them (e.g., from list_instances), partially compensating for the schema gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states the verb 'Close an instance' and details what that entails: killing its app and Xephyr display and verifying shutdown. It clearly distinguishes this from sibling tools like launch_app, status, or list_instances by specifying the termination action.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides guidance on when to use the tool (to close an instance) and explicitly states failure conditions—instance not running, survival of SIGTERM, or instance 1 being protected. It also gives an alternative method for instance 1 (stop from host with pkill Xephyr), offering clear usage boundaries.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

double_clickA

Double-click at absolute pixel coords in an instance (e.g. open a file icon, select a word).

Returns {"ok", "action", "instance", "detail"}. observe/settle_ms: see click(). Fails (isError) if the instance is down.

ParametersJSON Schema
NameRequiredDescriptionDefault
xYes
yYes
observeNonone
instanceNo
settle_msNo

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations, so description carries burden. It specifies return structure, failure condition (instance down), and references click() for two parameters. Does not detail side effects or potential destruction, but covers key behavioral aspects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Extremely concise: three sentences front-load purpose, then return format, then reference and failure. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequate for a simple tool: explains action, return, failure, and references for additional detail. Lacks parameter explanations and explicit sibling differentiation, but sufficient as minimal viable description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage 0%, description does not explain parameters beyond 'absolute pixel coords' for x/y. The observe, instance, and settle_ms parameters are mentioned but not described; refers to click() but no details here. Minimal added meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states 'Double-click at absolute pixel coords in an instance' with concrete examples like 'open a file icon, select a word'. Distinguishes from sibling 'click' (single click) and 'click_element' (by element).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides examples and references click() for observe/settle_ms, but does not explicitly state when to use versus alternatives like click_element or drag. Lacks exclusions or when-not-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

dragA

Drag (mouse-down → move → up) from (x1,y1) to (x2,y2) — for drawing, selecting, sliders.

Returns {"ok", "action", "instance", "detail"}. observe/settle_ms: see click(). Fails (isError) if the instance is down.

ParametersJSON Schema
NameRequiredDescriptionDefault
x1Yes
x2Yes
y1Yes
y2Yes
buttonNo
observeNonone
instanceNo
settle_msNo

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry full burden. It describes the sequence of events (mouse-down, move, up), return format, and a failure condition (instance is down). It references observe/settle_ms from click(), adding context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very concise (three sentences), front-loads the purpose and action, then covers return and failure. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

While the description covers core behavior and return values, it lacks detail on many optional parameters (button, observe, instance, settle_ms) and does not specify timeout or edge cases. With 8 parameters and no output schema, more completeness would be beneficial.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so description should add meaning. It explains x1,y1,x2,y2 as start and end points, and mentions button, observe, instance, settle_ms with a reference to click(). However, it does not elaborate on default values or behavior of all parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it performs a drag action (mouse-down → move → up) between two points, and lists use cases (drawing, selecting, sliders). This distinguishes it from siblings like click or move_mouse.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description mentions use cases but does not provide explicit when-to-use or when-not-to-use guidance, nor does it mention alternatives among siblings. The mention of failure when instance is down is a helpful constraint.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_screen_sizeA

Screen size of an instance + the active vision backend and server version.

Returns {"ok", "instance", "width", "height", "vision", "version"}. Call it once before coordinate math — coordinates are absolute pixels, top-left origin, within THIS instance's display. "vision" tells you how to ground: 'basic'/'local' → parse_screen + click_element; 'none' → screenshot + reason about pixels yourself. Fails (isError) if the instance is down.

ParametersJSON Schema
NameRequiredDescriptionDefault
instanceNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries the burden. It discloses return format, coordinate system (absolute pixels, top-left origin), and failure case. Does not explicitly state it's a read-only operation, but that is implied.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three concise sentences front-loading the key information: what it returns, when to call it, and how to interpret vision. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers return schema, usage context, coordinate system interpretation, and failure condition. For a tool with one optional parameter and output schema, this is fully complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so description must compensate. The parameter 'instance' is referenced in 'within THIS instance's display' but not explicitly listed or described. Does not explain default or behavior when omitted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it returns screen size, vision backend, and server version. It also distinguishes itself from siblings like screenshot and parse_screen by explaining how vision tells you which to use.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says 'Call it once before coordinate math' and provides conditional guidance on how to use based on the vision field. Also mentions failure condition (isError) when instance is down.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

launch_appA

Launch a GUI app in its OWN new Xephyr display/window and return its instance id.

command is any shell command ('chromium', 'xterm', 'gimp', 'inkscape file.svg', …). Chromium automatically gets X11 flags, a per-instance profile, downloads routed to files//, and a CDP debug port (needed by upload_file). Other GTK apps get D-Bus isolation so they render on this display, not the host.

Returns {"ok", "instance", "display", "name", "files_dir"} — pass that instance id to every control tool. Fails (isError) if Xephyr can't start on the host display, the command doesn't exist, or the app exits immediately. Each instance has its own cursor, so separate agents can drive separate instances in parallel; never share one instance between agents. The window appears on GLOVEBOX_HOST_DISPLAY (default :0). Give the app a moment to draw (settle_ms/wait_ms) before the first screenshot.

ParametersJSON Schema
NameRequiredDescriptionDefault
nameNo
sizeNo1440x900
commandYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully carries the burden. It details Xephyr display creation, Chromium special flags and CDP port, D-Bus isolation, return structure, failure conditions, and cursor separation for parallel agents.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with front-loaded purpose and separate details. Slightly verbose with some redundancy (Xephyr mentioned twice), but every sentence adds value. Could be trimmed slightly without losing clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (isolated display launch, parallel agents), the description is comprehensive. It explains return fields, failure modes, and usage caveats. The existing output schema is described in detail, so no gaps remain.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description thoroughly explains each parameter: 'command' is any shell command with examples, 'name' is optional with default empty, 'size' defaults to '1440x900'. This compensates fully for missing schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Launch', the resource 'GUI app in its OWN new Xephyr display/window', and the output 'return its instance id'. It effectively distinguishes from sibling tools like open_file (for files) and close_instance (for managing instances).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides concrete command examples, special handling for Chromium and other GTK apps, and warns against sharing instances between agents. Lacks explicit 'when not to use' direction, but context strongly implies it's for launching new app instances.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_filesA

The instance's staging folder (files// under the install dir) and its contents.

Returns {"ok", "instance", "dir", "count", "truncated", "files": […]} (capped at 200 entries, sorted). Drop files there (or reference any host path you can read) to open in apps via open_file() or an app's Open dialog; launch_app Chromium instances also download / 'save as' into it — call this after a download to confirm the file arrived. Browser uploads still go through upload_file().

ParametersJSON Schema
NameRequiredDescriptionDefault
instanceNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Since no annotations are provided, the description fully carries the burden. It details the return fields (ok, instance, dir, count, truncated, files), the 200-entry cap, sorting, and explains how files can be used by other tools (open_file, launch_app downloads). No behavior is left unclear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single paragraph that efficiently conveys the tool's purpose, return format, and usage context. It is front-loaded with the core function. Slightly dense, but every sentence adds value. Could be broken into bullet points for easier scanning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's role in a file management workflow, the description completely covers the return structure, limits, sorting, and relationships with siblings. The output schema exists, so return values are already documented; the description adds practical context for when and how to use the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single parameter instance is implied by the phrase 'The instance's staging folder' but is not explicitly described. Schema coverage is 0%, and the description does not explain that instance is an integer ID. Some compensation occurs through context, but a direct explanation would improve clarity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists the instance's staging folder contents and returns a file list. It distinguishes itself from siblings like open_file, upload_file, and launch_app by explaining how files in the staging folder are used and how the tool fits into workflows.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit guidance is provided: call this after a download to confirm file arrival; browser uploads still go through upload_file. The description also notes the return structure with a 200-entry cap and sorting, giving the agent clear when-to-use context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_instancesA

List all instances (running app windows): id, display, name, command, up.

Returns {"ok", "count", "instances": [{instance, display, name, command, up}, …]}. Probes displays :1–:12 for sandboxes started outside this server (e.g. start-display.sh), so it also finds instances this process didn't launch (their name/command show as unknown). Use it to discover what you can drive and to verify an instance is up before acting.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description fully covers behavioral traits: it reveals that the tool probes displays :1–:12 to find sandboxes started externally, notes that instances not launched by this process show 'unknown' name/command, and describes the exact JSON return structure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with two clear paragraphs: first defines the tool and its output, second explains an important behavioral nuance. Every sentence adds value without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has no parameters and an output schema exists, the description still explains the return format and key behavioral details (external instance detection). This is complete and sufficient for an agent to use the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so per guidelines the baseline is 4. Since there are no parameters to describe, the description appropriately adds no parameter information.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists all running app windows (instances) and specifies the returned fields (id, display, name, command, up). It is a unique resource distinct from sibling tools like click or launch_app, making its purpose unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage context: 'Use it to discover what you can drive and to verify an instance is up before acting.' However, it does not explicitly mention when not to use it or contrast with alternatives, so it lacks exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

move_mouseA

Move the mouse pointer to absolute pixel coords in an instance (no click).

Returns {"ok", "action", "instance", "detail"}. Useful for hover states (menus, tooltips) — follow with screenshot() to see the effect. Fails (isError) if the instance is down.

ParametersJSON Schema
NameRequiredDescriptionDefault
xYes
yYes
instanceNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description discloses that it does not click, returns a specific structure, fails if instance is down, and is useful for hover states. This is sufficient for a simple pointer move.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three concise sentences, front-loaded with purpose, each adding value: behavior, return, use case, and failure condition. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with 3 parameters and no enums, the description covers purpose, return values, use case, and error condition, making it complete for an agent to select and invoke correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so description should compensate. It mentions 'absolute pixel coords' for x and y and refers to 'instance', but does not detail ranges or units. The schema itself is clear enough for simple integer coordinates.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool moves the mouse pointer to absolute pixel coordinates without clicking, and distinguishes it from sibling tools like click and double_click.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a clear use case for hover states and suggests following with screenshot, but does not explicitly mention when not to use or name alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

open_fileA

Open a local file in an app ON an existing instance's display (NOT a new instance).

If app is given, runs <app> <filepath> there (e.g. app='gimp'); otherwise tries xdg-open. Relative paths resolve against the instance's staging folder files// (see list_files). Returns {"ok", "action", "instance", "opened", "app"}; fails (isError) if the file or the app doesn't exist. The app is started asynchronously — screenshot after a moment to confirm it drew. For BROWSER uploads use upload_file() instead. GTK apps get the same X11 + D-Bus handling as launch_app (so they render on this display, not the host).

ParametersJSON Schema
NameRequiredDescriptionDefault
appNo
filepathYes
instanceNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully discloses behavioral traits: it opens on existing display, runs specified app or xdg-open, resolves relative paths, returns specific fields, fails if file/app doesn't exist, starts asynchronously (suggesting screenshot confirmation), and notes GTK apps get same handling as launch_app. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is reasonably concise with two paragraphs. The first sentence front-loads the core purpose and key constraint. Every sentence serves a purpose, though the structure could be slightly more streamlined (e.g., grouping related details).

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (3 params, no annotations, output schema exists), the description is complete: it covers purpose, usage guidelines, parameter behavior, failure modes, async nature, and alternatives. The output schema is referenced by naming returned fields, fulfilling completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Despite 0% schema description coverage, the description adds meaningful semantics: it explains the app parameter (optional, runs that app), filepath (required, resolves relative to staging folder), and implicitly references instance (existing display). It does not detail the instance parameter's type or default, but the schema covers those.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool opens a local file in an app on an existing instance's display, explicitly distinguishing it from creating a new instance. It specifies the exact resource (local file) and context (existing display), making the purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use this tool versus alternatives: it says to use upload_file() for browser file uploads, and contrasts with launching a new instance. It also explains relative path resolution against the instance's staging folder, referencing list_files for context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

parse_screenA

Detect on-screen elements and return them with ids + pixel-centers (vision grounding).

Returns {"ok", "vision", "screen": [W,H], "annotated", "count", "truncated", "elements": [{id, type, label, interactive, center: [x,y]}, …]} — then click_element(id). Also saves a numbered Set-of-Mark image to /tmp/glovebox_annotated_.png. Backend = GLOVEBOX_VISION: 'local' (OmniParser, text + icons, ~2 s on GPU after a ~6 s first-call model load), 'basic' (tesseract, text only), 'none' (returns a note telling you to screenshot + reason about pixels yourself — this is normal, not an error). Elements are capped at 300 per call ("truncated": true when more were detected). Ids are only valid until the screen changes — re-parse after navigation.

ParametersJSON Schema
NameRequiredDescriptionDefault
instanceNo
box_thresholdNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully discloses behavior: it returns a detailed JSON structure, saves a Set-of-Mark image, explains backend options, element cap of 300 with truncation flag, and id validity scope. This is comprehensive for an agent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is relatively long but well-structured with bullet-like formatting and clear sections. Each part adds value, though it could be slightly more concise by removing redundant phrasing.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the complexity (multiple backends, return format, image saving, truncation, id validity), the description is quite complete. However, it lacks explanation of the parameters, which reduces completeness for a tool with two undocumented parameters. The output schema exists to document return values.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, meaning the schema itself provides no parameter descriptions. The tool description does not explain the two parameters (instance, box_threshold) at all, leaving the agent without guidance on what they control. This is a significant gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it detects on-screen elements and returns them with ids and pixel centers. It uses a specific verb 'detect' and resource 'on-screen elements', and distinguishes from sibling tools like click_element (which uses the ids) and screenshot (which is a raw image).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on when to use this tool (to detect elements for clicking) and explains backends (local, basic, none) with their trade-offs. It also notes that ids are only valid until screen changes, implying re-parse after navigation, but does not explicitly state when not to use it versus alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

press_keysA

Press a key or combo (xdotool syntax): 'Return', 'ctrl+a', 'Tab', 'ctrl+t', 'ctrl+l', 'F5'.

Returns {"ok", "action", "instance", "detail"}. Fails (isError) on an invalid keysym or a dead instance. Browser navigation: 'ctrl+l' → 'ctrl+a' → type_text(url) → 'Return'. observe/settle_ms: see click().

ParametersJSON Schema
NameRequiredDescriptionDefault
keysYes
observeNonone
instanceNo
settle_msNo

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

In the absence of annotations, the description covers return format (JSON with ok, action, instance, detail), failure conditions, and references observe/settle_ms as used in click(). It does not detail behavior for default parameter values or all edge cases, but provides sufficient transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with 3-4 sentences, front-loads the action, and includes examples and a workflow tip. No redundant information, but the reference to click() could be more self-contained.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 4 parameters and no output schema, the description covers the primary purpose, examples, return format, failure cases, and a usage scenario. It lacks detailed parameter descriptions but is adequate for a simple key-press tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, so the description must add meaning. It explains the 'keys' parameter with valid examples and syntax, but does not describe 'observe', 'instance', or 'settle_ms' in detail (just references click()). This adds partial value beyond schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states that the tool presses a key or combo using xdotool syntax, and provides specific examples like 'Return', 'ctrl+a', etc. It distinguishes itself from sibling tools like click, double_click, drag, scroll, and type_text by focusing on key presses.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit usage context, such as browser navigation sequence ('ctrl+l' → 'ctrl+a' → type_text(url) → 'Return'), and notes failure conditions (invalid keysym or dead instance). However, it does not explicitly exclude alternative tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

screenshotA

Screenshot of an instance's window (PNG image, full display).

The ONLY way to see the screen — always look before you act. For an element list with coordinates use parse_screen (basic/local vision). Prefer observe='screenshot' on action tools to act and see the result in one call. Fails (isError) if the instance is down.

ParametersJSON Schema
NameRequiredDescriptionDefault
instanceNo

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Despite no annotations, the description discloses that it is a read-only snapshot operation, notes it fails if the instance is down, and implies it is non-destructive. However, it does not explicitly state it does not modify state or require special permissions, but the nature of a screenshot makes that clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four concise sentences with key information front-loaded: purpose, alternatives, and failure condition. No redundant or extraneous content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple screenshot tool with one parameter and no output schema, the description covers purpose, usage guidance, failure condition, and return format (PNG). It is fully sufficient to enable correct use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only one parameter (instance) with 0% schema coverage. The description mentions 'an instance's window' hinting that the parameter identifies the instance, but does not explain its default value or range. Barely compensates for the lack of schema documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states it takes a full-display PNG screenshot of an instance's window. Explicitly distinguishes itself as the only way to see the screen and contrasts with parse_screen and observe='screenshot' on action tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit guidance: 'always look before you act', recommends using parse_screen for element list with coordinates, and suggests prefering observe='screenshot' on action tools to act and see result in one call. Clearly outlines when to use this tool versus alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrollA

Scroll the wheel at the current mouse position: positive = up, negative = down.

Returns {"ok", "action", "instance", "detail"}. One unit ≈ one wheel notch (a few lines). Scrolling targets the window under the pointer — move_mouse/click there first. Fails (isError) on amount=0 (a no-op) or a dead instance. observe/settle_ms: see click().

ParametersJSON Schema
NameRequiredDescriptionDefault
amountYes
observeNonone
instanceNo
settle_msNo

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully discloses behavior: direction semantics, return format, unit definition, prerequisite actions, error conditions, and references to click() for observe/settle_ms. This exceeds the burden.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is compact (two sentences plus a note about returns and error conditions). Every sentence provides value, no fluff. Front-loaded with the core action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and moderate complexity, the description covers return format, behavioral nuances (unit, prerequisite, errors), and parameter interpretation. It is complete enough for an agent to use correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Since schema has 0% parameter descriptions, the description adds crucial meaning: amount direction/unit, observe/settle_ms reference to click(). Instance is not explained but defaults to 1. Significant value added, but instance could be clarified.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool scrolls the wheel at the current mouse position, specifying positive/negative direction. It distinguishes from sibling tools like click, drag, and move_mouse by focusing on wheel action.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides guidance: scrolling targets the window under the pointer, and recommends moving the mouse/clicking there first. It also notes failure conditions (amount=0, dead instance). Implicitly suggests not using without prior pointer positioning, but lacks explicit alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

statusA

Server + sandbox status in one read-only call — run this first, and when filing bug reports.

Returns {"ok", "version", "vision", "host_display", "instances": [...same as list_instances], "deps": {xdotool, xclip, xephyr, wmctrl, tesseract, omniparser_weights: true|false}}. "vision" is the grounding backend (none|basic|local); "deps" shows whether each system tool is installed and (for 'local') whether the OmniParser weights are present. No side effects.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Explicitly declares 'No side effects' and read-only nature. Details the structure of the response, including fields like 'vision' and 'deps', with explanations of their meaning.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Concise and front-loaded: first sentence gives the core purpose, followed by a succinct breakdown of the return value. Every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given zero parameters and the presence of an output schema, the description fully covers the tool's behavior, including what the output contains and that it has no side effects.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

No parameters exist (input schema has zero properties, 100% coverage), so the description does not need to add parameter explanations. Baseline 4 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states it returns combined server and sandbox status in a single read-only call. It is distinct from sibling tools like click, screenshot, etc., which are not status-related.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly recommends 'run this first' and 'when filing bug reports', providing clear use cases. No alternative tools needed for status queries.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

type_textA

Type into the focused field — unicode-safe. Click the field first to focus it.

Returns {"ok", "action", "instance", "detail", "method": "xdotool"|"clipboard"}. Pure ASCII is typed via xdotool; text with any non-ASCII char (č/š/ž …) is inserted via the clipboard + ctrl+v, because xdotool's synthetic unicode keystrokes get silently dropped by some toolkits (e.g. Inkscape's GTK canvas) even though they work in browsers. Fails (isError) rather than silently losing characters if xclip is missing for non-ASCII text. observe/settle_ms: see click().

ParametersJSON Schema
NameRequiredDescriptionDefault
textYes
observeNonone
instanceNo
settle_msNo

TDQS

A3.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully discloses the dual method (xdotool vs clipboard), the rationale for the clipboard fallback (toolkit compatibility), and the fallback failure behavior (isError if xclip missing). Also documents the return format.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with the main purpose front-loaded. Slightly verbose in explaining the unicode workaround, but the detail is necessary for transparency.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Good coverage of core behavior and failure modes, but lack of detail on observe/instance/settle_ms parameters leaves a gap. Since there is no output schema, the return format is described, which is helpful.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, yet only 'text' is explained (typed input). The other three parameters (observe, instance, settle_ms) are merely referenced with a direction to 'see click()', providing no independent semantics. This is insufficient.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states 'Type into the focused field — unicode-safe', which clearly indicates the action (type) and resource (focused field). It distinguishes from siblings like click or press_keys, though not explicitly naming them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly instructs to click the field first to focus it. Provides context on when the clipboard fallback is used (non-ASCII characters) and failure conditions (missing xclip). Does not list when to avoid this tool in favor of alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

upload_fileA

Attach a local file to a page's via Chrome DevTools Protocol — use this for ALL browser uploads (logos, images, docs).

The sandbox's native GTK file picker is invisible to us AND hangs the renderer, so never click an upload button expecting a dialog — call this instead. Works on Chromium started by launch_app / start-display.sh (they open a per-instance --remote-debugging-port, 9222+N). selector targets the file input on the MAIN page (default = first file input). If the site only inserts the after you click its 'upload' control, click that first (it won't open a dialog we can see, but it wires up the input), then call this. Returns {"ok", "action", "instance", "detail"}; fails (isError) with the reason if the file is missing, no CDP target answers, or the selector matches nothing. NATIVE (non-browser) apps: use open_file() or drive the app's own Open dialog — it IS visible here. observe/settle_ms: see click().

ParametersJSON Schema
NameRequiredDescriptionDefault
observeNonone
filepathYes
instanceNo
selectorNoinput[type=file]
settle_msNo

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, but description fully covers behavioral traits: uses CDP, requires Chromium started by launch_app, selector targets main page, advises clicking first if input is dynamic, and details return format/failure reasons. Adds context on observe/settle_ms referencing click().

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Every sentence adds value: purpose, when to use, how to use, edge cases, return values, and references. Front-loaded with main action. No filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 5 parameters, no output schema, and no annotations, description provides complete context: prerequisites (Chromium, launch_app), dynamic input handling, return format, failure reasons, and distinction from sibling tools (open_file). No gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Despite 0% schema description coverage, description explains each parameter: filepath (required local path), selector (default first input), instance (integer), observe and settle_ms (see click()). Adds meaning beyond the schema definitions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description explicitly states the tool attaches a file to a page's file input via CDP, distinguishes from native file picker, and specifies its use for all browser uploads. Verb 'attach' and resource 'file input' are clear.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly advises against clicking upload buttons for dialogs (since native picker is invisible and hangs renderer), prescribes using this tool instead. Also contrasts with native apps: suggests open_file() for visible dialogs. Clear when-not and alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

wait_msA

Wait (e.g. for a page to load or an app to draw) — capped at 10000 ms.

Returns {"ok", "action", "requested_ms", "waited_ms", "clamped"} — "clamped": true means you asked for more than the cap and only waited_ms elapsed. Prefer settle_ms on the action itself when you also want to observe the result in the same call.

ParametersJSON Schema
NameRequiredDescriptionDefault
msYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, but description fully discloses behavior: cap at 10000 ms, returns {'ok', 'action', 'requested_ms', 'waited_ms', 'clamped'}, explains 'clamped' flag meaning when cap exceeded. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences: first defines purpose and cap, second explains return structure and alternative. No redundant words, information is front-loaded and efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple wait tool, description covers return values, behavior at cap, and alternative (settle_ms). No output schema visible but description provides details. Complete for its complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only one parameter 'ms' (integer) with 0% schema description coverage. Description adds the cap constraint (10000 ms) but does not explicitly state the parameter represents milliseconds. The tool name 'wait_ms' implies milliseconds, but description could be clearer.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states it's a 'wait' tool for page load or app draw. The verb 'wait' and resource implied (time) with specific use cases. Differentiates from siblings by mentioning 'settle_ms' as an alternative for observing results in the same call.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides clear context: wait for page load or app draw, capped at 10000 ms. Suggests preferring 'settle_ms' on the action to observe the result simultaneously. Lacks explicit when-not-to-use but covers key usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

A4.4/5.0
Disambiguation5/5

Each tool has a distinct, well-defined purpose. Clicking, dragging, scrolling, typing, parsing, etc. are all separate actions without overlap. Even similar tools like click and click_element are differentiated by coordinate vs element id.

Naming Consistency4/5

Most tools follow a verb_noun pattern (click_element, launch_app, type_text), but a few are just verbs (click, drag, scroll, screenshot, status). However, the pattern is still readable and predictable.

Tool Count5/5

19 tools is well-scoped for a GUI automation server. It covers launching, controlling, inspecting, and managing instances without being overly large or too minimal.

Completeness5/5

The tool set covers the full lifecycle: create/close instances, list them, interact via clicks, keys, type, scroll, drag, parse screen, get screenshots, handle files, and upload. No obvious gaps for typical automation tasks.

Maintenance

ActivitySlowing
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    B
    quality
    D
    maintenance
    An MCP server that provides AI agents with a full Ubuntu desktop environment inside Docker, enabling them to perform complex computer tasks like browsing, coding, testing, and GUI automation.
    36
    8
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    An MCP server that gives any AI assistant eyes and hands on your desktop — screenshots, clicking, typing, OCR, window management, accessibility-tree queries, workflow recording.
    5
    Apache 2.0
  • A
    license
    A
    quality
    A
    maintenance
    A headless, agent-controllable real browser as an MCP server that enables AI agents to navigate, click, fill, eval JavaScript, and take screenshots on localhost and allowed hosts, with no GUI required.
    27
    4
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    A framework-agnostic computer-use MCP server that exposes core desktop operations (screen capture, mouse, keyboard, and file access) as standard MCP tools, enabling any MCP-compatible agent to drive a computer.
    327
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/segentic-lab/glovebox-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server