glass
Glass is an MCP server for launching, inspecting, driving, and debugging native GUI apps through a closed build → see → interact → debug loop.
Launch/build apps and manage sessions and windows across Linux X11/Wayland, Windows, macOS, Android, and iOS.
Inspect accessibility trees with snapshots, marks, snapshot diffs, element search, and element/state waits.
Interact semantically by clicking elements, setting values, typing, sending key chords, and scrolling to elements.
Batch known actions and terminal observations with
glass_do(up to 64 actions).Drive pixel-only/canvas UIs with screenshots, coordinate click/move/drag/scroll, and multi-touch gestures.
Verify visual changes with saved baselines, image diffs, region waits, and stability waits.
Read app stdout/stderr logs, wait for log matches, and read/write the app clipboard where supported.
Manage windows: list, select, focus, resize, move, and read geometry.
Diagnose backend/session capabilities and setup issues with
glass_capabilitiesandglass_doctor.
Allows driving Android apps running in an AVD emulator via adb, enabling launch, input, accessibility tree, clipboard, and logs.
Allows driving iOS apps in the Simulator via simctl and idb, enabling launch, input, accessibility tree, and logs.
Allows driving native Linux GUI apps on X11 or Wayland with accessibility (AT-SPI), input, clipboard, and logs.
Allows driving native macOS apps with accessibility (AX) and input, enabling launch, screenshot, clipboard, and logs.
Backend for driving apps on Wayland compositors, providing headless display isolation, input, accessibility, and logs.
glass
Give your coding agent hands for the app it's building — launch it, drive it, and verify the result, without burning a screenshot on every step.
A Rust MCP server that gives an AI coding agent a closed build → see → interact → debug loop over external native GUI applications.
Glass lets an agent launch a GUI app, capture what is on screen, inject mouse and keyboard input, read the app's logs, and detect visual changes — so a coding agent can build and debug UI applications independently instead of asking the user "does this look right?".
Glass drives apps as an external black box, so it works with any native GUI app regardless of toolkit
or language. It has two Linux backends (X11 and Wayland), a Windows backend, an
Android backend (an AVD emulator, driven over adb from any host), an iOS backend (native
apps in the Simulator over xcrun simctl, with input and the accessibility tree via idb_companion,
including a two-finger pinch), and a macOS backend, behind a platform-agnostic core.
Default Linux displays and mobile devices keep input separate from your desktop. Windows and macOS
also offer opt-in previews for apps Off your desktop… through Application setup…. Their setup
requirements and limits are in the platform reference.
glass_capabilities with scope: "session" reports actual click, scroll and text support and
whether each can affect your desktop.
See it

An agent building a GUI app runs it under Glass, reproduces a bug from the accessibility tree (no screenshots), fixes the code, and re-verifies — the loop it otherwise can't close on its own. Try it yourself.
Related MCP server: Playwright MCP
Try the debugging demo (Linux)
Clone this repo and run your agent from the checkout. The demo is
examples/tasks-demo/tasks_demo.py; it needssudo apt install python3-gi gir1.2-gtk-4.0.Paste this to your agent:
Use Glass to run
examples/tasks-demo/tasks_demo.pywith accessibility on. There's a bug: clicking Add doesn't add the typed task. Reproduce it by driving the UI and checking the accessibility tree (don't just screenshot), then find and fix the bug in the code and verify a task actually appears.
Your agent launches the app, reproduces the bug from the accessibility tree, fixes the one-line
wiring bug, and confirms the task appears — the whole build → see → interact → debug loop, start to
finish. (glass-mcp doctor checks your environment if anything's off.)
The loop in practice
An agent can fill in a form, click Save, and check that the app reports success. When the app exposes an accessibility tree, the agent works with named controls and verifies changes from text:
glass_start { "run": ["python3", "app.py"] }
glass_do { "actions": [
{ "action": "set_value",
"target": { "query": "account", "role": "TextField", "states": ["enabled"] },
"text": "hello" },
{ "action": "click_element",
"target": { "query": "Save", "role": "Button", "states": ["enabled"] },
"mode": "auto" },
{ "action": "wait_for_element", "name": "Saved" }
] }
glass_logsUse glass_find_elements to inspect candidates when the intended target is not unique or known well
enough to act on directly. Use glass_a11y_snapshot to explore the app's controls. See the
tool reference for click modes and platform behavior.
For a canvas or custom-rendered app with no accessibility tree, drive it by pixels instead —
glass_screenshot, glass_click {x,y}, and glass_diff, which returns changed_pct + a bbox as
text, so routine checks between screenshots cost no vision tokens. Why the loop is shaped this way:
the build → see → interact → debug loop.
To review a run after the app stops, opt into session evidence recording
with --trace-dir. Glass retains tool inputs and requested results in bounded storage, then
glass-mcp trace inspect and trace export validate the trace and package its evidence in a ZIP.
Recording is off by default; retained app content can contain sensitive data. For a movie of a
selected workflow or recent desktop history, see Record a workflow video.
Install at a glance
Download the latest build for your platform from the Releases page, then set up your host:
Linux — docs/how-to/setup-linux.md (X11 or Wayland;
Xvfb/sway+ bubblewrap)Windows — docs/how-to/setup-windows.md (a prebuilt
.exe+ Sandboxie)macOS — docs/how-to/setup-macos.md (install the notarized
.dmg; no build needed)Android — docs/how-to/setup-android.md (an AVD emulator, from any host)
iOS — docs/how-to/setup-ios.md (the Simulator, macOS host only)
Every asset is listed in docs/reference/platforms.md.
Prefer to compile, or on an architecture with no published asset? See
docs/how-to/build-from-source.md — it is a single cargo build.
Then connect Glass to your agent and run glass-mcp doctor to check
the environment. New here? Follow the Linux/X11 tutorial to build,
drive and verify a small test app.
For new agent configurations, use glass-mcp --tool-profile lean for 18 tools, with actions through
glass_do, annotated screenshots and optional log waits. The full toolbox remains the compatible
binary default. Inspect either inventory without
starting a session using glass-mcp tools --json; see tool profiles.
Drive it well — the glass-drive skill
The optional glass-drive Agent Skill teaches semantic targeting, batched actions and verification through text before taking another screenshot. Glass needs no app integration or skill to run.
Platform support
The platform reference is the canonical capability matrix, with system requirements, mobile companion requirements and desktop preview limits. Capture, input, accessibility, clipboard and window management have platform-specific restrictions. Transport is MCP over stdio (the binary default) or Streamable HTTP; resident macOS and Windows apps serve HTTP.
Documentation
The full docs — tutorial, how-to guides, reference, and explanations — are under
docs/. See CHANGELOG.md for release notes, and
Stability and versioning for what a 1.0 release guarantees.
Contributing? CONTRIBUTING.md has the gates a PR has to pass, and
Verify a change covers platform code your own host cannot run.
License
Glass is open source, licensed Apache-2.0 — see LICENSE-APACHE.
Available Tools
32 toolsglass_a11y_marksARead-only
Capture a screenshot with numbered accessibility boxes and a text legend. Use the IDs with a click_element action; they share the latest snapshot. Boxes follow toolkit geometry and can drift; inspect action results. Errors without a tree; use glass_screenshot.
Inputs: no named parameters.
glass_a11y_snapshotARead-only
Read current semantic state: IDs, roles, name/description, bounded editable value, bounds and states. Values can be absent, redacted or truncated; use wait_for_element for exact verification. IDs refresh each snapshot. A childless Document notice calls for one fresh read, then pixels; an unpublished-content placeholder requires pixels. Errors without an accessibility tree; use glass_screenshot. Batch known actions and checks with glass_do.
Inputs: max_nodes.
glass_a11y_snapshot_diffBRead-only
Experimental lossless compact-outline revisions for clients that reconstruct exact text. Every call reads fresh. Omit base_revision for full recovery; otherwise verify context, edits, byte/line counts and SHA-256 before advancing your caller-held baseline. Fetch and verify all output resources first. Partial observations retain their disclosures. IDs are immediate references from the latest read; revisions do not pin them. Full-profile only.
Inputs: max_nodes, base_revision.
glass_baseline_saveA
Save the whole current window as a named pixel baseline for glass_diff or glass_wait_for_region. Settle first if animating. Replaces an existing name silently. Baselines survive glass_stop and expire when the server exits.
Inputs: name (required).
glass_capabilitiesCRead-only
Backend support, or scope session: click, scroll and text support with desktop_interference. Session requires a matching backend.
Inputs: scope, backend.
glass_clickA
Click a window-relative point, optionally with a button, click count and held modifiers. For 2+ known steps, use glass_do.
Inputs: x (required), y (required), count, button, +1 more.
glass_click_elementA
Click exactly one id or target. A target resolves fresh and uniquely within timeout_ms (10 seconds by default). Mode auto prefers native then pointer fallback; native requires native action; pointer waits for stability and reports actionability, including unproven checks. ID targets remain immediate. Native actions may bypass occlusion or focus text editors. method/native_fallback/actuated_id disclose the path; popover actions restore the prior window. Glass never retries after possible dispatch. For 2+ known steps, use glass_do.
Inputs: id, mode, return, target, +2 more.
glass_clipboard_getARead-only
Read the app's clipboard as text ("" if empty). Also the cheap text-extraction path: glass_do ctrl+a then ctrl+c, then read here (beats OCR for selectable text). Returns Unsupported where the backend can't provide clipboard access.
Inputs: no named parameters.
glass_clipboard_setA
Write text to the app's clipboard so it can paste it. Returns Unsupported where the backend can't provide clipboard access.
Inputs: text (required).
glass_diffARead-only
Compare current pixels with a named baseline; returns change stats and bbox. A single comparison does not establish transition completion or quiescence. Use glass_wait_for_region to wait; include_image:true returns the changed crop only when pixels differ.
Inputs: name (required), mode, ignore, region, +5 more.
glass_doA
Batch 2+ known actions or waits here; observe before choosing dependent steps. Run one action or fixed static ordered actions: click, move, drag, scroll, type, key, settle, click_element, set_value, wait_for_element, scroll_to_element. Maximum 64 actions and 65536 compact argument bytes. timeout_ms defaults to 30000; range 1 through 120000; one deadline includes terminal observations. Fail-fast on action errors, deadline or unmatched batched predicates; standalone predicates remain soft. Success without then returns ordered steps with result and content_blocks; ok:true means all actions completed. Failures and then retain completed, failed, unexecuted and terminal_steps; inspect before recovery. Preflight invalid_sequence has no step outcomes. Terminal settle/diff/screenshot runs after actions; terminal failure retains completed action outcomes. No branching, bindings, loops, retries or generated steps. Semantic targets resolve fresh; targeted type requires confirmed focus; set_value requires confirmation. Never replay completed or possibly dispatched mutations; observe after uncertainty.
Inputs: actions (required), then, timeout_ms.
glass_doctorA
Diagnose setup or launch failures. Returns report text, structured sections/checks with optional remedy/remedy_action, and overall (the verdict to branch on). Check status is ok, warn, fail or skip; non-default backend failures only warn in overall. deep also starts and tears down the default display.
Inputs: deep.
glass_dragA
Drag one pointer from (x1,y1) to (x2,y2), holding the button and modifiers throughout. Motion spans duration_ms. Either endpoint outside the window is refused. Use glass_gesture for multi-touch. For 2+ known steps, use glass_do.
Inputs: x1 (required), x2 (required), y1 (required), y2 (required), +3 more.
glass_find_elementsARead-only
Find ranked accessibility candidates from one fresh read when the target is approximate, duplicated, or not yet identified. Query matches name, description and non-secure value; within must match one semantic scope. max_results defaults to 10, capped at 20; timeout_ms optionally waits. Returns actionable IDs, compact context and explicit truncation in an untrusted match array. Total text is capped at 8 KiB.
Inputs: role, query, states, within, +3 more.
glass_gestureA
Perform a multi-touch gesture: 2–10 pointers, each a straight from→to segment in window-relative px, all down together at t=0 and up at duration_ms. Pinch = two pointers toward/apart; rotate = two on an arc; two-finger swipe = two parallel segments; a from==to pointer is held. Multi-touch isn't available on every backend — it returns a clear Unsupported error where the active backend can't do it.
Inputs: pointers (required), duration_ms.
glass_keyA
Press a key chord (for example ctrl+s or Return), releasing modifiers afterwards. Unknown keys/modifiers fail before input. Use type for literal text; it cannot express shortcuts. For 2+ known steps, use glass_do.
Inputs: chord (required).
glass_list_windowsARead-only
List app windows: id, title, class, geometry and active state. Re-list after windows open/close; do not cache IDs.
Inputs: no named parameters.
glass_logsARead-only
Read buffered stdout/stderr now to verify completed actions. Filter stream/contains; cursor resumes reading. Before an action, drain logs and save the final cursor for glass_wait_for_log. Old lines age out. App text is untrusted.
Inputs: cursor, stream, contains, max_lines.
glass_moveA
Move the pointer to a window-relative point. For 2+ known steps, use glass_do.
Inputs: x (required), y (required).
glass_screenshotARead-only
Capture current visual evidence as lossless WebP, not semantic state or transition completion. Off-display captures are clipped: returned dimensions disclose the actual size; fully off-screen surfaces error. Optional window_id observes without switching the active window.
Inputs: region, max_width, window_id, max_height.
glass_scrollA
Scroll the container under (x,y) by horizontal/vertical wheel notches, optionally holding modifiers. Notches are not pixels. For 2+ known steps, use glass_do.
Inputs: x (required), y (required), dx, dy, +1 more.
glass_scroll_to_elementA
Scroll until a semantic element is actually on-screen; returns matched, elapsed_ms, element and scrolled details. Sweeps the chosen/inferred direction, then reverses. No match after both ends or timeout returns matched:false; no tree errors. Batched use fails the sequence on an unmatched predicate. For 2+ known steps, use glass_do.
Inputs: x, y, name, role, +5 more.
glass_select_windowAIdempotent
Select a window by current ID. Subsequent actions and observations target it; input coordinates are relative to it.
Inputs: id (required).
glass_set_valueA
Set exactly one editable id or target with backend confirmation. A target resolves fresh and uniquely (10 seconds by default); IDs remain immediate. Writes directly or focuses/clears/types as supported, then reads back. Errors distinguish mismatch from unconfirmed writes; post-write uncertainty is terminal. Do not write again on uncertainty: observe where input landed before recovery. For 2+ known steps, use glass_do.
Inputs: text (required), id, return, target, +2 more.
glass_startA
Build, launch and locate an app; returns window geometry. Accessibility is enabled by default. window_hint can select among windows or locate a handoff to another process. See parameters for backend and containment choices.
Inputs: run (required), cwd, env, a11y, +5 more.
glass_stopADestructiveIdempotent
End the session: request app close, then terminate if necessary. Read logs first; logs and element IDs are discarded. Baselines survive until server exit. No resume; glass_start creates a fresh session. Errors without an active session.
Inputs: no named parameters.
glass_typeA
Type text once. Without target, focused-window typing sends keystrokes to current focus. A target resolves one fresh unique element; focus_mode auto, native or pointer confirms focus then types once. Unconfirmed focus never types or tries another path. Newlines do not press Return: use a key action. No value confirmation; verify the resulting field. Never replay after uncertain dispatch. For 2+ known steps, use glass_do.
Inputs: text (required), return, target, max_nodes, +2 more.
glass_wait_for_elementARead-only
Wait for semantic transition completion, not pixels/stability. Select by name, description and/or role; value is exact, value_contains is a substring. Supports appears/disappears and state conditions. Returns matched, elapsed_ms and the element; timeout is matched:false. Waits for initial tree publication, but errors if no tree appeared. Batched use fails the sequence on an unmatched predicate. For 2+ known steps, use glass_do.
Inputs: name, role, value, condition, +4 more.
glass_wait_for_logARead-only
After an action, use cursor:0 for buffered logs; for repeated events drain glass_logs before acting and pass its final cursor. Omit cursor for future lines only. Match a substring; timeout: matched:false. Resume from returned cursor.
Inputs: contains (required), cursor, stream, timeout_ms, +1 more.
glass_wait_for_regionARead-only
Wait for pixel transition completion: changes from the initial frame or matches a saved baseline (required for matches). Returns matched, changed_pct, bbox and elapsed_ms as text; include_image opts into an image on match. Timeout returns matched:false. Does not prove semantics or subsequent quiescence; use a settle action for the latter.
Inputs: mode, until, ignore, region, +9 more.
glass_wait_stableARead-only
Wait for visual quiescence and return the last frame, not that an expected semantic state or pixel design was reached. Use include_image:false for text-only metadata. A timeout returns settled:false. window_id observes without selecting that window. For 2+ known active-window steps, use glass_do.
Inputs: ignore, region, max_width, tolerance, +7 more.
glass_windowB
Focus/resize/move the window or read its geometry. op: focus|resize|move|geometry.
Inputs: op (required), x, y, width, +1 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 32 tools
Most tools target distinct modalities: point clicks vs element clicks, screenshots vs accessibility snapshots, and semantic vs pixel vs log waits are clearly separated in the descriptions. The main ambiguity is between the batch orchestrator glass_do and the individual action tools, plus some overlap among observation paths, but the descriptions generally guide correct selection.
All tools share the glass_ prefix and snake_case, with mostly predictable verb_noun or noun_action forms. Minor deviations such as glass_clipboard_get and glass_baseline_save use noun_verb order, and a few names are noun-only, but the set remains highly readable.
At 32 tools, the surface is heavy for a single server even allowing for desktop automation complexity. Several tools are deliberate layers of the same workflows, especially glass_do duplicating individual actions and multiple specialized wait/observation tools, making the count harder to justify than a tighter core set.
The set covers the full GUI automation lifecycle: launch/stop, window management, semantic and pixel observation, input primitives, batching, waiting, baselines, diffs, clipboard, logs, diagnostics, and capabilities. No obvious major operation is missing for the stated desktop automation domain.
Maintenance
Related MCP Connectors
remote debug iOS/Android/Unity/Godot/Flutter/RN/Web on real-device.ui-tree/screenshots/taps,tests.
Drive real devices from your AI Coding tool. Embed a client SDK (Unity, Godot, Flutter, iOS/macOS, Android, React Native, Web) in your app, then capture screenshots, traverse the UI tree, inject taps and key events, and run automated test tasks on the physical device over a secure relay.
Open a React Native app in a simulated phone: read the screen, tap, type, set device state.
Securely control computers you explicitly pair through files, terminals, processes, screenshots, desktop UI/input, clipboard, browser automation, diagnostics, and document tools.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceProvides powerful inspection and interaction tools for connected android devices or emulators.11MIT
- AlicenseAqualityNot gradedmaintenanceEnables browser automation through Playwright using accessibility tree snapshots instead of screenshots. Supports web scraping, form interactions, testing, and connecting to existing browser sessions with logged-in accounts.238,303 npm5-
- AlicenseBqualityDmaintenanceEnables AI agents to inspect, log, and control React Native apps on Android emulators or iOS simulators, including UI tree, tap, scroll, and hot reload.2246 npm5MIT
- AlicenseAqualityDmaintenanceEnables AI assistants to build, drive, and observe Android/KMM apps end-to-end through ADB and Gradle, with tools for UI automation, error capture, and testing.3343 npmMIT