wayland-computer-use-mcp
Targets GNOME desktops (via Ubuntu) for agent-driven GUI automation, including window capture, AT-SPI UI tree inspection, and input injection.
Targets KDE Plasma desktops (via Kubuntu) for agent-driven GUI automation, enabling window/app inspection, interaction, and desktop integration on Wayland.
Targets Kubuntu desktops for interactive GUI testing and automation, allowing agents to launch, inspect, and control applications on Wayland/KDE Plasma.
Targets Ubuntu desktops (GNOME) for interactive GUI testing and automation, enabling app lifecycle management, UI inspection, and input injection.
Provides GUI testing, automation, and desktop integration for Wayland environments, including live/virtual display modes, window capture, semantic AT-SPI UI inspection, and clamped input injection.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@wayland-computer-use-mcp open the settings app, find the Wi-Fi toggle, and turn it off"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
wayland-computer-use-mcp
wayland-computer-use-mcp is a high-performance Model Context Protocol (MCP) server providing an interactive GUI testing, automation, and desktop integration suite for Wayland environments (KDE Plasma 6 / KWin, GNOME Mutter, Hyprland, Sway, and generic Wayland).
Unlike conventional "computer use" agents that rely on high-latency video streaming and expensive pixel-based coordinate guessing, wayland-computer-use-mcp implements a Tree-First Hybrid Semantic Execution Model:
Atomic Programmatic Execution: Inspects semantic widget trees via AT-SPI2 D-Bus interfaces and executes actions (
DoAction,EditableText,Value,Text) silently without moving the physical cursor.Zero-Disruption Interactions: The desktop pointer only moves on explicit pointer tools (
click,hover,drag) or when Level 1 AT-SPI is unsupported and Level 2 fallback is needed.Playwright-Style Locators & Deltas: Supports optional
roledisambiguation (target="Save", role="push button"), event-driven UI settlement (watch_ui_events), and compact state diffs.Resilient Physical Fallback: Custom canvas widgets, dropdown popovers, and complex surfaces fall back smoothly to clamped physical pointer clicks, drags, discrete wheel scrolls, and keystrokes.
⚡ Token Efficiency: 90–95% Savings Over Vision-Only Approaches
Traditional screenshot-driven computer use models stream full monitor or window screenshots on every single action, consuming 1,500 to 3,500+ vision tokens per step. A 10-step interaction sequence consumes 25,000–35,000+ tokens, introduces substantial latency, and suffers from visual coordinate hallucinations.
wayland-computer-use-mcp reduces token expenditure by over 90%:
Interaction Tier | Modality / Tool | Typical Token Cost | Execution Latency | Determinism / Accuracy |
Traditional Computer Use | Full Monitor Screenshot | ~2,500 – 3,500 tokens | 1.5 – 3.0s | High coordinate hallucination risk |
Window Smart Crop |
| ~1,200 – 1,800 tokens | 0.8 – 1.2s | Visual ambiguity on dense layouts |
Collapsed 1D UI Tree |
| 100 – 300 tokens | < 150ms | 100% Deterministic (Node IDs) |
Reactive UI Delta |
| 20 – 60 tokens | < 50ms | Zero-token re-polling |
Why This Architecture Conserves Tokens:
1D Semantic Flattening: Automatically filters out invisible layout containers (
GtkBox,GtkOverlay,QBoxLayout), distilling only actionable widgets into a compact array of indexed elements (b1,e1,c1) with their role, state, and label.Direct Element Interaction: Programmatic invocation via
interact_with_node(node_id="b1")executes atomically without requiring intermediate verification screenshots.Reactive State Diffing: Every action automatically computes a pre- and post-interaction semantic delta, returning a concise markdown summary (e.g.
b1 [button]: text '0 clicks' ➔ '1 clicks'), eliminating redundant tree re-polling.Selective Visual Grounding: High-resolution window captures (
capture_window_frame) are utilized selectively for custom canvas layouts or styling/visual verification checkpoints.
Related MCP server: deskwright
Operating Modes & Isolation Boundaries
The server operates in two distinct display modes and supports fine-grained access scopes:
1. Display Modes & Blackbox Architecture
Virtual / Blackbox Mode (
--virtual,--virtual-window,WAYLAND_MCP_DISPLAY_MODE="virtual", orWAYLAND_MCP_DISPLAY_MODE="virtual-window"):Starts a private
XDG_RUNTIME_DIR, isolateddbus-daemon, isolated AT-SPI2 bus, and a dedicated nested/headless compositor (kwin_waylandvia direct EIS,labwc/swayvia pure-Pythonzwlr_screencopy_manager_v1+ virtual pointer/keyboard protocols, orweston), keeping your host pointer, focus, and clipboard untouched without triggering host portal prompts.--virtual: Headless virtual Wayland compositor for unattended/CI runs.--virtual-window/WAYLAND_MCP_DISPLAY_MODE="virtual-window"/WAYLAND_MCP_VIRTUAL_HEADED=1: Opens a Playwright-style visible nested compositor window so you can watch the agent drive an app in its own isolated window.
Live Host-Passthrough Mode (
--liveorWAYLAND_MCP_DISPLAY_MODE="live") (Default):Connects to the user's active desktop session via
$WAYLAND_DISPLAY, session D-Bus, and AT-SPI2 bus.Caches and reuses XDG Desktop Portal
restore_tokencredentials to prevent repeated permission prompts.Optional
--lazy-portal/WAYLAND_MCP_LAZY_PORTAL=1: Defers XDG RemoteDesktop/ScreenCast prompts until a screenshot (capture_window_frame) or physical pointer/keyboard fallback is requested, keeping pure AT-SPI interaction prompt-free.
2. Access Scopes
Window Isolation (
--window-onlyorWAYLAND_MCP_ACCESS_MODE="window") (Default):Prompts the user to select only the target application window in the XDG ScreenCast portal prompt.
Clamps all coordinate motions strictly within the detected window geometry boundaries.
Full Display (
--fullscreenorWAYLAND_MCP_ACCESS_MODE="fullscreen"):Grants capture and interaction access to the entire display output.
Dual Selection (
--allow-allorWAYLAND_MCP_ACCESS_MODE="both"):Allows either window or monitor selection during the portal handshake.
🔒 Security Architecture & Upstream Portal Confinement Disclosure
Upstream XDG Portal Architectural Limitation
Under current FreeDesktop XDG Desktop Portal specifications, selecting a single window in the ScreenCast permission prompt restricts video capture to that window; however, the RemoteDesktop portal protocol currently does not enforce server-side pointer boundary confinement. Once a remote desktop session is granted, the protocol allows input injection across the full display surface.
How wayland-computer-use-mcp Mitigates This Risk:
To guarantee safe operation despite upstream protocol constraints, this server implements five layers of client-side containment:
Strict Coordinate Boundary Clamping (
CoordinateClamper): All injected pointer coordinates are mathematically clamped to $[0 \le x \le W, 0 \le y \le H]$ of the target application surface. The server strictly forbids emitting coordinates outside the active window frame.Dynamic Geometry Drift Detection (
GeometryDivergenceDetector): Monitors the baseline window surface position and dimensions. If a window moves, resizes, or unminimizes unexpectedly while an action is pending, the operation is immediately aborted to prevent clicks from spilling into adjacent desktop surfaces.Hardware User Preemption (
UserInterventionDetector): Tracks physical hardware cursor activity. If the user moves the physical mouse or types on the keyboard, automated interactions pause instantly to yield control to the human operator.Dangerous Shortcut Blacklist (
ShortcutFilter): Blocks hazardous keyboard sequences (e.g.Super/Meta,Ctrl+Alt+Delete,Alt+F4, VT terminal switching).Virtual Display Sandbox Recommendation: For evaluating autonomous agents or untrusted scripts, execute with
--virtualto provide hardware-level process and display server isolation.
Quickstart
Run Directly via uvx (Pure-Python core, zero C-build dependencies)
uvx wayland-computer-use-mcpThe core package requires only pure/wheel dependencies (fastmcp, dbus-fast, pillow). Clipboard transfers use org.freedesktop.portal.Clipboard or native Wayland ext_data_control_manager_v1 / zwlr_data_control_manager_v1 (with wl-clipboard/qdbus/xclip fallbacks), keyboard typing adapts to the compositor's live EIS XKB keymap, and screenshots use native compositor capture, gst-launch-1.0 CLI, or org.freedesktop.portal.Screenshot.
Optional GStreamer Python bindings (only needed for in-process PipeWire
appsinkstreaming instead of CLI/native capture):uv pip install "wayland-computer-use-mcp[gstreamer]"Visible isolated Playwright-style window: Uses
kwin_wayland(built into KDE Plasma),labwc/sway, orweston:# No extra compositor install needed on KDE Plasma (`kwin_wayland`); on GNOME/other desktops install labwc (recommended) or sway: sudo apt install -y labwc # Fedora: sudo dnf install labwc / Arch: pacman -S labwc uvx wayland-computer-use-mcp --virtual-windowTarget app development libraries (only if building/running the GTK4 example rig):
Debian / Ubuntu / Kubuntu:
sudo apt install -y python3-gi gir1.2-gtk-4.0 gir1.2-adw-1 at-spi2-coreFedora / RHEL:
sudo dnf install -y python3-gobject gtk4 libadwaita at-spi2-coreArch Linux / Manjaro:
sudo pacman -S --needed python-gobject gtk4 libadwaita at-spi2-core
Install in Virtual Environment
git clone https://github.com/Niklas-Ehrenfried/wayland_computer_use_mcp.git
cd wayland_computer_use_mcp
uv syncMCP Client Configuration
1. Claude Desktop (claude_desktop_config.json)
{
"mcpServers": {
"wayland-computer-use": {
"command": "uvx",
"args": ["wayland-computer-use-mcp"],
"env": {
"WAYLAND_MCP_DISPLAY_MODE": "live",
"WAYLAND_MCP_ACCESS_MODE": "window"
}
}
}
}2. Cursor (.cursor/mcp.json)
{
"mcpServers": {
"wayland-computer-use": {
"command": "uvx",
"args": ["wayland-computer-use-mcp", "--live", "--window-only"]
}
}
}3. Antigravity IDE / Gemini Code Assist (mcp_config.json)
{
"mcpServers": {
"wayland-computer-use": {
"command": "uvx",
"args": ["wayland-computer-use-mcp"],
"env": {
"WAYLAND_MCP_DISPLAY_MODE": "live",
"WAYLAND_MCP_ACCESS_MODE": "window"
}
}
}
}4. Roo Code / Cline (cline_mcp_settings.json)
{
"mcpServers": {
"wayland-computer-use": {
"command": "uvx",
"args": ["wayland-computer-use-mcp"],
"env": {
"WAYLAND_MCP_DISPLAY_MODE": "live",
"WAYLAND_MCP_ACCESS_MODE": "window"
}
}
}
}Exposed Tool Suite (22 Default Tools + 2 Opt-In Tools)
1. Tree-First Semantic Navigation & Macro Presets (5 Tools)
inspect_ui_tree(pid, max_depth): Returns the collapsed 1D interactive element list (b1,e1,c1,sw1,li1) with widget names, roles, compactstates, and coordinates.interact_with_node(target, action, text, value, role, pid, settle_timeout_ms, ...): Dispatches semantic interactions by compact ID (b1) or widget label (node_idkept as deprecated alias; optionalrolenarrows matches like Playwright'sgetByRole). Executes Level 1 AT-SPI directly without moving the mouse cursor, falling back to physical input only when required. Returns structured JSON UI deltas.batch_actions(actions, pid): Executes a batch of sequential UI operations atomically with step-by-step delta tracking, optional per-step"role"filtering, and automaticbatch_failure.pngtrace capture when a live step fails.preset_workflow(action, name, description, steps, pid): Manages and executes reusable multi-step interaction presets (list,run,view,save,delete) stored as persistent JSON artifacts.watch_ui_events(timeout_ms, pid): Streams AT-SPI2 D-Bus accessibility events (Object:StateChanged,ChildrenChanged,TextChanged,Window:Activate) directly to observe async UI changes.
2. Clamped Physical Input (8 Tools)
click(x, y, button, pid): Executes a mouse click clamped to window bounds.double_click(x, y, button, pid): Dispatches a standard mouse double-click.right_click(x, y, pid): Dispatches a right-click (context menu).hover(x, y, duration_ms, pid): Moves pointer without clicking, activating Wayland tooltips or hover highlights.drag(start_x, start_y, end_x, end_y, pid): Performs a clamped mouse drag gesture.scroll(dx, dy, pid): Dispatches pointer wheel ticks viaNotifyPointerAxisDiscreteand continuous deltas.type_text(text, x, y, pid): Types text using evdev keycodes with automated clipboard paste fallback for strings > 30 characters.key_combination(keys, pid): Sends modifier hotkeys (e.g.["ctrl", "s"],["alt", "tab"]). All 8 input tools accept an optionalpidto explicitly target a specific application window.
3. Visual Grounding & Inspection (1 Tool)
capture_window_frame(crop_box, save_artifact, pid): Captures a high-resolution window frame. In-memory MCPImageContentby default; saves rolling disk cache whensave_artifact=True.
4. Process Lifecycle & Crash Interception (5 Tools)
launch_app(target, args, restart, cwd): Spawns Python GUI scripts or system apps with automatic virtual environment discovery. Immediately returns the initial interactive element tree so agents can act without a separateinspect_ui_treecall.restart_app(pid): Gracefully terminates and re-launches an active process, preserving original launch arguments and working directory. Strictly validates process ownership: unmanaged or user-opened applications raisePermissionErrorto preserve desktop consistency.terminate_app(pid): Terminates application processes cleanly (SIGTERMescalated toSIGKILL).list_managed_apps(): Lists all active processes managed by the MCP server, pruning dead PIDs.get_app_logs(pid, lines): Retrieves console output and crash tracebacks from a thread-safe circular buffer.
5. OS & Desktop Integration (3 Default Tools + 2 Opt-In Tools)
clipboard_read(): Reads text from clipboard (org.freedesktop.portal.Clipboardand native Waylandext_data_control_manager_v1/zwlr_data_control_manager_v1first, falling back towl-paste,qdbus,xclip).clipboard_write(text): Writes text to clipboard (org.freedesktop.portal.Clipboardand native Wayland data-control first, falling back towl-copy,qdbus,xclip).window_control(action, pid): Controls window state (minimize,maximize,restore,close).
Opt-in desktop shortcut tools (enable with --enable-desktop-installer or WAYLAND_MCP_ENABLE_DESKTOP_INSTALLER=1):
install_to_desktop(app_name, target, ...): Generates a.desktoplauncher in~/.local/share/applicationsfor universal targets (Python scripts with auto-venv, shell scripts, JS, AppImages, binaries, Flatpak IDs, URLs, or$PATHcommands) with nearby icon discovery, monogram SVG fallback, and worktree badging.uninstall_from_desktop(app_id): Removes desktop launchers and associated icons.
MCP Prompts & Resources
Prompt:
wayland_automation_guide: Best-practice agent instructions for tree-first navigation, delta-to-delta flow, and fallback rules.Resource:
wayland://system_prompt: Dynamic guidelines for LLM agent integration into client system prompts.
Interactive Test Rig
A complete 14-component GTK4/Adwaita verification application is provided in examples/test_gui_app.py:
uv run python examples/test_gui_app.pyRun the automated live end-to-end integration test suite:
uv run pytest tests/test_live_example_app.py -vTested Environments & Compositor Support
Compositor / Desktop | Status | Verified Capabilities |
KDE Plasma 6 ( | ✅ Live-tested | All 22 MCP tools verified live in both |
GNOME Shell ( | ✅ Live-tested |
|
wlroots ( | 🧪 Unit-tested | Backend implemented ( |
GNOME Note: On GNOME Shell, enable "Remember This Selection" in the Remote Desktop portal dialog if you want persistent portal restore tokens across runs, and either share the target window directly or select a monitor when prompted.
License
MIT License. See LICENSE for details.
Available Tools
22 toolsbatch_actionsBatch ActionsA
Executes a sequence of consecutive UI interactions with fail-fast reporting.
Supported step schemas:
{"action": "interact"|"click"|"double_click"|"right_click", "target": "b1"}
{"action": "type"|"set_value"|"clear", "target": "e1", "text": "hello"}
{"action": "click", "x": 100, "y": 200, "button": "left"}
{"action": "key", "keys": ["ctrl", "a"]}
{"action": "hover", "target": "b1"}
{"action": "wait", "ms": 200}
Inputs: actions (required), pid. Returns: object
capture_window_frameCapture Window FrameBRead-onlyIdempotent
Captures window frame as in-memory ImageContent (saves PNG when save_artifact=True).
Inputs: pid, crop_box, save_artifact.
clickClickC
Dispatches a mouse click clamped within the active window boundaries.
Inputs: x (required), y (required), pid, button.
clipboard_readClipboard ReadARead-onlyIdempotent
Reads current text contents from the Wayland clipboard.
Inputs: no named parameters. Returns: result
clipboard_writeClipboard WriteCIdempotent
Writes text into the Wayland clipboard.
Inputs: text (required). Returns: result
double_clickDouble ClickC
Dispatches a double-click within the active window boundaries.
Inputs: x (required), y (required), pid, button.
dragDragC
Performs a clamped drag-and-drop gesture from start to end coordinates.
Inputs: end_x (required), end_y (required), start_x (required), start_y (required), +1 more.
get_app_logsGet App LogsBRead-onlyIdempotent
Retrieves captured stdout and stderr execution logs for an application process.
Inputs: pid, lines. Returns: result
hoverHoverA
Positions cursor at coordinates without clicking to trigger hover states or tooltips.
Inputs: x (required), y (required), pid, duration_ms.
inspect_ui_treeInspect Ui TreeBRead-onlyIdempotent
Inspects semantic accessibility tree and returns functional hierarchy + 1D interactive nodes.
Includes compact functional component hierarchy, actionable elements, screen bounds, and process health by default. Set include_raw_tree=True only when debugging raw AT-SPI nodes.
Inputs: pid, max_depth, include_raw_tree. Returns: object
interact_with_nodeInteract With NodeB
Interacts with an interactive UI widget by 1D node ID (e.g. 'b1', 'e1') or label.
Executes Level 1 AT-SPI actions silently (without moving the cursor), falling back to
Level 2 coordinate/keyboard input only when AT-SPI is unsupported, and returns the
post-action UI tree delta (modified/appeared widgets). Optional role narrows name matches
like Playwright's getByRole.
Supports actions: click, type, set_value, clear, select_all, select_range, copy,
paste, cut, drag_select, double_click, right_click, hover.
Inputs: pid, role, text, value, +6 more.
key_combinationKey CombinationC
Dispatches allowed key combinations (e.g. ['ctrl', 'c']) after security filtering.
Inputs: keys (required), pid.
launch_appLaunch AppC
Launches a Python GUI script or system app with auto-focus and portal session binding.
If restart=True or if the app is already running, gracefully terminates the old process and launches a fresh instance.
Inputs: target (required), cwd, args, restart. Returns: object
list_managed_appsList Managed AppsARead-onlyIdempotent
Lists all active application processes managed by this MCP session.
Inputs: no named parameters. Returns: object
preset_workflowPreset WorkflowA
Manages and executes reusable multi-step interaction macro presets.
Actions:
'list' (default if name is None): Lists all saved presets with descriptions.
'run' (default if name is provided): Executes the preset sequence atomically.
'view': Returns the full step sequence and metadata for the specified preset.
'save': Saves or updates a preset sequence to .agents/artifacts/presets/.json.
'delete': Deletes a preset artifact.
Step schema matches batch_actions:
{"action": "interact", "target": "b1"}
{"action": "type", "target": "e1", "text": "value"}
{"action": "click", "x": 100, "y": 200, "button": "left"}
{"action": "key", "keys": ["ctrl", "a"]}
{"action": "scroll", "dx": 0, "dy": 5}
{"action": "hover", "target": "b1"}
{"action": "wait", "ms": 50}
Inputs: pid, name, steps, action, +2 more. Returns: object
restart_appRestart AppB
Gracefully terminates and re-launches an active process launched by this MCP session, preserving its original launch command, arguments, and working directory.
NOTE: Only applications launched by this MCP server can be restarted. External applications or user-opened windows cannot be restarted by the MCP server.
Inputs: pid. Returns: object
right_clickRight ClickC
Dispatches a right-click within the active window boundaries.
Inputs: x (required), y (required), pid.
scrollScrollB
Dispatches discrete mouse wheel scroll deltas (positive dy=down, negative dy=up).
Inputs: dx (required), dy (required), pid.
terminate_appTerminate AppBDestructiveIdempotent
Terminates an owned application process and closes idle virtual windows.
Inputs: pid. Returns: object
type_textType TextB
Injects keyboard characters. If coordinates (x, y) are provided, clicks first to focus.
Inputs: text (required), x, y, pid.
watch_ui_eventsWatch Ui EventsBRead-onlyIdempotent
Awaits UI settlement by monitoring real-time AT-SPI2 D-Bus mutation events.
Returns captured mutations and whether the UI has settled, eliminating the need to re-query the full accessibility tree after triggers.
Inputs: pid, timeout_ms. Returns: object
window_controlWindow ControlADestructive
Controls window state across any active Wayland compositor (KDE, GNOME, Hyprland, Sway).
Supported actions:
'minimize': Minimizes the target application window.
'maximize': Maximizes the target application window.
'restore': Restores or unminimizes/unmaximizes the target application window.
'focus': Activates and brings the target application window to the foreground.
'close': Requests a clean close of the target window.
Inputs: action (required), pid. Returns: object
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 22 tools
Several tools overlap heavily: interact_with_node already supports click, double_click, right_click, hover, type, set_value, and clear, yet there are separate click/double_click/right_click/hover/type_text/key_combination tools, leaving the agent to guess whether to target a semantic node ID or raw coordinates. batch_actions and preset_workflow also both execute step sequences, differing only in persistence. Descriptions help (AT-SPI vs coordinate fallback), but boundaries remain fuzzy.
Everything is snake_case, but the conventions are mixed: verb_noun (launch_app, inspect_ui_tree, capture_window_frame), bare verbs (click, hover, drag, scroll), noun_verb (clipboard_read, clipboard_write), and noun phrases (window_control, batch_actions, preset_workflow). Readable but not a single predictable pattern.
At 22 tools the surface is on the heavy side for the domain, and much of that weight comes from the low-level mouse/keyboard primitives that duplicate what interact_with_node and batch_actions already do. The app-lifecycle and clipboard groups clearly earn their place, but the input layer could be consolidated.
Coverage is broad for a Wayland computer-use server: window state, app lifecycle with logs, accessibility tree inspection, semantic and raw input, batching, presets, UI event watching, frame capture, and clipboard read/write. Minor gaps exist around window enumeration/positioning and full-screen (non-window) capture, but core agent workflows are supported.
Maintenance
Related MCP Connectors
A cloud Linux desktop for your AI: browse, click, type, read the screen and run commands.
Cloud desktops for AI agents: create a Linux computer, see its screen, click, type and run tasks.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
Related MCP Servers
- AlicenseNot gradedqualityAmaintenanceEnables AI agents to automate Linux desktop GUI by launching and interacting with Wayland applications in isolated virtual KWin sessions, or connecting to live desktops for collaborative automation.159 PyPI81MIT
- AlicenseAqualityBmaintenanceEnables AI agents to operate a real GNOME Wayland desktop through accessibility-tree widget actions, pointer/keyboard input, OCR, window management, and screen capture, optionally on a private headless session.3328 PyPI11Apache 2.0
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to automate a Linux desktop natively on Wayland, with 68 tools for screen capture, input simulation, browser automation, OCR, accessibility, media, audio, clipboard, notifications, filesystem, system, memory, and network operations through the MCP protocol.MIT
- AlicenseNot gradedqualityCmaintenanceEnables an AI agent to see and control a Linux desktop via Wayland/Hyprland, providing screenshots, structured desktop state, pointer/keyboard input, semantic window/workspace tools, and a policy engine for safe execution.Apache 2.0