wayland-computer-use-mcp

wayland-computer-use-mcp is a high-performance Model Context Protocol (MCP) server providing an interactive GUI testing, automation, and desktop integration suite for Wayland environments (KDE Plasma 6 / KWin, GNOME Mutter, Hyprland, Sway, and generic Wayland).

Unlike conventional "computer use" agents that rely on high-latency video streaming and expensive pixel-based coordinate guessing, wayland-computer-use-mcp implements a Tree-First Hybrid Semantic Execution Model:

• Atomic Programmatic Execution: Inspects semantic widget trees via AT-SPI2 D-Bus interfaces and executes actions (DoAction, EditableText, Text) directly without coordinate ambiguity.

• Observable Cursor Tracing: Visibly translates the pointer over target elements prior to interaction, ensuring live tracking and transparency for human observers.

• Resilient Physical Fallback: Custom canvas widgets, dropdown popovers, and complex surfaces fall back smoothly to clamped physical pointer clicks, drags, discrete wheel scrolls, and keystrokes.

⚡ Token Efficiency: 90–95% Savings Over Vision-Only Approaches

Traditional screenshot-driven computer use models stream full monitor or window screenshots on every single action, consuming 1,500 to 3,500+ vision tokens per step. A 10-step interaction sequence consumes 25,000–35,000+ tokens, introduces substantial latency, and suffers from visual coordinate hallucinations.

wayland-computer-use-mcp reduces token expenditure by over 90%:

Why This Architecture Conserves Tokens:

1. 1D Semantic Flattening: Automatically filters out invisible layout containers (GtkBox, GtkOverlay, QBoxLayout), distilling only actionable widgets into a compact array of indexed elements (b1, e1, c1) with their role, state, and label.

2. Direct Element Interaction: Programmatic invocation via interact_with_node(node_id="b1") executes atomically without requiring intermediate verification screenshots.

3. Reactive State Diffing: Every action automatically computes a pre- and post-interaction semantic delta, returning a concise markdown summary (e.g. b1 [button]: text '0 clicks' ➔ '1 clicks'), eliminating redundant tree re-polling.

4. Selective Visual Grounding: Set-of-Marks visual overlays (take_labeled_screenshot) are utilized exclusively for layout or styling validation checkpoints.

Related MCP server: kwin-mcp

Operating Modes & Isolation Boundaries

The server operates in two distinct display modes and supports fine-grained access scopes:

1. Display Modes

• Live Mode (--live or WAYLAND_MCP_DISPLAY_MODE="live") (Default):

  • Connects to the user's active desktop session via $WAYLAND_DISPLAY.

  • Interfaces with the active session D-Bus and AT-SPI2 bus.

  • Automatically caches and reuses XDG Desktop Portal restore_token credentials to prevent repeated permission prompts.

  • Physically moves the desktop pointer so human operators can follow agent actions in real time.

• Virtual / Isolated Mode (--virtual or WAYLAND_MCP_DISPLAY_MODE="virtual"):

  • Connects to or launches an isolated virtual Wayland compositor (e.g. weston --backend=headless-backend.so, kwin_wayland --virtual, or gamescope).

  • Completely separates agent actions from personal desktop workspaces, enabling unattended, headless, or CI/CD test automation.

2. Access Scopes

• Window Isolation (--window-only or WAYLAND_MCP_ACCESS_MODE="window") (Default):

  • Prompts the user to select only the target application window in the XDG ScreenCast portal prompt.

  • Clamps all coordinate motions strictly within the detected window geometry boundaries.

• Full Display (--fullscreen or WAYLAND_MCP_ACCESS_MODE="fullscreen"):

  • Grants capture and interaction access to the entire display output.

• Dual Selection (--allow-all or WAYLAND_MCP_ACCESS_MODE="both"):

  • Allows either window or monitor selection during the portal handshake.

🔒 Security Architecture & Upstream Portal Confinement Disclosure

How wayland-computer-use-mcp Mitigates This Risk:

To guarantee safe operation despite upstream protocol constraints, this server implements five layers of client-side containment:

1. Strict Coordinate Boundary Clamping (CoordinateClamper): All injected pointer coordinates are mathematically clamped to $[0 \le x \le W, 0 \le y \le H]$ of the target application surface. The server strictly forbids emitting coordinates outside the active window frame.

2. Dynamic Geometry Drift Detection (GeometryDivergenceDetector): Monitors the baseline window surface position and dimensions. If a window moves, resizes, or unminimizes unexpectedly while an action is pending, the operation is immediately aborted to prevent clicks from spilling into adjacent desktop surfaces.

3. Hardware User Preemption (UserInterventionDetector): Tracks physical hardware cursor activity. If the user moves the physical mouse or types on the keyboard, automated interactions pause instantly to yield control to the human operator.

4. Dangerous Shortcut Blacklist (ShortcutFilter): Blocks hazardous keyboard sequences (e.g. Super/Meta, Ctrl+Alt+Delete, Alt+F4, VT terminal switching).

5. Virtual Display Sandbox Recommendation: For evaluating autonomous agents or untrusted scripts, execute with --virtual to provide hardware-level process and display server isolation.

Quickstart

Run Directly via uvx (Zero installation required)

uvx wayland-computer-use-mcp

Install in Virtual Environment

git clone https://github.com/your-org/wayland-computer-use-mcp.git
cd wayland-computer-use-mcp
uv venv --python python3 --system-site-packages
source .venv/bin/activate
uv pip install -e .

MCP Client Configuration

1. Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "wayland-computer-use": {
      "command": "uvx",
      "args": ["wayland-computer-use-mcp"],
      "env": {
        "WAYLAND_MCP_DISPLAY_MODE": "live",
        "WAYLAND_MCP_ACCESS_MODE": "window"
      }
    }
  }
}

2. Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "wayland-computer-use": {
      "command": "uvx",
      "args": ["wayland-computer-use-mcp", "--live", "--window-only"]
    }
  }
}

3. Antigravity IDE / Gemini Code Assist (mcp_config.json)

{
  "mcpServers": {
    "wayland-computer-use": {
      "command": "uvx",
      "args": ["wayland-computer-use-mcp"],
      "env": {
        "WAYLAND_MCP_DISPLAY_MODE": "live",
        "WAYLAND_MCP_ACCESS_MODE": "window"
      }
    }
  }
}

4. Roo Code / Cline (cline_mcp_settings.json)

{
  "mcpServers": {
    "wayland-computer-use": {
      "command": "uvx",
      "args": ["wayland-computer-use-mcp"],
      "env": {
        "WAYLAND_MCP_DISPLAY_MODE": "live",
        "WAYLAND_MCP_ACCESS_MODE": "window"
      }
    }
  }
}

Exposed Tool Suite (21 Tools)

1. Tree-First Semantic Navigation

• inspect_ui_tree(pid, max_depth): Returns the collapsed 1D interactive element list (b1, e1, c1) with widget names, roles, states, and coordinates.

• interact_with_node(node_id, action, text, ...): Dispatches semantic interactions. Visibly traces cursor, executes AT-SPI action, and falls back to physical input if required. Auto-scrolls viewport if target is off-screen.

• click_element_by_label(label, role): Resolves widgets by visible label or role and performs a targeted click.

• batch_actions(actions, pid): Executes a batch of sequential UI operations atomically without intermediate screenshot pauses.

2. Clamped Physical Input

• click(x, y, button): Executes a mouse click clamped to window bounds.

• double_click(x, y, button): Dispatches a standard mouse double-click.

• right_click(x, y): Dispatches a right-click (context menu).

• hover(x, y, duration_ms): Moves pointer without clicking, activating Wayland tooltips or hover highlights.

• drag(start_x, start_y, end_x, end_y): Performs a clamped mouse drag gesture.

• scroll(dx, dy): Dispatches pointer wheel ticks via NotifyPointerAxisDiscrete and continuous deltas.

• type_text(text, x, y): Types text using evdev keycodes with automated clipboard paste fallback for strings > 30 characters.

• key_combination(keys): Sends modifier hotkeys (e.g. ["ctrl", "s"], ["alt", "tab"]).

3. Visual Grounding & Inspection

• capture_window_frame(crop_box): Captures a high-resolution window frame. Yields Markdown image preview links and MCP standard ImageContent blocks.

• take_labeled_screenshot(): Captures window frame annotated with numbered Set-of-Marks boundary badges.

4. Process Lifecycle & Crash Interception

• launch_app(script_path, args, cwd): Spawns Python GUI scripts with automatic virtual environment discovery.

• terminate_app(pid): Terminates application processes cleanly (SIGTERM escalated to SIGKILL).

• get_app_logs(pid, lines): Retrieves console output and crash tracebacks from a thread-safe 200-line circular buffer.

5. OS & Desktop Integration

• clipboard_read(): Reads text from the Wayland clipboard (wl-paste).

• clipboard_write(text): Writes text to the Wayland clipboard (wl-copy).

• window_control(action, pid): Controls window state (minimize, maximize, restore, close).

• install_to_desktop(app_id, name, ...): Generates a valid Linux .desktop launcher with worktree detection and version badging.

• uninstall_from_desktop(app_id): Removes desktop launchers and associated icons.

Interactive Test Rig

A complete 14-component GTK4/Adwaita verification application is provided in examples/test_gui_app.py:

uv run python examples/test_gui_app.py

Run the automated live end-to-end integration test suite:

uv run pytest tests/test_live_example_app.py -v

License

MIT License. See LICENSE for details.