kwin-mcp

Model Context Protocol server for Linux desktop GUI automation on KDE Plasma 6 Wayland

A Model Context Protocol (MCP) server that enables AI agents (Claude Code, Cursor, and other MCP clients) to launch, interact with, and observe any Wayland application in a fully isolated virtual KWin session -- without affecting the user's desktop. It also supports live desktop automation by connecting to an existing KWin session (real desktop or container) for collaborative workflows. With 31 MCP tools covering mouse, keyboard, touch, clipboard, accessibility tree inspection, screenshot capture, and window management, kwin-mcp provides everything needed for end-to-end GUI testing and desktop automation on Linux.

Table of Contents

• Why kwin-mcp?

• Use Cases

• Quick Start

• Configuration

• Available Tools

• How It Works

• System Requirements

• Installation

• Limitations

• End-to-End Testing

• Contributing

• License

Related MCP server: yale-agent

Why kwin-mcp?

• Isolated sessions -- Each session runs in its own dbus-run-session + kwin_wayland --virtual sandbox. Your host desktop is never affected.

• Live session support -- Connect to a real KDE Plasma desktop or a KWin instance inside a container (e.g. systemd-nspawn) for collaborative "share my screen" workflows.

• No screenshots required for interaction -- The AT-SPI2 accessibility tree gives the AI agent structured widget data (roles, names, coordinates, states, available actions), so it can interact with UI elements without relying solely on vision.

• Zero authorization prompts -- Uses KWin's private EIS (Emulated Input Server) D-Bus interface directly, bypassing the XDG RemoteDesktop portal. No user confirmation dialogs.

• Works with any Wayland app -- Anything that runs on KDE Plasma 6 Wayland works: Qt, GTK, Electron, and more. Input is injected via the standard libei protocol.

• Full input coverage -- Mouse, keyboard, multi-touch, and clipboard -- all injected through the isolated session for complete desktop automation.

Use Cases

Automated GUI Testing

Run end-to-end GUI tests for KDE/Qt/GTK applications in headless isolated sessions. kwin-mcp launches each app in its own virtual KWin compositor, interacts via mouse, keyboard, and touch input, then verifies results through screenshots and the accessibility tree -- all without a physical display.

AI-Driven Desktop Automation

Let AI agents like Claude Code autonomously operate desktop applications. The agent reads the accessibility tree to understand the UI, performs actions through 31 MCP tools, and observes the results via screenshots -- creating a complete feedback loop for any Wayland application.

Live Desktop Collaboration

Connect to your real desktop session and let Claude observe and interact with what you see. Use session_connect or pass --default-live-session to make live mode the default. Also supports attaching to KWin running inside containers (e.g. systemd-nspawn) for isolated agent desktops.

Headless GUI Testing in CI/CD

Integrate Linux desktop GUI testing into CI/CD pipelines. kwin-mcp's virtual sessions require no X11 or physical display server, making it suitable for headless environments like GitHub Actions or GitLab CI runners on Linux.

Kiosk and Embedded Device Automation

Automate kiosk interfaces and embedded Linux desktops running KDE Plasma or a bare KWin Wayland compositor. Use session_start for isolated virtual testing of kiosk UIs, or session_connect to attach directly to a live kiosk or embedded device session for real-time automation and diagnostics.

Quick Start

Requires KDE Plasma 6 on Wayland. See System Requirements for details.

1. Install

# Using uv (recommended)
uv tool install kwin-mcp

# Or using pip
pip install kwin-mcp

2. Configure Claude Code

Add to your project's .mcp.json:

{
  "mcpServers": {
    "kwin-mcp": {
      "command": "uvx",
      "args": ["kwin-mcp"]
    }
  }
}

3. Use it

Ask Claude Code to launch and interact with any GUI application:

Start a KWin session, launch kcalc, and press the buttons to calculate 2 + 3.

Claude Code will autonomously start an isolated session, launch the app, read the accessibility tree to find buttons, click them, and take a screenshot to verify the result.

Configuration

Recommended: install as a plugin

The fastest way to wire kwin-mcp into your editor is to install one of the bundled plugins. Each plugin auto-registers the MCP server and ships the kwin-desktop-automation skill, which teaches the agent which tool to call when (session-mode selection, the observe → act → verify loop, US-QWERTY vs Unicode typing, AT-SPI2 surface-local coordinates, and other platform pitfalls).

Claude Code — install the plugin from the marketplace:

/plugin marketplace add isac322/kwin-mcp
/plugin install kwin-mcp@kwin-mcp

OpenCode — add the npm plugin to your opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": ["@isac322/kwin-mcp-opencode"]
}

For the full integration guide (manual fallback, customising the skill, troubleshooting), see docs/ai-agent-integration.md.

Claude Code

Add to your project's .mcp.json:

{
  "mcpServers": {
    "kwin-mcp": {
      "command": "uvx",
      "args": ["kwin-mcp"]
    }
  }
}

Or if installed globally:

{
  "mcpServers": {
    "kwin-mcp": {
      "command": "kwin-mcp"
    }
  }
}

Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "kwin-mcp": {
      "command": "uvx",
      "args": ["kwin-mcp"]
    }
  }
}

Running Directly

# As an installed script
kwin-mcp

# As a Python module
python -m kwin_mcp

# Interactive CLI (REPL for rapid testing)
kwin-mcp-cli

# Live session mode (default to real desktop instead of virtual)
kwin-mcp --default-live-session
kwin-mcp-cli --default-live-session

Available Tools

Session Management (3 tools)

Observation (3 tools)

Mouse Input (6 tools)

Keyboard Input (5 tools)

Touch Input (4 tools)

Clipboard (2 tools)

Window Management (4 tools)

UI Polling (1 tool)

Advanced (3 tools)

Frame capture: Many action tools accept an optional screenshot_after_ms parameter (e.g., [0, 50, 100, 200, 500]) that captures screenshots at specified delays (in milliseconds) after the action completes. This is useful for observing transient UI states like hover effects, click animations, and menu transitions without extra MCP round-trips. Frame capture uses the fast KWin ScreenShot2 D-Bus interface (~30-70ms per frame).

How It Works

Claude Code / AI Agent
  |
  |  MCP (stdio)
  v
kwin-mcp server  (31 tools)       kwin-mcp-cli (interactive REPL)
  |                                  |
  +--- both delegate to AutomationEngine (core.py) ---+
  |
  |-- session_start (virtual) ---> dbus-run-session
  |                                 |-- at-spi-bus-launcher
  |                                 +-- kwin_wayland --virtual
  |                                       +-- [your app]
  |
  |-- session_connect (live) ----> existing KWin (real desktop / container)
  |
  |-- screenshot ---------------> KWin ScreenShot2 D-Bus (spectacle fallback)
  |
  |-- accessibility_tree -------> AT-SPI2 (via PyGObject)
  |-- find_ui_elements ---------> AT-SPI2 (via PyGObject)
  |-- wait_for_element ----------> AT-SPI2 (polling)
  |
  |-- mouse_* ------------------> KWin EIS D-Bus --> libei
  |-- keyboard_* ---------------> KWin EIS D-Bus --> libei
  |-- touch_* ------------------> KWin EIS D-Bus --> libei
  |    +-- screenshot_after_ms -> KWin ScreenShot2 D-Bus (fast frame capture)
  |
  |-- keyboard_type_unicode ----> wtype / wl-copy + Ctrl+V
  |-- clipboard_* --------------> wl-copy / wl-paste (wl-clipboard)
  |
  |-- launch_app / list_windows / focus_window
  |                                |-- subprocess spawn
  |                                +-- AT-SPI2 (via PyGObject)
  |
  |-- dbus_call -----------------> dbus-send (generic D-Bus)
  |-- read_app_log --------------> log file read
  +-- wayland_info --------------> wayland-info

Triple Isolation (+ Optional Home Isolation)

kwin-mcp provides three layers of isolation from the host desktop:

1. D-Bus isolation -- dbus-run-session creates a private session bus. The isolated session's services (KWin, AT-SPI2, portals) are invisible to the host.

2. Display isolation -- kwin_wayland --virtual creates its own Wayland compositor with a virtual framebuffer. No windows appear on the host display.

3. Input isolation -- Input events are injected through KWin's EIS interface into the isolated compositor only. The host desktop receives no input from kwin-mcp.

4. Home directory isolation (optional) -- When isolate_home=true is set in session_start, a temporary HOME directory is created with isolated XDG directories (XDG_CONFIG_HOME, XDG_DATA_HOME, XDG_CACHE_HOME, XDG_STATE_HOME). Apps in the session cannot read or modify host user settings (e.g. ~/.config/kdeglobals), improving test reproducibility and safety. XDG_RUNTIME_DIR is intentionally not isolated because the Wayland socket resides there.

Input Injection

Mouse, keyboard, and touch events are injected through KWin's private org.kde.KWin.EIS.RemoteDesktop D-Bus interface. This returns a libei file descriptor that allows low-level input emulation without requiring the XDG RemoteDesktop portal (which would show a user authorization dialog). The connection uses:

• Absolute pointer positioning for precise coordinate-based interaction

• evdev keycodes with full US QWERTY mapping for keyboard input

• Smooth drag interpolation (10+ intermediate steps) for realistic drag operations

• EIS touch emulation for multi-touch gestures (tap, swipe, pinch, multi-finger swipe)

Screenshot Capture

The normal Wayland capture path tries KWin's org.kde.KWin.ScreenShot2 D-Bus interface first and uses the spectacle CLI as a fallback. Action tools with screenshot_after_ms use the same path for frame bursts, and Pillow converts ScreenShot2's raw ARGB pipe data to PNG. The Docker visual QA suite also has an explicit test-only X11 backend: when KWIN_MCP_X11_SCREENSHOT=1 is set for a server connected to the nested KWin/Xvfb fixture, captures use scrot. Normal virtual and live Wayland sessions do not opt into this backend.

Accessibility Tree

The AT-SPI2 accessibility bus within the isolated session is queried via PyGObject (gi.repository.Atspi). This provides a structured tree of all UI widgets with their roles (button, text field, menu item, etc.), names, states (focused, enabled, visible, etc.), screen coordinates, and available actions (click, toggle, etc.).

System Requirements

Optional dependencies:

Installing System Dependencies

sudo pacman -S kwin spectacle at-spi2-core python-gobject dbus-python-common

# Optional: for clipboard and Unicode input
sudo pacman -S wl-clipboard wtype wayland-utils

sudo dnf install kwin-wayland spectacle at-spi2-core python3-gobject dbus-python

# Optional: for clipboard and Unicode input
sudo dnf install wl-clipboard wtype wayland-utils

sudo zypper install kwin6 spectacle at-spi2-core python3-gobject python3-dbus-python

# Optional: for clipboard and Unicode input
sudo zypper install wl-clipboard wtype wayland-utils

sudo apt install kwin-wayland spectacle at-spi2-core python3-gi gir1.2-atspi-2.0 python3-dbus

# Optional: for clipboard and Unicode input
sudo apt install wl-clipboard wtype wayland-utils

Installation

Using uv (recommended)

uv tool install kwin-mcp

Using pip

pip install kwin-mcp

From source

git clone https://github.com/isac322/kwin-mcp.git
cd kwin-mcp
uv sync
uv run kwin-mcp

Limitations

• US QWERTY keyboard layout only -- keyboard_type supports US QWERTY only. For non-ASCII text (Korean, CJK, etc.), use keyboard_type_unicode, which requires wtype or wl-clipboard installed.

• KDE Plasma 6+ required -- Older KDE versions or other Wayland compositors (GNOME, Sway) are not supported.

• AT-SPI2 availability varies -- Some applications may not fully expose their widget tree via AT-SPI2.

• Touch input is EIS-emulated -- Touch events are emulated through KWin's EIS interface, not from a real touchscreen device. Most applications handle emulated touch correctly, but some may behave differently from physical touch.

• Clipboard requires opt-in -- Clipboard tools (clipboard_get, clipboard_set) are disabled by default because wl-copy can hang in isolated sessions. Enable with enable_clipboard=true in session_start, and ensure wl-clipboard is installed.

• QMenu (native context menus) may not appear in AT-SPI2 -- Qt's AT-SPI2 bridge has incomplete support for popup menus on Wayland. Context menus may not be visible in accessibility_tree or find_ui_elements. Workaround: click by coordinates derived from the parent widget's rectangle plus the client origin from window_geometry.

• Screen edge triggers ignore EIS pointer events -- Auto-hide panels and layer-shell strips do not react when the pointer reaches a screen edge through EIS. Use dbus_call to invoke KWin scripting or a keyboard shortcut instead of trying to hover the edge.

• KWin claims multi-finger touch gestures -- Three- and four-finger swipes are consumed by the compositor as global gestures and never reach the application; use fingers=2 when the target is the app itself.

• AT-SPI2 coordinates are surface-local, not screen-global -- Wayland clients do not know their global screen position (by design). Coordinates returned by find_ui_elements and accessibility_tree are relative to the window's top-left corner, while mouse_click and touch_tap take screen coordinates. Convert them by adding the client origin from window_geometry: clicking a reported rectangle verbatim lands on whatever occupies that screen position instead.

End-to-End Testing

The Docker suite currently collects 99 tests against the packaged application, not an editable source checkout. docker/e2e.Dockerfile builds a wheel, installs it with the locked MCP 1.x dependency set into /opt/kwin-mcp-venv, and runs both AutomationEngine tests and the installed kwin-mcp console entry point. The MCP tests initialize a real client/server session over stdio JSON-RPC.

Run the complete suite from the repository root:

scripts/run-e2e-docker.sh

The runner builds the image for Docker's native architecture, creates artifacts/e2e/<UTC-timestamp>-<pid>/, runs pytest, and prints the artifact path. Additional arguments pass directly to pytest:

scripts/run-e2e-docker.sh -- tests/e2e/test_mcp_protocol.py -v
scripts/run-e2e-docker.sh -- -k "visual or screenshot" -v

Coverage includes:

• virtual KWin engine tests for session lifecycle, AT-SPI2 observation, window geometry and control, EIS pointer/keyboard/touch input, clipboard, cleanup, and error handling;

• exact input-schema checks for all 31 registered tools, plus installed-server stdio calls through every MCP wrapper;

• nested visual tests that start Xvfb and a test-owned KWin compositor inside the container, connect the installed MCP server to it, and verify pixels as well as accessibility state;

• KCalc before/after pixel transitions and a deterministic GUI probe for mouse hover, cursor inclusion, animation frame bursts, and CJK text (GUI 검증 42) rendered differently from a tofu control (□□);

• screenshot retention and failure behavior, environment provenance, installed distribution metadata, console entry points, and process/socket cleanup.

The container uses software rendering and needs no --privileged, --cap-add, GPU, or device flags. Its nested Xvfb server is part of the visual fixture; the host does not need an X server. CI runs the same image natively on both architectures:

A completed run retains environment.json, junit.xml, pytest.log, nested-KWin/Xvfb/MCP logs, and visual PNG evidence below its artifact directory. Failed runs also collect Docker inspect, container log, and process-list diagnostics.

One legacy success test for ScreenShot2 on KWin's exact --virtual backend remains intentionally skipped because that backend does not return capture data in this container. Error propagation for that path is tested at both engine and MCP stdio levels; screenshot success, cursor pixels, and frame bursts are tested through the explicit nested X11/scrot visual mode. See docker/README.md for the process topology, complete test-file inventory, evidence layout, and targeted commands.

Contributing

Contributions are welcome! See CONTRIBUTING.md for development setup, code style guidelines, and the pull request process.

git clone https://github.com/isac322/kwin-mcp.git
cd kwin-mcp
uv sync
uv run ruff check src/
uv run ruff format --check src/
uv run ty check src/

License

MIT