Wayland MCP Server

Model Context Protocol server for Wayland desktop automation

Features  •  Installation  •  Usage  •  API  •  Security

Overview

Wayland MCP Server lets AI assistants interact with a Wayland desktop through the Model Context Protocol. It captures the screen for visual analysis by a vision language model, controls the pointer and keyboard, and chains actions into multi-step workflows.

It was built around two observations. Screenshot and automation tooling on Wayland is fragile: it depends on which binaries are installed, which protocol extensions the compositor advertises, and which portals are present on the session bus. And the usual remedy, giving an automation tool root or write access to /dev/input, is a security hole the user pays for every day.

This server answers both. It probes the actual capabilities of the session and picks the best backend that can work, and its preferred path needs no privilege at all: the compositor itself mediates input through the XDG portals.

Quick example

# AI Assistant: "Take a screenshot and tell me what's on screen"
#   Captures the screen, analyzes it with a VLM, and describes what it sees.

# AI Assistant: "Click the OK button"
#   Locates the button from the screenshot, moves the pointer, and clicks.

# AI Assistant: "Fill out this form with test data"
#   Chains pointer and keyboard actions to complete the form.

Related MCP server: yale-agent

Features

Visual analysis

• Screenshot capture with precision ruler overlays

• VLM-powered image analysis via OpenRouter or Google Gemini

• Multiple vision models (Claude, GPT-4V, Gemini, Qwen)

• Side-by-side image comparison and diff detection

Pointer automation

• Absolute and relative pointer positioning

• Click operations (left, right, middle)

• Drag and drop with coordinate precision

• Bidirectional scrolling (vertical and horizontal)

Keyboard control

• Text input simulation

• Individual key press events

• Complex key combinations

Action sequences

• Chain multiple operations together

• Flexible syntax: chain:action1;action2;action3

• Example: chain:click:100,200;type:hello;press:Enter

Installation

Prerequisites

• Python 3.8 or higher

• A Wayland session (GNOME, KDE Plasma, COSMIC, Hyprland, Sway, ...)

• No privileged setup and no root. See Backends for what each path needs; the server reports it itself through the describe_environment tool.

PyGObject is recommended but optional: it unlocks the XDG portal backends, the only path that works on every desktop.

sudo apt install python3-gi        # Debian, Ubuntu, Pop!_OS
sudo dnf install python3-gobject   # Fedora
sudo pacman -S python-gobject      # Arch

Quick install

uvx wayland-mcp-kurojs

With pip, into any environment:

pip install wayland-mcp-kurojs

To run the very latest main instead of the released build, install from this repository:

uvx "wayland-mcp-kurojs @ git+https://github.com/kurojs/wayland-mcp"

The wayland-mcp name on PyPI belongs to an older upstream build; this project publishes as wayland-mcp-kurojs. To let the ephemeral environment see a system PyGObject, use a virtual environment that keeps system packages visible:

uv venv --system-site-packages
uv pip install "wayland-mcp-kurojs @ git+https://github.com/kurojs/wayland-mcp"

From source

git clone https://github.com/kurojs/wayland-mcp.git
cd wayland-mcp
uv venv --system-site-packages
uv pip install -e ".[dev]"

Input control setup

There is none. Input goes through the org.freedesktop.portal.RemoteDesktop portal, which synthesizes pointer and keyboard events with no privilege at all. Your compositor asks for permission the first time; the returned restore token is kept in $XDG_STATE_HOME/wayland-mcp/remote-desktop-token (mode 0600) and replayed afterwards, so you are not asked again. Set WAYLAND_MCP_NO_PERSIST=1 to be prompted every time instead.

If you ran the upstream sudo ./setup.sh, undo it. That script made every /dev/input/event* device world-writable and installed a udev rule (KERNEL=="event*", MODE="0666") to keep them that way across reboots, plus a setuid bit and a NOPASSWD sudoers entry for evemu-event. While that rule is in place, any local process can read your keystrokes, passwords included. scripts/legacy-evemu-setup.sh documents the exact rollback commands. This project removed the script from the install path and needs none of it.

Usage

MCP configuration

The server supports two VLM providers.

Option 1: OpenRouter (multiple models via proxy)

{
  "mcpServers": {
    "wayland": {
      "command": "uvx",
      "args": ["wayland-mcp-kurojs"],
      "env": {
        "OPENROUTER_API_KEY": "sk-or-v1-...",
        "VLM_PROVIDER": "openrouter",
        "VLM_MODEL": "qwen/qwen2.5-vl-72b-instruct:free",
        "XDG_RUNTIME_DIR": "/run/user/1000",
        "WAYLAND_DISPLAY": "wayland-0"
      }
    }
  }
}

Option 2: Google Gemini direct (native API, faster)

{
  "mcpServers": {
    "wayland": {
      "command": "uvx",
      "args": ["wayland-mcp-kurojs"],
      "env": {
        "GEMINI_API_KEY": "AIza...",
        "VLM_PROVIDER": "gemini",
        "VLM_MODEL": "gemini-2.5-flash",
        "XDG_RUNTIME_DIR": "/run/user/1000",
        "WAYLAND_DISPLAY": "wayland-0"
      }
    }
  }
}

Example for Claude Desktop (~/.config/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "wayland": {
      "command": "uvx",
      "args": ["wayland-mcp-kurojs"],
      "env": {
        "GEMINI_API_KEY": "AIza...",
        "VLM_PROVIDER": "gemini",
        "VLM_MODEL": "gemini-2.5-flash",
        "XDG_RUNTIME_DIR": "/run/user/1000",
        "WAYLAND_DISPLAY": "wayland-0"
      }
    }
  }
}

Environment variables

Everything below is optional. The server starts, captures the screen and controls input with none of it set.

Getting API keys:

• OpenRouter: openrouter.ai → Keys section

• Google Gemini: Google AI Studio

MCP clients strip the environment. The reference stdio transport passes only HOME, LOGNAME, PATH, SHELL, TERM and USER to the server it launches, which removes exactly what a Wayland tool needs. The server reconstructs XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS and WAYLAND_DISPLAY from the filesystem when they are missing, so no client-side configuration is required. It declines to guess only when several compositor sockets are present; set WAYLAND_DISPLAY in your client config then.

Backends

Backends are chosen by capability, never by compositor name: the server probes which binaries exist and at what version, which portal interfaces are on the session bus, and which globals the compositor advertises, then picks the highest priority backend whose requirements are actually met. Call describe_environment to see what it decided and why.

Capture, highest priority first:

Input, highest priority first:

evemu is last on purpose and is never a requirement. Nothing in this project changes /dev/input permissions.

Desktop environment compatibility

Only the COSMIC row was verified on real hardware; the others follow from each backend's stated requirements and the unit tests that pin the selection, not from a live run.

Known limitations

• Region capture needs grim + slurp, or a portal that offers an interactive picker. cosmic-screenshot cannot select a region without a human, so it reports that instead of blocking an MCP call.

• Keyboard layout: the portal backend types by keysym and is layout independent. ydotool and evemu type by position, so on a non-US layout they produce whatever those positions mean there. Asking for ASAP 42 on AZERTY gives QSQP 'é.

• cosmic-comp discards the first synthesized event after a session starts. The portal backend absorbs that with a zero-distance warm-up motion; without it, the first click or keystroke of every session is silently lost.

• Absolute pointer positioning through the portal needs a ScreenCast stream attached to the session. Where the portal refuses one, absolute moves raise an explanatory error instead of silently misplacing the cursor; use a relative move, or another input backend.

Example commands

Through an MCP client, you can request actions like:

• "Take a screenshot and analyze what's on the screen"

• "Move the mouse to coordinates (100, 200) and click"

• "Type 'hello world' and press Enter"

• "Click at (50, 50), then drag to (200, 200)"

Available tools

The server exposes the following MCP tools.

Screen capture

• capture_screenshot - take a screenshot with optional ruler overlays

• capture_and_analyze - capture and analyze using VLM in one step

Vision analysis

• analyze_screenshot - analyze an existing screenshot with custom prompt

• compare_images - compare two screenshots to detect differences

Pointer control

• move_mouse - move cursor to coordinates (absolute or relative)

• click_mouse - perform left click at current position

• drag_mouse - drag between two coordinate points

• scroll_mouse - vertical scroll (positive=up, negative=down)

Action execution

• execute_action - execute a single action or chain multiple actions

Action chain syntax

Combine multiple actions with semicolons:

chain:action1;action2;action3

Supported actions:

• type:text - type a text string

• press:key - press a specific key

• click: or click:x,y - click at position or current location

• move_to:x,y - move to absolute coordinates

• move_to:rel:x,y - move relative to current position

• drag:x1,y1:x2,y2 - drag from point to point

• scroll:amount - scroll vertically (typical values: 15-120)

• scroll:horizontal:amount - scroll horizontally

Example chains:

chain:move_to:100,200;click:;type:hello;press:Enter
chain:click:50,50;drag:50,50:200,200
chain:scroll:120;move_to:rel:0,-50;click:

Security

This server grants extensive control over your desktop environment: full pointer and keyboard control, screen capture, and the ability to execute arbitrary input sequences. Only use it with trusted AI models and MCP clients, review action chains before execution in sensitive contexts, and consider running it in a sandboxed or test environment. The AI can perform any action you could perform manually.

Permission model

No setuid binaries, no sudoers entries, no udev rules, no input group membership. The nominal path is the XDG portal: your compositor asks you once, mediates every event, and can revoke the grant at any time. The stored restore token is a capability. Treat it like a credential. Delete $XDG_STATE_HOME/wayland-mcp/remote-desktop-token to revoke it locally, and revoke the permission in your desktop settings to invalidate it entirely.

The fallback backends need more, and get chosen only when nothing better exists: ydotool wants a writable /dev/uinput, evemu a writable /dev/input/event*. The server checks whether that access already exists. It never asks for it, and it never widens it.

Architecture

                    +-----------------------------------+
                    |         MCP Client Layer          |
                    |    (Claude, Cursor, VS Code)      |
                    +------------------+----------------+
                                       |
                              MCP Protocol (stdio/HTTP)
                                       |
                    +------------------v----------------+
                    |        Wayland MCP Server         |
                    |    +----------------------+       |
                    |    |    Core Components    |       |
                    |    +----------------------+       |
                    |    |   FastMCP Handler     |       |
                    |    |   Action Processor    |       |
                    |    |   Chain Parser        |       |
                    |    +----------------------+       |
                    +----+-----------------+------------+
                         |                 |
        +----------------v----+      +-----v-------------+
        |      Vision         |      |       Input       |
        |      Control        |      |      Control      |
        +---------------------+      +-------------------+
        |  VLM analysis       |      |  portal           |
        |  image comparison   |      |  wtype            |
        |                     |      |  ydotool          |
        +---------------------+      +-------------------+
                         |                 |
                         +--------+--------+
                                  |
                        +---------v---------+
                        |  Screen Capture   |
                        +-------------------+
                        |  cosmic-screenshot|
                        |  grim / slurp     |
                        |  legacy tools     |
                        |  portal           |
                        +-------------------+
                                  |
                        +---------v---------+
                        | Wayland Compositor |
                        +-------------------+

Troubleshooting

Start here, whatever the symptom. Call the describe_environment tool: it reports which binaries were found, which portal interfaces are on the bus, which backend was selected for capture, keyboard and pointer, and, when one is unavailable, exactly what each candidate would have needed.

Everything reports as unavailable when launched from an MCP client

• Almost always the stripped environment. The server recovers XDG_RUNTIME_DIR, DBUS_SESSION_BUS_ADDRESS and WAYLAND_DISPLAY on its own, but it refuses to guess between several compositor sockets. Set WAYLAND_DISPLAY in the env block of your client config.

• Check the log (/tmp/wayland-mcp.log) for the line listing what it recovered.

Input control not working

• No sudo step exists any more. If you are looking for setup.sh, read the warning under Input control setup.

• If a permission dialog never appears and nothing happens, your desktop may have no RemoteDesktop portal (wlroots compositors do not ship one). Install wtype for keyboard, ydotool for pointer.

• If the permission was denied once, the stored token is stale: delete $XDG_STATE_HOME/wayland-mcp/remote-desktop-token and try again.

• Verify against a real window rather than guessing: python scripts/verify_input.py clicks, types and scrolls, then reads back what the widgets actually received. Do not touch the machine while it runs.

Screenshots failing

• On COSMIC, a grim older than 1.5 cannot work: the compositor implements only ext-image-copy-capture. Selection already accounts for this; if you forced WAYLAND_MCP_CAPTURE_BACKEND=grim, unset it.

• Install a portal (xdg-desktop-portal plus the backend for your desktop) and PyGObject for the universal fallback.

Typing produces the wrong characters

• You are on a non-US layout with a keycode-based backend (ydotool, evemu), which types positions rather than characters. The portal backend types by keysym and is layout independent.

VLM analysis not working

• Vision is optional: capture and input do not need it. The analysis tools say precisely which variable to set when no provider is configured.

• Confirm the key matches the provider you selected with VLM_PROVIDER.

Server won't start

• Check Python: python3 --version (needs 3.8+).

• Reinstall: uv pip install -e ".[dev]", then pytest. The test suite runs headless and does not need a graphical session.

Contributing

Contributions are welcome. Please see CONTRIBUTING.md for guidelines.

Project structure

wayland-mcp/
├── wayland_mcp/            # Main package
│   ├── server_mcp.py       # MCP server implementation
│   ├── screen_utils.py     # Screenshot and VLM analysis
│   ├── mouse_utils.py      # Pointer control functions
│   ├── keyboard_utils.py   # Keyboard input handling
│   ├── chain_processor.py  # Action chain parser
│   ├── session_env.py      # Recovers session vars MCP clients strip
│   └── backends/           # Capability-selected capture and input backends
│       ├── base.py         # Capabilities probe and backend interfaces
│       ├── detect.py       # Selection by priority and requirements
│       ├── portal.py       # Shared GDBus plumbing for XDG portals
│       ├── capture_*.py    # cosmic-screenshot, grim, legacy tools, portal
│       └── input_*.py      # portal RemoteDesktop, wtype, ydotool, evemu
├── tests/                  # Headless tests: selection, keycodes, session env
├── scripts/
│   ├── verify_input.py     # End-to-end input check against a real window
│   └── legacy-evemu-setup.sh  # Documents how to undo the upstream setup.sh
├── README.md               # This file
├── CONTRIBUTING.md         # Contribution guidelines
└── pyproject.toml          # Package metadata

License

GPL-3.0. See LICENSE for the full text.

Copyright (C) 2026 wayland-mcp contributors.