Wayland MCP Server

Model Context Protocol server for Wayland desktop automation

Features • Installation • Usage • API • Security

Overview

Wayland MCP Server enables AI assistants to interact with your Wayland desktop through the Model Context Protocol. It provides screenshot capture with VLM analysis, mouse control, keyboard input, and action chaining capabilities.

Why This Project?

Existing Wayland screenshot and automation tools often have reliability issues. This project provides a robust, MCP-native solution specifically designed for AI-driven desktop automation on modern Linux systems.

Quick Example

Features

Visual Analysis

• Screenshot capture with precision ruler overlays

• VLM-powered image analysis via OpenRouter or Google Gemini

• Multiple vision model support (Claude, GPT-4V, Gemini, Qwen)

• Side-by-side image comparison and diff detection

Mouse Automation

• Absolute and relative cursor positioning

• Click operations (left, right, middle button)

• Drag and drop with coordinate precision

• Bidirectional scrolling (vertical/horizontal)

Keyboard Control

• Text input simulation

• Individual key press events

• Complex key combinations

Action Sequences

• Chain multiple operations together

• Flexible syntax: chain:action1;action2;action3

• Example: chain:click:100,200;type:hello;press:Enter

Installation

Prerequisites

• Python 3.8 or higher

• Wayland compositor (GNOME, KDE Plasma, Hyprland, Sway, etc.)

• grim and slurp for screenshots (usually pre-installed)

Quick Install

From Source

Input Control Setup

For mouse and keyboard automation, run the setup script:

What it does:

• Installs evemu-tools package

• Configures setuid for evemu-event

• Adds user to input group

• Creates udev rules for device access

After setup, log out and back in for group changes to take effect.

Usage

MCP Configuration

The server supports two VLM providers:

Option 1: OpenRouter (multiple models via proxy)

Option 2: Google Gemini Direct (native API, faster)

Example for Claude Desktop (~/.config/Claude/claude_desktop_config.json):

Note: See CONFIG_EXAMPLES.md for more configuration examples including Cursor, OpenRouter models, and VLM provider options.

Environment Variables

Getting API Keys:

• OpenRouter: openrouter.ai → Keys section

• Google Gemini: Google AI Studio

Desktop Environment Compatibility

Example Commands

Through an MCP client, you can request actions like:

• "Take a screenshot and analyze what's on the screen"

• "Move the mouse to coordinates (100, 200) and click"

• "Type 'hello world' and press Enter"

• "Click at (50, 50), then drag to (200, 200)"

Available Tools

The server exposes the following MCP tools:

Screen Capture

• capture_screenshot - Take a screenshot with optional ruler overlays

• capture_and_analyze - Capture and analyze using VLM in one step

Vision Analysis

• analyze_screenshot - Analyze an existing screenshot with custom prompt

• compare_images - Compare two screenshots to detect differences

Mouse Control

• move_mouse - Move cursor to coordinates (absolute or relative)

• click_mouse - Perform left click at current position

• drag_mouse - Drag between two coordinate points

• scroll_mouse - Vertical scroll (positive=up, negative=down)

Action Execution

• execute_action - Execute single action or chain multiple actions

Action Chain Syntax

Combine multiple actions with semicolons:

Supported Actions:

• type:text - Type a text string

• press:key - Press a specific key

• click: or click:x,y - Click at position or current location

• move_to:x,y - Move to absolute coordinates

• move_to:rel:x,y - Move relative to current position

• drag:x1,y1:x2,y2 - Drag from point to point

• scroll:amount - Scroll vertically (typical values: 15-120)

• scroll:horizontal:amount - Scroll horizontally

Example Chains:

Security

⚠️ IMPORTANT SECURITY CONSIDERATIONS

This server grants extensive control over your desktop environment:

• Full mouse and keyboard control

• Screen capture capabilities

• Ability to execute arbitrary input sequences

Best Practices

• Only use with trusted AI models and MCP clients

• Review action chains before execution in sensitive contexts

• Consider running in a sandboxed or test environment

• Be aware that the AI can perform any action you could perform manually

Permission Model

The setup script requires sudo access to:

• Install system packages (evemu-tools)

• Modify file permissions

• Configure udev rules

After setup, the server runs with your user privileges but can control input devices through configured permissions.

Architecture

Troubleshooting

Input control not working

• Ensure you ran sudo ./setup.sh

• Log out and back in after setup

• Verify you're in the input group: groups | grep input

Screenshots failing

• Check if grim is installed: which grim

• Verify WAYLAND_DISPLAY matches your session: echo $WAYLAND_DISPLAY

VLM analysis not working

• Confirm OPENROUTER_API_KEY is set correctly

• Check API key permissions on OpenRouter dashboard

• Test model availability: some models have usage limits

Server won't start

• Check Python version: python3 --version (needs 3.8+)

• Verify all dependencies: pip install -e .

• Look for port conflicts if using custom WAYLAND_MCP_PORT

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

Project Structure

License

GPL-3.0 License - See LICENSE for details.

Acknowledgments

• Built on the Model Context Protocol

• Uses FastMCP for server implementation

• Inspired by the need for reliable Wayland automation tools