Wayland MCP Server
Supports desktop automation and visual context analysis for GNOME sessions running under the Wayland compositor.
Integrates with Google Gemini models to provide VLM-powered visual analysis of screenshots, allowing the server to identify and interact with on-screen elements.
Enables screen capture and input control for the Hyprland tiling Wayland compositor.
Provides automation capabilities for KDE Plasma Wayland sessions, enabling AI assistants to interact with the KDE desktop environment.
Provides a native solution for automating modern Linux desktop environments via screenshot capture and input device simulation through evemu.
Supports automation and analysis for Sway compositors, facilitating mouse and keyboard control through an AI-driven interface.
Enables AI-driven desktop automation for Wayland-based systems, including visual analysis through screenshots, absolute/relative mouse control, and keyboard input simulation.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Wayland MCP ServerTake a screenshot, find the browser icon, and click it"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Model Context Protocol server for Wayland desktop automation
Features • Installation • Usage • API • Security
Overview
Wayland MCP Server lets AI assistants interact with a Wayland desktop through the Model Context Protocol. It captures the screen for visual analysis by a vision language model, controls the pointer and keyboard, and chains actions into multi-step workflows.
It was built around two observations. Screenshot and automation tooling on
Wayland is fragile: it depends on which binaries are installed, which protocol
extensions the compositor advertises, and which portals are present on the
session bus. And the usual remedy, giving an automation tool root or write
access to /dev/input, is a security hole the user pays for every day.
This server answers both. It probes the actual capabilities of the session and picks the best backend that can work, and its preferred path needs no privilege at all: the compositor itself mediates input through the XDG portals.
Quick example
# AI Assistant: "Take a screenshot and tell me what's on screen"
# Captures the screen, analyzes it with a VLM, and describes what it sees.
# AI Assistant: "Click the OK button"
# Locates the button from the screenshot, moves the pointer, and clicks.
# AI Assistant: "Fill out this form with test data"
# Chains pointer and keyboard actions to complete the form.Related MCP server: yale-agent
Features
Visual analysis
Screenshot capture with precision ruler overlays
VLM-powered image analysis via OpenRouter or Google Gemini
Multiple vision models (Claude, GPT-4V, Gemini, Qwen)
Side-by-side image comparison and diff detection
Pointer automation
Absolute and relative pointer positioning
Click operations (left, right, middle)
Drag and drop with coordinate precision
Bidirectional scrolling (vertical and horizontal)
Keyboard control
Text input simulation
Individual key press events
Complex key combinations
Action sequences
Chain multiple operations together
Flexible syntax:
chain:action1;action2;action3Example:
chain:click:100,200;type:hello;press:Enter
Installation
Prerequisites
Python 3.10 or higher
A Wayland session (GNOME, KDE Plasma, COSMIC, Hyprland, Sway, ...)
No privileged setup and no root. See Backends for what each path needs; the server reports it itself through the
describe_environmenttool.
PyGObject is recommended but optional: it unlocks the XDG portal backends, the only path that works on every desktop.
sudo apt install python3-gi # Debian, Ubuntu, Pop!_OS
sudo dnf install python3-gobject # Fedora
sudo pacman -S python-gobject # ArchThe portal capture backend works without GStreamer. These packages are only
needed for the cursor in portal screenshots: GStreamer's PipeWire source and its
introspection data, e.g.
sudo apt install gstreamer1.0-pipewire gir1.2-gstreamer-1.0 on Debian and Ubuntu.
Quick install
uvx wayland-mcp-kurojsWith pip, into any environment:
pip install wayland-mcp-kurojsTo run the very latest main instead of the released build, install from this repository:
uvx "wayland-mcp-kurojs @ git+https://github.com/kurojs/wayland-mcp"The wayland-mcp name on PyPI belongs to an older upstream build; this project
publishes as wayland-mcp-kurojs. To let the ephemeral environment see
a system PyGObject, use a virtual environment that keeps system packages
visible:
uv venv --system-site-packages
uv pip install "wayland-mcp-kurojs @ git+https://github.com/kurojs/wayland-mcp"From source
git clone https://github.com/kurojs/wayland-mcp.git
cd wayland-mcp
uv venv --system-site-packages
uv pip install -e ".[dev]"Input control setup
There is none. Input goes through the org.freedesktop.portal.RemoteDesktop
portal, which synthesizes pointer and keyboard events with no privilege at all.
Your compositor asks for permission the first time; the returned restore token
is kept in $XDG_STATE_HOME/wayland-mcp/remote-desktop-token (mode 0600) and
replayed afterwards, so you are not asked again. Set WAYLAND_MCP_NO_PERSIST=1
to be prompted every time instead.
If you ran the upstream
sudo ./setup.sh, undo it. That script made every/dev/input/event*device world-writable and installed a udev rule (KERNEL=="event*", MODE="0666") to keep them that way across reboots, plus a setuid bit and a NOPASSWD sudoers entry forevemu-event. While that rule is in place, any local process can read your keystrokes, passwords included.scripts/legacy-evemu-setup.shdocuments the exact rollback commands. This project removed the script from the install path and needs none of it.
Usage
MCP configuration
The server supports two VLM providers.
Option 1: OpenRouter (multiple models via proxy)
{
"mcpServers": {
"wayland": {
"command": "uvx",
"args": ["wayland-mcp-kurojs"],
"env": {
"OPENROUTER_API_KEY": "sk-or-v1-...",
"VLM_PROVIDER": "openrouter",
"VLM_MODEL": "qwen/qwen2.5-vl-72b-instruct:free",
"XDG_RUNTIME_DIR": "/run/user/1000",
"WAYLAND_DISPLAY": "wayland-0"
}
}
}
}Option 2: Google Gemini direct (native API, faster)
{
"mcpServers": {
"wayland": {
"command": "uvx",
"args": ["wayland-mcp-kurojs"],
"env": {
"GEMINI_API_KEY": "AIza...",
"VLM_PROVIDER": "gemini",
"VLM_MODEL": "gemini-2.5-flash",
"XDG_RUNTIME_DIR": "/run/user/1000",
"WAYLAND_DISPLAY": "wayland-0"
}
}
}
}Example for Claude Desktop (~/.config/Claude/claude_desktop_config.json):
{
"mcpServers": {
"wayland": {
"command": "uvx",
"args": ["wayland-mcp-kurojs"],
"env": {
"GEMINI_API_KEY": "AIza...",
"VLM_PROVIDER": "gemini",
"VLM_MODEL": "gemini-2.5-flash",
"XDG_RUNTIME_DIR": "/run/user/1000",
"WAYLAND_DISPLAY": "wayland-0"
}
}
}
}Environment variables
Everything below is optional. The server starts, captures the screen and controls input with none of it set.
Variable | Description | Default |
Vision analysis (optional feature) | ||
|
|
|
| OpenRouter key | - |
| Google Gemini key | - |
| Azure AI Foundry | - |
| Model identifier | provider default |
| JSON file to read a key from, if you keep it out of the environment | - |
Backend selection | ||
| Force a capture backend by name | auto |
| Force an input backend by name | auto |
|
| unset |
|
| unset: only with a stored permission |
| Where to keep the portal restore token |
|
Behaviour | ||
|
| unset |
| Log file path |
|
Session | ||
| Recovered automatically when your MCP client strips them; set them explicitly if you run several compositors | auto |
Getting API keys:
OpenRouter: openrouter.ai → Keys section
Google Gemini: Google AI Studio
MCP clients strip the environment. The reference stdio transport passes only
HOME,LOGNAME,PATH,SHELL,TERMandUSERto the server it launches, which removes exactly what a Wayland tool needs. The server reconstructsXDG_RUNTIME_DIR,DBUS_SESSION_BUS_ADDRESSandWAYLAND_DISPLAYfrom the filesystem when they are missing, so no client-side configuration is required. It declines to guess only when several compositor sockets are present; setWAYLAND_DISPLAYin your client config then.
Backends
Backends are chosen by capability, never by compositor name: the server probes
which binaries exist and at what version, which portal interfaces are on the
session bus, and which globals the compositor advertises, then picks the highest
priority backend whose requirements are actually met. Call describe_environment
to see what it decided and why.
Capture, highest priority first:
Backend | Needs | Notes |
| the binary, shipped with COSMIC | silent, no dialog; full screen only |
|
| region capture via |
| the respective binary | the upstream cascade, preserved |
|
| works everywhere; may prompt. Draws the cursor from a ScreenCast stream when a portal permission is already stored, and the |
Input, highest priority first:
Backend | Needs | Pointer | Keyboard |
|
| yes, absolute | yes, layout-independent |
|
| no | yes |
|
| yes | yes, US layout only |
|
| approximate | yes, US layout only |
evemu is last on purpose and is never a requirement. Nothing in this project
changes /dev/input permissions.
Desktop environment compatibility
Desktop | Capture | Input | Notes |
COSMIC (Pop!_OS 24.04) | Yes ( | Yes ( | Verified end to end on |
GNOME | Yes ( | Yes ( | No |
KDE Plasma | Yes ( | Yes ( | |
Hyprland, Sway, wlroots | Yes ( | Partial: |
|
Others | Partial: falls back to | Depends |
|
Only the COSMIC row was verified on real hardware; the others follow from each backend's stated requirements and the unit tests that pin the selection, not from a live run.
Known limitations
Region capture needs
grim+slurp, or a portal that offers an interactive picker.cosmic-screenshotcannot select a region without a human, so it reports that instead of blocking an MCP call.Keyboard layout: the portal backend types by keysym and is layout independent.
ydotoolandevemutype by position, so on a non-US layout they produce whatever those positions mean there. Asking forASAP 42on AZERTY givesQSQP 'é.cosmic-comp discards the first synthesized event after a session starts. The portal backend absorbs that with a zero-distance warm-up motion; without it, the first click or keystroke of every session is silently lost.
Absolute pointer positioning through the portal needs a ScreenCast stream attached to the session. Where the portal refuses one, absolute moves raise an explanatory error instead of silently misplacing the cursor; use a relative move, or another input backend.
The cursor in
portalscreenshots: the Screenshot portal has no cursor option, and GNOME leaves the pointer out. Withinclude_mouse(the default) theportalcapture backend therefore takes the frame from a RemoteDesktop session's ScreenCast stream, which embeds the cursor. It needs GStreamer'spipewiresrc, and by default a portal permission has to be stored already -- using any input tool once stores one -- because opening that session without one is what raises the remote-control dialog, and a screenshot is not worth that.WAYLAND_MCP_CURSOR_STREAM=1forces the attempt without a stored permission and may prompt;=0turns the cursor path off. A stream that fails falls back to the Screenshot portal for the following 60 seconds, and is tried again after that. A locked screen, where GNOME refuses new sessions, only costs the cursor of that one capture. Thecursorfield of the result says which one you got.
Example commands
Through an MCP client, you can request actions like:
"Take a screenshot and analyze what's on the screen"
"Move the mouse to coordinates (100, 200) and click"
"Type 'hello world' and press Enter"
"Click at (50, 50), then drag to (200, 200)"
Available tools
The server exposes the following MCP tools.
Screen capture
capture_screenshot- take a screenshot with optional ruler overlayscapture_and_analyze- capture and analyze using VLM in one step
Vision analysis
analyze_screenshot- analyze an existing screenshot with custom promptcompare_images- compare two screenshots to detect differences
Pointer control
move_mouse- move cursor to coordinates (absolute or relative)click_mouse- perform left click at current positiondrag_mouse- drag between two coordinate pointsscroll_mouse- vertical scroll (positive=up, negative=down)
Action execution
execute_action- execute a single action or chain multiple actions
Action chain syntax
Combine multiple actions with semicolons:
chain:action1;action2;action3Supported actions:
type:text- type a text stringpress:key- press a specific keyclick:orclick:x,y- click at position or current locationmove_to:x,y- move to absolute coordinatesmove_to:rel:x,y- move relative to current positiondrag:x1,y1:x2,y2- drag from point to pointscroll:amount- scroll vertically (typical values: 15-120)scroll:horizontal:amount- scroll horizontally
Example chains:
chain:move_to:100,200;click:;type:hello;press:Enter
chain:click:50,50;drag:50,50:200,200
chain:scroll:120;move_to:rel:0,-50;click:Security
This server grants extensive control over your desktop environment: full pointer and keyboard control, screen capture, and the ability to execute arbitrary input sequences. Only use it with trusted AI models and MCP clients, review action chains before execution in sensitive contexts, and consider running it in a sandboxed or test environment. The AI can perform any action you could perform manually.
Permission model
No setuid binaries, no sudoers entries, no udev rules, no input group
membership. The nominal path is the XDG portal: your compositor asks you once,
mediates every event, and can revoke the grant at any time. The stored restore
token is a capability. Treat it like a credential. Delete
$XDG_STATE_HOME/wayland-mcp/remote-desktop-token to revoke it locally, and
revoke the permission in your desktop settings to invalidate it entirely.
The fallback backends need more, and get chosen only when nothing better exists:
ydotool wants a writable /dev/uinput, evemu a writable
/dev/input/event*. The server checks whether that access already exists. It
never asks for it, and it never widens it.
Architecture
+-----------------------------------+
| MCP Client Layer |
| (Claude, Cursor, VS Code) |
+------------------+----------------+
|
MCP Protocol (stdio/HTTP)
|
+------------------v----------------+
| Wayland MCP Server |
| +----------------------+ |
| | Core Components | |
| +----------------------+ |
| | FastMCP Handler | |
| | Action Processor | |
| | Chain Parser | |
| +----------------------+ |
+----+-----------------+------------+
| |
+----------------v----+ +-----v-------------+
| Vision | | Input |
| Control | | Control |
+---------------------+ +-------------------+
| VLM analysis | | portal |
| image comparison | | wtype |
| | | ydotool |
+---------------------+ +-------------------+
| |
+--------+--------+
|
+---------v---------+
| Screen Capture |
+-------------------+
| cosmic-screenshot|
| grim / slurp |
| legacy tools |
| portal |
+-------------------+
|
+---------v---------+
| Wayland Compositor |
+-------------------+Troubleshooting
Start here, whatever the symptom. Call the describe_environment tool: it
reports which binaries were found, which portal interfaces are on the bus, which
backend was selected for capture, keyboard and pointer, and, when one is
unavailable, exactly what each candidate would have needed.
Everything reports as unavailable when launched from an MCP client
Almost always the stripped environment. The server recovers
XDG_RUNTIME_DIR,DBUS_SESSION_BUS_ADDRESSandWAYLAND_DISPLAYon its own, but it refuses to guess between several compositor sockets. SetWAYLAND_DISPLAYin theenvblock of your client config.Check the log (
/tmp/wayland-mcp.log) for the line listing what it recovered.
Input control not working
No
sudostep exists any more. If you are looking forsetup.sh, read the warning under Input control setup.If a permission dialog never appears and nothing happens, your desktop may have no
RemoteDesktopportal (wlroots compositors do not ship one). Installwtypefor keyboard,ydotoolfor pointer.If the permission was denied once, the stored token is stale: delete
$XDG_STATE_HOME/wayland-mcp/remote-desktop-tokenand try again.Verify against a real window rather than guessing:
python scripts/verify_input.pyclicks, types and scrolls, then reads back what the widgets actually received. Do not touch the machine while it runs.
Screenshots failing
On COSMIC, a
grimolder than 1.5 cannot work: the compositor implements onlyext-image-copy-capture. Selection already accounts for this; if you forcedWAYLAND_MCP_CAPTURE_BACKEND=grim, unset it.Install a portal (
xdg-desktop-portalplus the backend for your desktop) and PyGObject for the universal fallback.
Typing produces the wrong characters
You are on a non-US layout with a keycode-based backend (
ydotool,evemu), which types positions rather than characters. Theportalbackend types by keysym and is layout independent.
VLM analysis not working
Vision is optional: capture and input do not need it. The analysis tools say precisely which variable to set when no provider is configured.
Confirm the key matches the provider you selected with
VLM_PROVIDER.
Server won't start
Check Python:
python3 --version(needs 3.10+).Reinstall:
uv pip install -e ".[dev]", thenpytest. The test suite runs headless and does not need a graphical session.
Contributing
Contributions are welcome. Please see CONTRIBUTING.md for guidelines.
Project structure
wayland-mcp/
├── wayland_mcp/ # Main package
│ ├── server_mcp.py # MCP server implementation
│ ├── screen_utils.py # Screenshot and VLM analysis
│ ├── mouse_utils.py # Pointer control functions
│ ├── keyboard_utils.py # Keyboard input handling
│ ├── chain_processor.py # Action chain parser
│ ├── session_env.py # Recovers session vars MCP clients strip
│ └── backends/ # Capability-selected capture and input backends
│ ├── base.py # Capabilities probe and backend interfaces
│ ├── detect.py # Selection by priority and requirements
│ ├── portal.py # Shared GDBus plumbing for XDG portals
│ ├── capture_*.py # cosmic-screenshot, grim, legacy tools, portal
│ └── input_*.py # portal RemoteDesktop, wtype, ydotool, evemu
├── tests/ # Headless tests: selection, keycodes, session env
├── scripts/
│ ├── verify_input.py # End-to-end input check against a real window
│ └── legacy-evemu-setup.sh # Documents how to undo the upstream setup.sh
├── README.md # This file
├── CONTRIBUTING.md # Contribution guidelines
└── pyproject.toml # Package metadataLicense
GPL-3.0. See LICENSE for the full text.
Copyright (C) 2026 wayland-mcp contributors.
Available Tools
10 toolsanalyze_screenshotAnalyze ScreenshotB
Analyze a screenshot with a VLM. Requires a configured VLM provider.
| Name | Required | Description | Default |
|---|---|---|---|
| prompt | Yes | ||
| image_path | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It discloses one meaningful operational constraint — the dependency on a configured VLM provider — which is genuine added value beyond structured fields, but it says nothing about supported image formats, latency/cost, or failure behavior when the provider is unconfigured.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with zero filler, and the core purpose is front-loaded ahead of the prerequisite. Nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be described, but the definition still falls short: no annotations, 0% parameter documentation, no differentiation from capture_and_analyze or compare_images. For a two-required-param vision tool in a crowded sibling set, the description is under-specified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for two required parameters, and the description adds essentially nothing: it never explains that image_path is a filesystem path or that prompt is the VLM instruction. The names are suggestive but not documented, so an agent gets no format or usage detail from either source.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Analyze a screenshot') and adds the mechanism ('with a VLM'), which cleanly separates it from capture_screenshot. It does not, however, distinguish itself from siblings like compare_images or capture_and_analyze, leaving the agent to infer which analysis entry point to pick.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The only guidance is a prerequisite ('Requires a configured VLM provider'); there is no statement of when to use this tool versus compare_images (for two images) or capture_and_analyze (when no image path exists yet). For a tool sitting in a cluster of overlapping image/vision siblings, that omission is costly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
capture_and_analyzeCapture And AnalyzeB
Capture then analyze the screen. Requires a configured VLM provider.
| Name | Required | Description | Default |
|---|---|---|---|
| prompt | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It does disclose the two-step capture-then-analyze flow and the VLM provider prerequisite, which is genuine context beyond structured fields, but says nothing about permissions, latency, or failure modes if no provider is configured.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, zero waste, with the action front-loaded and the prerequisite trailing. Appropriately sized for the tool's simplicity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need no explanation, and the prerequisite is covered. But the undocumented prompt parameter and the absence of any differentiation from capture_screenshot/analyze_screenshot leave gaps for a tool an agent must choose among nine siblings.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is one parameter (prompt) at 0% schema description coverage, and the description never mentions it. It fails to explain that the prompt drives the visual analysis question, which is the only semantic the schema lacks.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific compound action — capture the screen, then analyze it — which is precise about the verb and resource. However, it does not distinguish itself from the closely related siblings capture_screenshot and analyze_screenshot, leaving the agent to infer that this is the combined variant.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Requires a configured VLM provider" is a useful prerequisite, but there is no guidance on when to prefer this tool over chaining capture_screenshot with analyze_screenshot. The agent gets a precondition but no selection criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
capture_screenshotCapture ScreenshotC
Capture the screen, with measurement rulers drawn on the result.
| Name | Required | Description | Default |
|---|---|---|---|
| filename | No | Where to write the PNG. | screenshot.png |
| include_mouse | No | Draw the cursor, when the selected backend can. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full disclosure burden. It omits the side effect that it writes a PNG to disk (only the schema hints at this), says nothing about output format, permissions, or whether the capture is of the full screen versus a window. The 'measurement rulers' detail is the sole behavioral content.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single lean sentence with the core action front-loaded and zero filler. It is efficient, though arguably terse given the tool writes a file and has two configurable parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists and schema coverage is complete, so return-value explanation is not needed. Still, for a tool with a disk-write side effect and no annotations, and with a near-identical sibling (capture_and_analyze), the description is thin on the details an agent needs to choose and call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both parameters (filename, include_mouse) are already fully documented in the schema. The description adds no parameter-level information beyond the baseline, and notably does not mention the cursor/mouse behavior that include_mouse controls.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Capture the screen') plus a distinguishing detail ('with measurement rulers drawn on the result') that separates it from analyze_screenshot and compare_images. However, it never contrasts itself with the closest sibling, capture_and_analyze, so an agent must guess which capture tool is appropriate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no when-to-use, when-not-to-use, or alternative-tool guidance. Given siblings like capture_and_analyze and analyze_screenshot, the description leaves the selection decision entirely to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
click_mouseClick MouseA
Simulate left mouse click at current position. Returns: dict: { 'success': bool, 'error': str (if failed) } Example: click_mouse() # Clicks at current cursor position
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It usefully specifies the left button and the success/error return shape, but omits important context such as irreversibility/side effects of clicking, required screen state, or modifier-key behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The action is front-loaded in the first sentence, and the example is compact. The Returns block duplicates the existing output schema, which is mild redundancy but not harmful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter tool with an output schema, the description covers the core action and gives an example call. It is largely complete, though it could say more about preconditions (cursor already positioned) and when to prefer this over sibling actions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the baseline is 4 and there is nothing for the description to clarify. The empty schema is consistent with the parameterless call shown in the example.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: "Simulate left mouse click at current position," which is unambiguous. It does not explicitly name or differentiate itself from siblings like move_mouse or drag_mouse, though the "at current position" phrasing hints at the distinction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: "at current position" suggests a prior move_mouse call is needed, but the description never states when to use this versus drag_mouse or execute_action. No prerequisites or exclusions are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
compare_imagesCompare ImagesC
Compare two images using a VLM. Requires a configured VLM provider.
| Name | Required | Description | Default |
|---|---|---|---|
| img1_path | Yes | ||
| img2_path | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It discloses one useful prerequisite (a configured VLM provider), but says nothing about failure behavior when no provider is set, whether image data leaves the machine, rate/cost implications, or whether the two paths must share format or dimensions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, front-loaded with the action and followed by the prerequisite. Nothing is padded, though brevity here shades into under-specification rather than efficiency.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained. For a two-parameter tool this is borderline adequate, but the type of comparison (semantic diff, similarity score, description comparison) and the input path semantics remain unaddressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for both parameters, so the description must compensate and does not. It never mentions img1_path or img2_path, their expected format, whether they are local paths or URLs, or ordering semantics (does img1 vs img2 matter for the comparison?).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Compare) and resource (two images) plus the mechanism (VLM), which is enough to distinguish it from the mouse-control siblings. It does not, however, differentiate itself from analysis-oriented siblings like analyze_screenshot or describe_environment, which plausibly also use a VLM.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The only guidance is a prerequisite ('Requires a configured VLM provider'). There is no statement of when to choose this over analyze_screenshot, capture_and_analyze, or describe_environment, nor any exclusions or preconditions on the images themselves.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
describe_environmentDescribe EnvironmentA
Report which backends this machine offers, and which were selected.
Call this first when a capture or an input action fails: it says what is installed, what the compositor advertises, and what that adds up to.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry behavioral disclosure. The verb 'Report' implies a read-only, non-destructive operation, and the description explains what information it surfaces (installed backends, compositor advertising, summary). However, it never explicitly states that it has no side effects, requires no special permissions, or is safe to call repeatedly. The gap is moderate given the read-only nature of the verb.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core purpose. The second sentence efficiently adds the key usage trigger. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter diagnostic tool with an output schema, the description supplies the essential purpose and usage context. The output schema covers return values, and no parameters require explanation. Nothing critical is missing for an agent to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing for the description to add beyond the schema. Baseline for zero parameters is 4; the description appropriately avoids discussing inputs.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Report') and resource ('which backends this machine offers, and which were selected'). It is clearly a diagnostic/environment inspection tool, distinct from the action-oriented siblings (move_mouse, click_mouse, etc.) even though it does not name them explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a clear trigger: 'Call this first when a capture or an input action fails.' This tells the agent when to reach for the tool. It does not provide explicit when-not-to-use conditions or name alternative diagnostic tools, so it falls short of a full 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
drag_mouseDrag MouseB
Perform drag operation between coordinates. Args: x1, y1: Start position x2, y2: End position Returns: dict: { 'success': bool, 'error': str (if failed) } Example: drag_mouse(100, 100, 200, 200) # Drags from (100,100) to (200,200)
| Name | Required | Description | Default |
|---|---|---|---|
| x1 | Yes | ||
| x2 | Yes | ||
| y1 | Yes | ||
| y2 | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It does disclose the success/error return shape and gives a concrete call example, but says nothing about which mouse button is used, drag duration/step count, modifier keys, coordinate origin, or whether the call blocks — all relevant behavioral traits for a drag operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded core sentence followed by compact Args and a single illustrative example; little waste. The Returns block is somewhat redundant since an output schema exists, but it does not bloat the text significantly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so the return-value explanation is not strictly needed. Given a 4-parameter, all-required input with 0% schema coverage and no annotations, the description is minimally adequate but omits coordinate semantics and interaction behavior an agent would want.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it does label x1,y1 as the start position and x2,y2 as the end position. However, it adds no detail on units, coordinate system origin, screen bounds, or ordering constraints, leaving the semantics thin for an all-required 4-integer API.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource ('Perform drag operation between coordinates') and the example reinforces it as a mouse drag from (x1,y1) to (x2,y2). This clearly separates it from siblings like move_mouse or click_mouse, though it never names those alternatives explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this over move_mouse or click_mouse, no mention of prerequisites, and no exclusions. The example shows syntax but not selection criteria, so an agent must infer usage from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
execute_actionExecute ActionA
Execute system actions with chaining support. Handles both single actions and chained sequences. Args: action: Action string in format: Single: "prefix:params" (e.g. "click:100,200") Chain: "chain:action1;action2" (e.g. "chain:click:100,200;type:hello") Supported Actions: type:text - Type text press:key - Press key click/click: - Click at current position (both formats supported) move_to:x,y - Move to absolute coordinates (default) move_to:rel:x,y - Move relative to current position drag:x1,y1:x2,y2 - Drag between points scroll:amount - Vertical scroll (positive=up, negative=down) Note: Each unit = 1 scroll notch (120 = high-def scroll). Typical: 15-120. scroll:horizontal:amount - Horizontal scroll Note: Each unit = 1 scroll notch (120 = high-def scroll). Typical: 15-120. Returns: dict: {'success': bool, 'error': str}
Declared as a dict because that is what every code path returns. It was
annotated -> bool while returning dicts, and a strict MCP client rejects the
structured content against the declared schema, so the tool was unusable
from a conforming client.
Example: execute_action("click:100,200") execute_action("chain:click:100,200;type:hello;press:Enter")
| Name | Required | Description | Default |
|---|---|---|---|
| action | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden and does a good job: it documents the return shape ({'success': bool, 'error': str}), discloses chaining semantics, per-action behavior, and even scroll unit semantics (120 = high-def notch, typical 15-120). It omits any statement about side effects, reversibility, or whether actions are destructive, which for an input-simulation tool would be useful, keeping it out of the 5 range.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded purpose, then a well-organized action reference and example. It is long but most lines earn their place as a DSL reference. The 'Declared as a dict... strict MCP client rejects' paragraph is a meta-note about a past annotation bug rather than agent-facing invocation guidance, which costs a point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter tool with an output schema present, the description is nearly complete for invocation: format, actions, units, and examples are all covered. The remaining gap is selection context against the many overlapping siblings, which for a tool of this complexity an agent would benefit from.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% and there is a single 'action' parameter, yet the description fully compensates: it defines the string format for single vs chained actions, enumerates every supported action with its argument syntax, and gives concrete examples. This is meaning that the bare schema does not provide.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a clear verb+resource ('Execute system actions with chaining support') and even scopes the tool as a general action executor that handles both single and chained sequences. However, it never distinguishes itself from siblings like click_mouse, move_mouse, drag_mouse, and scroll_mouse, which cover the very same underlying actions the DSL exposes. An agent cannot tell from the description why it should route to this general tool versus the specific sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied through examples ('execute_action("click:100,200")') and the format spec. There is no explicit when-to-use guidance, and critically no guidance on when to prefer this tool over the overlapping click_mouse / drag_mouse / scroll_mouse siblings. The reader must infer the choice from the action list.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
move_mouseMove MouseB
Move mouse to specified screen coordinates. Args: x: Horizontal position (0 = left) y: Vertical position (0 = top) relative: If True, moves relative to current position (default: False) Returns: dict: { 'success': bool, 'error': str (if failed) } Examples: move_mouse(100, 200) # Moves to absolute x=100, y=200 move_mouse(10, 10, relative=True) # Moves 10px right and down
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| relative | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It hints at failure via the error string but says nothing about coordinate bounds/off-screen behavior, negative coordinates, required permissions or desktop-session preconditions, or side effects. Return-shape detail is largely redundant given an output schema already exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose is front-loaded in the first line, followed by structured Args/Returns/Examples sections where each line carries information. It is somewhat longer than necessary because the Returns block duplicates the output schema, but nothing is padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 3-parameter tool with zero schema coverage, the description covers every parameter, both call modes, and error signalling, and an output schema covers returns. The remaining gap is behavioral/environmental (bounds, session requirements) and sibling routing, not the mechanics of calling it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must supply all parameter meaning — and it does: x with origin at left, y with origin at top, and relative with its default and its meaning relative to the current position. Two worked examples further disambiguate absolute versus relative calls, fully compensating for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening sentence gives a specific verb+resource ('Move mouse') plus the scope ('to specified screen coordinates'), which is unambiguous and separable from click_mouse/drag_mouse/scroll_mouse. It does not explicitly name or contrast with those siblings, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use move_mouse versus click_mouse, drag_mouse, or execute_action, nor any prerequisite (e.g. an active desktop session). The examples demonstrate invocation syntax only, not selection criteria, leaving the agent to infer context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scroll_mouseScroll MouseB
Scroll vertically (positive=up, negative=down). Note: Each unit represents one notch on the scroll wheel (120 units = high-definition scroll). Typical values range from 2-3 to 5-10 for normal scrolling.
| Name | Required | Description | Default |
|---|---|---|---|
| amount | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It does disclose useful behavioral detail: sign convention (positive=up, negative=down), the unit of measure (one wheel notch), the 120-unit high-definition scale, and typical value ranges. However, it omits prerequisites such as whether the target window must be focused and what side effects or returns to expect.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short lines, front-loaded with the core direction semantics before the supporting note on units. The parenthetical about high-definition scroll is slightly tangential but still earns its place by clarifying the unit scale.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be described. For a one-parameter tool with no annotations, the description covers value semantics well but leaves the operational context (focus requirements, what an agent should do before/after scrolling) unspecified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does: it defines the sign convention, the physical meaning of each unit, and realistic value ranges for the single 'amount' parameter. It never names the parameter 'amount' explicitly, which is a minor gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb+resource combination ('Scroll vertically' on the mouse wheel), which is unambiguous against siblings like move_mouse, click_mouse, and drag_mouse. It does not explicitly name or contrast those siblings, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use scroll_mouse versus the other mouse tools, nor any prerequisites (e.g., window focus, cursor position). The scroll direction semantics are explained, but the when-to-use decision is left entirely to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
10 tool updates
v0.6.1- First observed
analyze_screenshot - First observed
capture_and_analyze - First observed
capture_screenshot - First observed
click_mouse - First observed
compare_images - First observed
describe_environment - First observed
drag_mouse - First observed
execute_action - First observed
move_mouse - First observed
scroll_mouse
TDQS
Scored across 10 tools
The individual mouse tools (move_mouse, click_mouse, drag_mouse, scroll_mouse) are distinct, but execute_action duplicates all of them and adds more. Descriptions hint that execute_action is for chaining, yet the boundary between simple and composite usage remains ambiguous.
All tool names use consistent snake_case with a verb_noun pattern (e.g., move_mouse, capture_screenshot, execute_action). The only minor deviation is capture_and_analyze, but it still follows the same convention.
Ten tools is well within the ideal range for a desktop automation server. Each tool covers a distinct operation or a useful composite, and the count does not feel bloated.
The surface covers mouse control, screenshots, VLM analysis, and environment introspection. Keyboard actions are only available through execute_action, which is a minor gap, but core workflows are supported.
Maintenance
Related MCP Connectors
Cloud desktops for AI agents: create a Linux computer, see its screen, click, type and run tasks.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
AI-powered web automation. Navigate websites using AI agents for one page or a thousand
AI-powered web automation. Navigate websites using AI agents for one page or a thousand
Related MCP Servers
- AlicenseNot gradedqualityAmaintenanceEnables AI agents to automate Linux desktop GUI by launching and interacting with Wayland applications in isolated virtual KWin sessions, or connecting to live desktops for collaborative automation.216 PyPI74MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to automate a Linux desktop natively on Wayland, with 68 tools for screen capture, input simulation, browser automation, OCR, accessibility, media, audio, clipboard, notifications, filesystem, system, memory, and network operations through the MCP protocol.MIT
- AlicenseNot gradedqualityDmaintenanceEnables an AI agent to see and control a Linux desktop via Wayland/Hyprland, providing screenshots, structured desktop state, pointer/keyboard input, semantic window/workspace tools, and a policy engine for safe execution.Apache 2.0
- AlicenseBqualityBmaintenanceEnables AI assistants to automate Hyprland desktop environments through screenshots, mouse/keyboard input, window management, workspaces, clipboard, app launching, and OCR.271MIT