desktop-mcp

Desktop vision and control for LLMs on Linux, including Wayland — the model takes a screenshot of the whole screen and clicks, types and scrolls on what it sees. Plain MCP over stdio, so it works in Claude Code, Claude Desktop, Cursor, or any other MCP client.

Read this first: you may not need this

Anthropic ships computer use natively, and where that works you should use it instead — it is better integrated and actually maintained:

• macOS and Windows: Claude Desktop has computer use built in, in Cowork and Claude Code. It drives your real desktop. (docs)

• Building your own agent: the Claude API has a first-party computer tool (GA as the computer_toolset_20260801 toolset). (docs)

The gap this fills is Linux. As of August 2026 computer use is not available in the Linux desktop app, and desktop automation on Wayland is genuinely awkward in a way X11 never was:

• COSMIC's compositor does not implement wlr-screencopy, so grim fails outright and every screenshot tool needs a per-compositor path.

• There is no protocol to query the cursor position, and none to enumerate another client's windows.

• Input has to go in below the compositor via /dev/uinput, because the commonly packaged ydotool exposes a relative pointer whose long jumps get mangled by pointer acceleration.

If you are on macOS or Windows, this repo is mostly redundant. If you are on Linux — particularly Wayland — it is the working option.

Desktop-automation MCP servers are not a new category; there are thousands of MCP servers. What is specific here is the Wayland-native absolute-uinput input path, measured pixel-exact on COSMIC.

Related MCP server: mcp-linux-desktop

Status

Written to be cross-platform, but only one configuration has actually been exercised. The rest is code paths, not evidence — treat it accordingly.

What was tested, and how

Verified end-to-end on:

• Pop!_OS 24.04 LTS, kernel 7.0.11, COSMIC (cosmic-comp 1.0.0), Wayland, single 2560x1440 display, Python 3.13.

Results:

Untested and worth knowing: multi-monitor layouts, HiDPI scaling, non-US keyboard layouts on the raw-keycode path (wtype handles those correctly; uinput keycodes assume US), and every non-COSMIC compositor.

Reports from other platforms are welcome — desktop_info dumps everything needed to diagnose a backend problem.

Install

git clone <this repo> && cd desktop-mcp
uv venv && uv pip install -e .

macOS, Windows, and Linux/X11 also need the input dependencies:

uv pip install -e ".[gui]"

Linux: one-time permission setup

Injecting input on Wayland means writing to /dev/uinput, which is root-owned by default:

bash scripts/setup-linux.sh   # needs sudo once
# then log out and back in

macOS: one-time permission setup

Grant Screen Recording and Accessibility to whichever app launches the server (Terminal, iTerm, Claude Desktop) in System Settings → Privacy & Security.

Wire it up

Claude Code:

claude mcp add desktop -- /absolute/path/to/desktop-mcp/.venv/bin/python -m desktop_mcp.server

Claude Desktop (claude_desktop_config.json) and Cursor (.cursor/mcp.json) use the same shape:

{
  "mcpServers": {
    "desktop": {
      "command": "/absolute/path/to/desktop-mcp/.venv/bin/python",
      "args": ["-m", "desktop_mcp.server"]
    }
  }
}

Tools

Coordinates

Screenshots are downscaled (1400px long edge by default) before being sent, so the model works in image pixels; the server maps them back to screen pixels on every call. Pass exactly the coordinates you read off the screenshot.

This is the detail most home-grown computer-use tools get wrong. Miss it and every click lands at a consistent fraction of where it was aimed.

Why /dev/uinput instead of ydotool on Linux

Commonly packaged ydotool builds expose a relative pointer, so a jump to (1280, 720) is delivered as one large relative motion — which the compositor runs through pointer acceleration. The cursor lands near the target rather than on it, and the error grows with distance. An absolute uinput device sidesteps acceleration: measured 5/5 exact on a 2560x1440 screen. ydotool remains a fallback when /dev/uinput is not writable.

Click timing

Compositors dispatch pointer motion asynchronously, and toolkits ignore a button press that arrives before they have processed the motion that put the cursor over the widget. Measured on COSMIC: a 30ms gap between move and press reliably highlights a button but never activates it — the hover state is proof the coordinates are right, which makes this failure easy to misread as a targeting bug. 250ms activates it. A press also has to be held (~90ms) to register as a real click rather than a stray event.

Both delays are tunable; lower them only if your desktop tolerates it.

Safety

This gives a model the same reach over the machine as the person sitting at it. It can read anything on screen — open password managers, private messages, tokens — and click anything, including destructive buttons. Treat a session as if you had handed someone your keyboard.

Environment variables:

ctrl+alt+F1–F12 are always blocked; VT switching can drop the graphical session out from under you.

A read-only reviewer setup:

DESKTOP_MCP_READ_ONLY=1 python -m desktop_mcp.server

Verify

.venv/bin/python scripts/mcp_smoke.py   # real stdio MCP round trip