# desktop-mcp

Desktop vision and control for LLMs on **Linux, including Wayland** — the model
takes a screenshot of the whole screen and clicks, types and scrolls on what it
sees. Plain MCP over stdio, so it works in Claude Code, Claude Desktop, Cursor,
or any other MCP client.

## Read this first: you may not need this

Anthropic ships computer use natively, and where that works you should use it
instead — it is better integrated and actually maintained:

- **macOS and Windows**: Claude Desktop has computer use built in, in Cowork and
  Claude Code. It drives your real desktop.
  ([docs](https://support.claude.com/en/articles/14128542-let-claude-use-your-computer-in-cowork))
- **Building your own agent**: the Claude API has a first-party `computer` tool
  (GA as the `computer_toolset_20260801` toolset).
  ([docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool))

**The gap this fills is Linux.** As of August 2026 computer use is not available
in the Linux desktop app, and desktop automation on Wayland is genuinely awkward
in a way X11 never was:

- COSMIC's compositor does not implement `wlr-screencopy`, so `grim` fails
  outright and every screenshot tool needs a per-compositor path.
- There is no protocol to query the cursor position, and none to enumerate
  another client's windows.
- Input has to go in below the compositor via `/dev/uinput`, because the
  commonly packaged `ydotool` exposes a *relative* pointer whose long jumps get
  mangled by pointer acceleration.

If you are on macOS or Windows, this repo is mostly redundant. If you are on
Linux — particularly Wayland — it is the working option.

Desktop-automation MCP servers are not a new category; there are thousands of
MCP servers. What is specific here is the Wayland-native absolute-uinput input
path, measured pixel-exact on COSMIC.

## Status

Written to be cross-platform, but only one configuration has actually been
exercised. The rest is code paths, not evidence — treat it accordingly.

| Platform | Screenshot | Input | Tested |
|---|---|---|---|
| Linux / Wayland | `cosmic-screenshot`, `grim`, `spectacle`, GNOME D-Bus | `/dev/uinput` | **Yes** — COSMIC only |
| Linux / X11 | `maim`, ImageMagick `import`, `mss` | `/dev/uinput`, `xdotool` | No |
| macOS | `screencapture` | `pyautogui` | No |
| Windows | `mss`, PowerShell | `pyautogui` | No |

### What was tested, and how

Verified end-to-end on:

- **Pop!_OS 24.04 LTS**, kernel 7.0.11, **COSMIC** (`cosmic-comp` 1.0.0),
  Wayland, single 2560x1440 display, Python 3.13.

Results:

| Check | Method | Result |
|---|---|---|
| Screenshot | `cosmic-screenshot` backend | 2560x1440 captured, downscaled to 1400x788 |
| Pointer accuracy | 5 target points, read back through XWayland | 5/5 exact, 0px error |
| Buttons and scroll | `xev` event log | `ButtonPress`/`Release` for buttons 1, 3, 4 |
| Drag | `xev` motion count | 26 `MotionNotify` events between press and release |
| Typing | Both `wtype` and raw uinput keycodes | Text landed in a focused editor |
| Chords | `ctrl+a` in an editor | Applied |
| Click via MCP | Full stdio round trip against a native Wayland app | New tab opened |

Untested and worth knowing: multi-monitor layouts, HiDPI scaling, non-US
keyboard layouts on the raw-keycode path (`wtype` handles those correctly;
uinput keycodes assume US), and every non-COSMIC compositor.

Reports from other platforms are welcome — `desktop_info` dumps everything
needed to diagnose a backend problem.

## Install

```bash
git clone <this repo> && cd desktop-mcp
uv venv && uv pip install -e .
```

macOS, Windows, and Linux/X11 also need the input dependencies:

```bash
uv pip install -e ".[gui]"
```

### Linux: one-time permission setup

Injecting input on Wayland means writing to `/dev/uinput`, which is root-owned
by default:

```bash
bash scripts/setup-linux.sh   # needs sudo once
# then log out and back in
```

### macOS: one-time permission setup

Grant **Screen Recording** and **Accessibility** to whichever app launches the
server (Terminal, iTerm, Claude Desktop) in
System Settings → Privacy & Security.

## Wire it up

Claude Code:

```bash
claude mcp add desktop -- /absolute/path/to/desktop-mcp/.venv/bin/python -m desktop_mcp.server
```

Claude Desktop (`claude_desktop_config.json`) and Cursor
(`.cursor/mcp.json`) use the same shape:

```json
{
  "mcpServers": {
    "desktop": {
      "command": "/absolute/path/to/desktop-mcp/.venv/bin/python",
      "args": ["-m", "desktop_mcp.server"]
    }
  }
}
```

## Tools

| Tool | Purpose |
|---|---|
| `screenshot` | Capture the desktop |
| `click` | Left/right/middle click, single or double |
| `move` | Move the pointer without clicking (hover) |
| `drag` | Press, move, release — selections, sliders, window moves |
| `scroll` | Wheel scroll under a point |
| `type_text` | Type into the focused element |
| `key` | Chords like `ctrl+s`, `alt+tab`, `cmd+space` |
| `cursor_position` | Where the pointer is |
| `list_windows` | Open windows, where the OS permits it |
| `wait` | Pause, then look again |
| `desktop_info` | Environment, active backends, safety settings |

### Coordinates

Screenshots are downscaled (1400px long edge by default) before being sent, so
the model works in **image** pixels; the server maps them back to screen pixels
on every call. Pass exactly the coordinates you read off the screenshot.

This is the detail most home-grown computer-use tools get wrong. Miss it and
every click lands at a consistent fraction of where it was aimed.

### Why `/dev/uinput` instead of `ydotool` on Linux

Commonly packaged `ydotool` builds expose a *relative* pointer, so a jump to
(1280, 720) is delivered as one large relative motion — which the compositor
runs through pointer acceleration. The cursor lands near the target rather than
on it, and the error grows with distance. An absolute uinput device sidesteps
acceleration: measured 5/5 exact on a 2560x1440 screen. `ydotool` remains a
fallback when `/dev/uinput` is not writable.

### Click timing

Compositors dispatch pointer motion asynchronously, and toolkits ignore a
button press that arrives before they have processed the motion that put the
cursor over the widget. Measured on COSMIC: a 30ms gap between move and press
reliably *highlights* a button but never *activates* it — the hover state is
proof the coordinates are right, which makes this failure easy to misread as a
targeting bug. 250ms activates it. A press also has to be held (~90ms) to
register as a real click rather than a stray event.

Both delays are tunable; lower them only if your desktop tolerates it.

## Safety

This gives a model the same reach over the machine as the person sitting at it.
It can read anything on screen — open password managers, private messages,
tokens — and click anything, including destructive buttons. Treat a session as
if you had handed someone your keyboard.

Environment variables:

| Variable | Default | Effect |
|---|---|---|
| `DESKTOP_MCP_READ_ONLY` | off | Screenshots allowed, all input refused |
| `DESKTOP_MCP_RATE_LIMIT` | `240` | Max input actions per minute |
| `DESKTOP_MCP_BLOCK_KEYS` | — | Extra comma-separated chords to refuse |
| `DESKTOP_MCP_AUDIT_LOG` | — | Append every action to this file |
| `DESKTOP_MCP_MAX_EDGE` | `1400` | Screenshot long edge in pixels |
| `DESKTOP_MCP_MOVE_SETTLE` | `0.25` | Seconds between moving the pointer and pressing |
| `DESKTOP_MCP_PRESS_HOLD` | `0.09` | Seconds a mouse button is held down |

`ctrl+alt+F1`–`F12` are always blocked; VT switching can drop the graphical
session out from under you.

A read-only reviewer setup:

```bash
DESKTOP_MCP_READ_ONLY=1 python -m desktop_mcp.server
```

## Verify

```bash
.venv/bin/python scripts/mcp_smoke.py   # real stdio MCP round trip
```