# kwin-mcp

An MCP (Model Context Protocol) server that controls **native Wayland windows
on KDE Plasma** from an AI agent.

It does what `cua-driver` cannot on Linux/Wayland: see and drive the real
desktop. `cua-driver` (trycua) only enumerates X11/XWayland clients, so on a
KDE Wayland session it sees 1 of ~20 windows. `kwin-mcp` sees all of them.

It is built entirely on KDE-native primitives, so it needs no modifications to
trycua's binary and no root daemon. You point your MCP client (Claude Code,
Codex, Hermes, etc.) at `server.py` and get the same capabilities cua offers on
X11: window listing, screenshots, clicks, typing, dragging, key presses, and
(optionally) AT-SPI element targeting.

---

## Install

kwin-mcp targets **KDE Plasma on Wayland**. Two paths, pick one:

### Path A - Standalone CLI via pip (recommended for manual use)

You get the `kwin-mcp` command globally. No clone, no venv:

```bash
pipx install kwin-mcp-server        # or: uv tool install kwin-mcp-server
kwin-mcp --doctor                    # readiness report
kwin-mcp                             # stdio MCP server
```

Then wire it into any agent with one command (still no clone):

```bash
kwin-mcp setup hermes      # or: claude | codex | cursor | vscode | opencode | openclaw | antigravity | pi | zed | windsurf
kwin-mcp setup list        # show all supported agents
kwin-mcp setup check       # preflight only, no wiring
kwin-mcp setup verify      # preflight + confirm the real server starts and reports ready
```

### Path B - Agent wiring without a pip install (curl / uvx)

If you have not installed the package, the agent can fetch it on first launch via `uvx` (like `npx`). No clone still:

```bash
curl -sSL https://raw.githubusercontent.com/Samthesurf/kwin-mcp/main/setup.sh | bash -s hermes
# or, if you have uv installed locally:
uvx --from git+https://github.com/Samthesurf/kwin-mcp kwin-mcp setup hermes --uvx
```

Both paths run a preflight first. If a system dep is missing it prints
the exact install command and stops, so you never get a half-wired config.

> Do not do both. Path A installs the `kwin-mcp` binary and wires it as
> `command: kwin-mcp`. Path B wires it as `uvx --from git+... kwin-mcp`.
> If you already did `pipx install`, just use `kwin-mcp setup`; no `uvx` or
> `git clone` is needed.

### System deps (one-time, either path)

```bash
sudo pacman -S kdotool spectacle                 # Arch
sudo usermod -aG input "$USER"                   # allow /dev/uinput
# log out and back in so the new group applies
```

Not on Arch? See the [Dependencies](#dependencies) table below.

---

## What it can do

| Tool | Purpose |
|------|---------|
| `list_windows` | Enumerate **every** top-level window (native Wayland + XWayland), with UUID, title, class, pid, geometry |
| `active_window` | Return the currently focused window |
| `capture` | Screenshot the desktop (`mode=desktop`) or a specific window (`mode=window`, `window_id=...`); crops to exact window bounds |
| `click` / `double_click` | Click at screen or window-local coordinates, OR target an element by `element_index` or semantic `role`/`name`/`text` |
| `drag` | Drag between two points (screen or window-local) |
| `type` | Type a string into the focused target |
| `press_key` | Press a key, optionally with modifiers (e.g. `["ctrl"]`) |
| `scroll` | Scroll the wheel up/down |
| `get_window_state` | AT-SPI accessibility tree for a window (index, role, name, bounds, state flags, actions, editable) |
| `click_element` | Click an AT-SPI element by index |
| `perform_action` | Invoke any AT-SPI action on an element (press, activate, toggle, ...) |
| `set_value` | Write a value to a settable element (text fields, sliders, spinners) |
| `focus_element` | Move keyboard focus to an AT-SPI element directly (no pixel coords) |
| `focused_element` | Report which element currently owns keyboard focus |
| `keyboard_navigate` | Move keyboard focus next/prev through the focusable elements |
| `paste` | Paste text via the Wayland clipboard + Ctrl+V (fast, preserves non-ASCII) |
| `activate` / `raise` / `minimize` / `close_window` | Window management |
| `get_cursor_position` | Current pointer location |
| `health` | Environment/dependency diagnostics |
| `doctor` | One JSON readiness report (platform, windowing, input, AT-SPI, screenshot, portals, blockers) |
| `history_status` | Computer History: is encrypted action-history capture on, and how much is stored? |
| `history_query` | Computer History: bounded, metadata-only slice of past kwin-mcp actions |
| `history_control` | Computer History: local enable/disable/pause/resume/flush/delete (user-owned) |

Windows are identified by a stable KDE window UUID of the form
`{xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx}` (exactly what `kdotool` prints).

---

## Readiness report (`doctor`) and safety contract

### `doctor` / `kwin-mcp-doctor`

Run `kwin-mcp --doctor` (or `kwin-mcp-doctor`) to get a single structured JSON
document describing the desktop, the windowing backend (with a *live* window
list probe), the input path, AT-SPI, the screenshot path, and XDG portal
availability. It ends with a `readiness` summary carrying explicit `blockers`
and a `recommended_next_step`, so an MCP host or a human can render one report
instead of parsing prose:

```bash
kwin-mcp --doctor | jq .readiness
```

The same report is exposed as the `doctor` MCP tool.

### MCP safety annotations

Since v0.2 every tool carries an MCP `ToolAnnotations` so hosts can warn before
invoking a mutating tool:

| Class | Tools | Contract |
|-------|-------|----------|
| Read-only observation | `list_windows`, `active_window`, `get_window_state`, `get_cursor_position`, `health`, `doctor` | `readOnlyHint=true` |
| UI-state mutators | `capture`, `activate`, `raise_window`, `minimize`, `scroll` | `readOnlyHint=false`, `destructiveHint=false` |
| Desktop-action mutators | `click`, `click_element`, `drag`, `type_text`, `press_key`, `perform_action`, `set_value`, `close_window` | `destructiveHint=true` (+ `openWorldHint=true`) |

Annotations are safety hints, not an authorization system. Treat any call that
could submit, delete, send, or purchase as requiring user approval.

## Troubleshooting

### Electron / Chromium apps show an empty AT-SPI tree

Electron and Chromium build their accessibility tree lazily and only when the
app opts in. Until then `get_window_state` returns
`{"available": true, "elements": [], "count": 0}` (the result carries a `hint`
when this happens). It is an app-side setting, not a kwin-mcp bug. Fix it one
of two ways:

- Launch the app with `--force-renderer-accessibility`, or
- call `app.setAccessibilitySupportEnabled(true)` inside the app's main process.

### Clicks into an unfocused window do not land (Wayland)

Wayland routes synthetic input to the focused surface; a click into an
unfocused window is silently dropped. kwin-mcp's window-targeted tools
(`click` with `window_id`, `click_element`, `drag`) focus the window first and
then verify delivery (focus state + pointer position). When verification
fails the result reports `ok: false` with a
`input-not-delivered (window not focused?)` warning instead of a false
success. If you see that warning, activate the window first (`activate`) and
retry, or prefer element targeting (`click_element` / `perform_action`),
which uses AT-SPI `Action.DoAction` and needs neither focus nor a cursor.

### Element clicks hit the wrong control

`click_element` activates an element's AT-SPI action directly (protocol-level,
coordinate-free). If an element exposes no AT-SPI actions, the tool falls back
to clicking its on-screen center with synthetic pointer events; that fallback
is reported as `method: "coordinate_fallback"` in the result. If you need
guaranteed correct targeting on an app whose coordinates look wrong, use
`perform_action` (AT-SPI only, never falls back to pixels).

## Computer History

A port of Cua Driver's encrypted, metadata-only **Computer History** preview
(`libs/cua-driver/docs/computer-history-*.md`). It gives you a local, inspectable
record of what kwin-mcp did, when, and which app it targeted, without turning the
server into a screen recorder or keylogger.

**Privacy boundary (permanent):** history records only fixed-field metadata. It
never stores screenshots, typed text, clipboard contents, raw tool arguments or
results, accessibility trees, window titles, URLs, or file paths. Every event is a
CloudEvents 1.0 envelope on `urn:kwin-mcp:schema:history-event:v0`.

**Encrypted at rest:** each event is sealed with AES-256-GCM before any bytes hit
disk (no plaintext fallback). The key is a 256-bit in-memory secret; `delete`
destroys the key and erases the store.

**Opt-in, off by default.** Nothing is recorded until you enable it. Agents can
only *read* history (`history_status`, `history_query`); capture lifecycle,
retention, and deletion are owned locally (mirroring Cua's
`history_control_requires_local_cli`).

Enable it from the server process (e.g. via the `history_control` tool, or a
local CLI), then let an agent query bounded slices:

```json
{ "tool": "history_control", "arguments": { "operation": "enable" } }
{ "tool": "history_query",   "arguments": { "limit": 50, "since_sequence": 1 } }
```

| Tool | Purpose |
|------|---------|
| `history_status` | Read-only: supported, enabled, paused, encrypted, retention/quota, bytes used, dropped events, health. Never returns events. |
| `history_query` | Read-only: a bounded, metadata-only event slice (`limit` 1..200, optional `session_id` / `since_sequence` / `until_sequence`). A successful read appends an encrypted access record (not returned). |
| `history_control` | Local only: `enable` / `disable` / `pause` / `resume` / `flush` / `delete` the encrypted store. |

Recorded events cover the 14 mutating/action tools (`click`, `drag`, `type_text`,
`paste`, `press_key`, `scroll`, `click_element`, `perform_action`, `set_value`,
`focus_element`, `activate`, `raise_window`, `minimize`, `close_window`) as
`action_started` / `action_completed` envelopes, classified by effect
(confirmed, partial, unverifiable, suspected_noop, refused, failed) and route
(synthetic_events, trusted_input, global_input, accessibility, system_api).

## Dependencies

### System packages (must be installed on the machine)
These are the KDE/Wayland tools the server shells out to. Install with your
distro's package manager.

| Tool | Package (Arch) | Package (Debian/Ubuntu) | Used for |
|------|----------------|--------------------------|----------|
| `kdotool` | `kdotool` (AUR) | `kdotool` (build from source) | Window enumeration, geometry, focus |
| `spectacle` | `spectacle` | `kde-spectacle` | Screen capture |
| `ydotool` | `ydotool` | `ydotool` | (Optional) alternative input backend reference |
| `grim` | `grim` | `grim` | (Optional) future per-output capture |

On Arch this machine already had `kdotool`, `spectacle`, `grim`, `ydotool`,
`slurp`, and `busctl` available.

### Kernel / group requirements (input)
Synthetic input is sent through a virtual device on `/dev/uinput`. You must:

1. Be a member of the `input` group:
   ```bash
   groups | grep -w input || sudo usermod -aG input "$USER"
   # then log out and back in
   ```
2. Have write access to `/dev/uinput` (group `input` owns it:
   `crw-rw---- root input`). No root daemon (`ydotoold`) is required because
   `python-uinput` opens the device directly as a group member.

Verify with:
```bash
ls -l /dev/uinput          # should show group 'input' with rw
id -nG | tr ' ' '\n' | grep -x input   # should print 'input'
```

### Python packages
```bash
python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
# AT-SPI element/action/value targeting + semantic clicks work out of the box:
# kwin-mcp talks to AT-SPI directly over D-Bus via jeepney (already a
# dependency), so no pyatspi is required. On distros where the legacy pyatspi
# module happens to be installed, it is used as a fallback backend.
```

Installed and verified on this build: `mcp 1.28.1`, `python-uinput 1.0.1`,
`Pillow 12.3.0` (Python 3.14).

---

## Running

```bash
. .venv/bin/activate

# dependency preflight (also run automatically by setup.sh)
python server.py --check

# JSON readiness report
python server.py --doctor

# stdio MCP server (for Claude/Codex/Hermes MCP clients)
python server.py

# or via the convenience wrapper
python run.py

# Streamable HTTP transport on 127.0.0.1:8080
python server.py --http 8080
```

The smoothest path is the one-command `kwin-mcp setup` described in `## Install` — no local venv needed.

### Wiring into an MCP client (what `kwin-mcp setup` does)

`kwin-mcp setup <agent>` preflights, then injects the correct entry into
your agent's config (and never half-wires). The manual equivalent is to
point the client at the installed command:

```json
{
  "mcpServers": {
    "kwin-mcp": { "command": "kwin-mcp", "args": [] }
  }
}
```

If you used `--uvx` (no pip install), the wired entry is instead
`{ "command": "uvx", "args": ["--from", "git+https://github.com/Samthesurf/kwin-mcp", "kwin-mcp"] }`.

Supported agents: `hermes` (~/.hermes/config.yaml), `claude` (~/.claude.json),
`codex` (~/.codex/config.toml), `cursor`, `vscode` (settings.json),
`opencode` (~/.config/opencode/opencode.json), `openclaw`, `antigravity`,
`pi`, `zed` (context_servers), `windsurf`. Run `kwin-mcp setup list` for the full table with paths.
`./setup.sh` is still supported as a legacy alias (`./setup.sh hermes --uvx`).

**Manual examples:**
- **Claude Code**: `claude mcp add kwin-mcp -- kwin-mcp` (pip path) or `claude mcp add kwin-mcp -- uvx --from git+https://github.com/Samthesurf/kwin-mcp kwin-mcp` (uvx path)

The server is self-sufficient about its environment: when an MCP client does
not forward `DBUS_SESSION_BUS_ADDRESS` / `WAYLAND_DISPLAY` / `DISPLAY` /
`XDG_RUNTIME_DIR`, the server discovers the correct session values from
`/run/user/<uid>/` so `kdotool` and `spectacle` always work.

No API keys, no network calls, no cloud. Everything runs locally against your
compositor.

### Running from a local checkout (alternative)

If you prefer a local venv instead of `uvx`:

```bash
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
python server.py            # stdio MCP server
python server.py --check   # dependency preflight
```

---

## How it works (and the Wayland caveats)

On Wayland there is no X server between apps and the compositor, so input
cannot be injected "into a specific window" the way cua-driver does on X11.
The bridge follows a **focus-then-inject** model:

1. `kdotool windowactivate <uuid>` raises and focuses the target window.
2. The virtual pointer (a `python-uinput` device) is moved to the target
   coordinate. Because the compositor applies mouse acceleration and uinput
   only emits *relative* motion, movement is **closed-loop**: read the real
   cursor, emit a bounded delta, re-read, repeat until within ~3 px. This makes
   absolute positioning deterministic.
3. The click / key / drag is emitted on the now-focused window.

What this costs versus X11 (inherent to Wayland, not a bug):

- **No background targeting.** The window must be focused first; the real
  cursor moves. It is not invisible the way background X11 input can be.
- **Single cursor.** Parallel multi-pointer drags (cua's `parallel_mouse_drag`)
  are not available on Wayland.
- **Secure-input surfaces** (some password fields, the lock screen) may reject
  synthetic input.
- **Small focus race.** Between focusing and injecting there is a brief window
  where focus could shift; the code waits ~250 ms after activation.

Screenshots use `spectacle` in background/non-interactive mode. On KDE Wayland
`--background` can occasionally race the compositor and capture the lock-screen
splash instead of the live desktop; the capture path adds a settle delay and a
variance-based validation that retries up to 3 times, so the returned frame is
always the real desktop.

AT-SPI (`get_window_state`, `click_element`, `perform_action`, `set_value`,
semantic clicks) works for GTK/Qt/KDE apps that expose an accessibility tree.
It talks to AT-SPI directly over D-Bus (via `jeepney`, a pure-Python client),
so it needs no `pyatspi` and works on Arch; the legacy `pyatspi` module is used
only as a fallback if present. It degrades gracefully to coordinate input when
no AT-SPI backend is available.

---

## Project layout

```
kwin-mcp/
├── server.py              # MCP server (FastMCP) exposing all tools
├── run.py                 # convenience entry point
├── requirements.txt
├── pyproject.toml
├── mcp-config.example.json
├── README.md
└── kwin_bridge/
    ├── __init__.py
    ├── windows.py         # kdotool wrapper: enumerate/geometry/focus/close
    ├── screenshot.py      # spectacle wrapper + crop + retry/validate
    ├── input.py           # /dev/uinput virtual pointer+keyboard, closed-loop move
    ├── a11y.py            # AT-SPI front-end (semantic resolve / action / value)
    ├── atspi_dbus.py      # pure-D-Bus AT-SPI backend (jeepney, no pyatspi)
    ├── doctor.py          # structured JSON readiness report
    └── preflight.py       # actionable dependency check
```

---

## Testing

A quick smoke test against the live desktop:

```bash
. .venv/bin/activate
python - <<'PY'
from kwin_bridge import windows, screenshot, input as inp
ws = windows.list_windows()
print("windows:", len(ws))
wid = ws[0].window_id
print("capturing", wid)
p = screenshot.capture_window(wid, "/tmp/test.png")
print("shot:", p)
inp.click_window(wid, 100, 100)
inp.type_text("hello from kwin-mcp")
PY
```

---

## License

MIT. Use it, fork it, ship it.