ccdesk
by asloth
README.md
# ccdesk
**Claude Code can't see your desktop on Linux. ccdesk gives it eyes and hands.**
It also works with other MCP clients: see [Using it with opencode](#using-it-with-opencode).
A small MCP server that lets Claude Code take screenshots, move and click the mouse, type,
press keys, list and focus windows, use the clipboard and read the accessibility tree, plus a
skill that teaches the safe loop: look, one action, verify.
*Recorded by `bench/record_demo.py`: a real ccdesk session on a private Xvfb screen, not a mockup.*
## Results
Same GTK signup form filled end to end. Claude Code 2.1.304, Opus 5.5, `claude -p`, no user
settings or hooks, 3 runs per arm, all 9 succeeded. Input tokens processed, median:
| | no ccdesk | screenshots only | ui_tree + changes | + `--always-load` |
|---|---|---|---|---|
| Input tokens | 89,456 | 117,743 | 63,361 | **56,681** |
| Wall time | 30 s | 22 s | 15 s | **15 s** |
| Images read | 0 to 3 | 3 | **0** | 0 |
- **37% fewer input tokens** than doing it without ccdesk, and 2x faster.
- The saving is fewer round trips, not cheaper looks: one `ui_tree()` replaces several
screenshots, and after the first tree every action returns only the lines that changed.
- Screenshots alone are the expensive path (118k), because a full-screen first look plus one
tool call per click costs more than the shell commands Claude writes when it has no help.
- Since then `screenshot` shows the picture in its answer instead of needing a Read: 136k → 78k on
that path in a later batch (median of 5, a different setup, so don't compare it with this table).
- Raw numbers, method and caveats: [`bench/RESULTS.md`](bench/RESULTS.md). Reproduce with
`python bench/run.py`.
## Quickstart
```sh
sudo apt install xdotool scrot wmctrl xclip x11-utils x11-xserver-utils # Debian/Ubuntu, X11
pipx install ccdesk # from PyPI; today the wheel on the Releases page, or pipx install . in a checkout
ccdesk doctor # session type, helper tools, screen size, accessibility bus
ccdesk install # register the MCP server with Claude Code and install the skill
```
Restart Claude Code and run `/mcp`: `ccdesk` should be listed as connected. To try it on a
private screen instead of yours, see [Headless](#headless-let-it-work-while-you-use-your-pc).
### Other distributions
On Fedora: `sudo dnf install xdotool scrot wmctrl xclip xprop xwininfo xrandr`. On Arch:
`sudo pacman -S xdotool scrot wmctrl xclip xorg-xprop xorg-xwininfo xorg-xrandr`.
`xwininfo` gives the size of the whole screen across monitors and `xrandr` lists the
monitors; without them ccdesk falls back to `xdotool`, which recent releases report as
one monitor only.
On KDE Plasma under Wayland, the X11 tools are not what you need: see
[KDE Plasma](#kde-plasma-wayland). Plasma on X11 uses the X11 tools above.
## Why it exists
There's no Claude Desktop app on Linux. ccdesk fills that gap with the system tools you
probably already have (`xdotool`, `scrot`, `wmctrl`, `xclip`). The core is standard-library
Python; the only dependency is the official `mcp` package.
## Wiring it into Claude Code
`ccdesk install` does two things:
1. Copies `skills/computer-use/SKILL.md` to `~/.claude/skills/computer-use/SKILL.md`
(or `$CLAUDE_CONFIG_DIR/skills/...`).
2. Runs `claude mcp add --scope user ccdesk -- /path/to/ccdesk mcp`, after removing any
earlier `ccdesk` registration.
Flags you pass to `install` are forwarded to the server: `ccdesk install --read-only`,
`ccdesk install --allow-launch` or `ccdesk install --confirm-keys "super+x"`.
`--always-load` puts the tool schemas in every session's prompt instead of behind Claude
Code's tool search: one round trip less when you use the desktop, ~2.6k tokens more per
request when you don't (see `bench/RESULTS.md`). `--dry-run`
prints the exact file, size, checksum and commands, and changes nothing:
```text
$ ccdesk install --dry-run
ccdesk install --dry-run (nothing is written)
[1] would write /home/hermes/.claude/skills/computer-use/SKILL.md
6198 bytes, sha256 b0dbc9581ca05b94…, copied from /home/hermes/Projects/ccdesk/skills/computer-use/SKILL.md
[2] would run: claude mcp remove --scope user ccdesk # ignored if not registered yet
[3] would run: claude mcp add --scope user ccdesk -- /home/hermes/Projects/ccdesk/.venv/bin/ccdesk mcp
```
To wire it by hand instead:
```sh
claude mcp add --scope user ccdesk -- ccdesk mcp
mkdir -p ~/.claude/skills/computer-use && cp skills/computer-use/SKILL.md ~/.claude/skills/computer-use/
```
Restart Claude Code and run `/mcp`. `ccdesk` should be listed as connected.
## Using it with opencode
ccdesk is a standard MCP server, so opencode can run it too. Let opencode write the entry, with
ccdesk's full path. This stops if ccdesk is not on your PATH:
```sh
bin=$(command -v ccdesk) && opencode mcp add --global ccdesk -- "$bin" mcp # add --headless for a private screen
opencode mcp list # ccdesk should say connected
```
The first `mcp list` from a folder can say `No MCP servers configured` while the server starts. Wait
a few seconds and run it again. If opencode was already running when you added it, run
`opencode service restart` first.
Then keep its tools as ordinary tools, in `~/.config/opencode/opencode.json`. Running `mcp add` again
removes this setting. Keep the command it wrote, with its full path:
```json
{
"mcp": {
"servers": {
"ccdesk": {
"type": "local",
"command": ["/full/path/to/ccdesk", "mcp"],
"codemode": false
}
}
}
}
```
opencode's default, Code Mode, has the model call tools from code. ccdesk's loop relies on each
result and screenshot reaching the model directly, so `codemode: false` is the recommended setting.
opencode already reads the skill from `~/.claude/skills`, so `ccdesk install` covers it. Without
Claude Code, copy it yourself:
```sh
mkdir -p ~/.config/opencode/skills/computer-use
cp skills/computer-use/SKILL.md ~/.config/opencode/skills/computer-use/
```
To restrict the tools, opencode's permission rules match them as `ccdesk_<tool>`.
**Tested on opencode v2.0.32**, once, with `ccdesk mcp --headless`, `codemode: false` and
`deepseek-v4.1-flash-free`: its agent launched the benchmark form, described it from a screenshot,
then filled and submitted it through `ui_tree`, `set_value` and `click_element`, and the saved values
were correct. Screenshots need a model with vision. **Not tested:** Code Mode with a full task, input
on your real screen, or other models. The token numbers in this README were measured on Claude Code
only.
## Headless: let it work while you use your PC
```sh
ccdesk install --headless # or: ccdesk mcp --headless [--size 1600x900]
ccdesk peek --open # see what the agent's screen shows right now
```
With `--headless` the server starts its own invisible screen and does everything there, so
you keep your mouse, keyboard, windows and clipboard while Claude works. When Claude Code
closes the server, the screen and everything on it go away (also if the server is killed:
its helpers die with it).
What it starts: `Xvfb` on a free display number, a window manager (the first of openbox,
fluxbox, icewm, matchbox, xfwm4, else bspwm with a stub config; `--wm none|CMD` to choose),
and a private D-Bus session with its own accessibility bus. The private bus means:
- `ui_tree` sees only the agent's apps, never yours.
- Apps don't hand their window to a copy already running on your screen (Firefox, GNOME apps).
- Apps on that screen can't reach your keyring, notifications or other session services.
- Accessibility is on for GTK, Qt and Firefox without touching your settings.
The screen starts empty, so `--headless` turns on `launch`. A browser that is already open
on your screen locks its profile; give the agent its own:
`firefox --no-remote --profile ~/.cache/ccdesk/firefox` (create the folder first) or
`chromium --user-data-dir=~/.cache/ccdesk/chromium --force-renderer-accessibility`.
It runs X11 apps, and works the same from an X11 or a Wayland session (GNOME and KDE
included): apps on it are told to use X11.
Needs `Xvfb` (Arch: `xorg-server-xvfb`, Debian/Ubuntu: `xvfb`), plus `dbus-daemon` and
`at-spi2-core`, which desktops already have. `ccdesk peek` saves a screenshot of every running
headless screen to `~/.cache/ccdesk/peek-N.png`; for a live view, run
`x11vnc -display :N -localhost` and any VNC viewer (`doctor()` shows the display).
### Both screens: let Claude pick per task
Headless can't see your logged-in browser, and copying a browser profile to get its cookies
is a bad idea. Register a second server on your real screen, and Claude chooses one per task:
```sh
claude mcp add --scope user ccdesk -- ccdesk mcp --allow-launch --headless
claude mcp add --scope user ccdesk-screen -- ccdesk mcp
```
Use `ccdesk` for work that doesn't need your accounts, and `ccdesk-screen` for anything that
does, like a web console or a chat you're signed in to. Keep your hands off the mouse while
Claude works on `ccdesk-screen`. Restart Claude Code after adding a server; it reads MCP
servers only at startup.
### Several agents on one screen
A screen has one pointer and one keyboard focus. Every input action takes a lock on the display
for the focus check and the clicks or keys, so two ccdesk servers on the same display take turns
one action at a time and never interleave inside an action. Waiting for the app to settle and
comparing the window afterwards only reads the screen, so it runs after the lock is released.
A `batch` call holds the lock for all of its steps. Give each server
a name with `--agent`:
```sh
ccdesk mcp --agent bot-1
ccdesk mcp --agent bot-2
```
With `--agent`, when another agent has moved focus away from the window this agent activated,
ccdesk activates that window again and sends the input. Without it, ccdesk refuses, because the
one who moved focus is probably you. Audit entries carry the agent's name.
Screenshots show the drawn cursors of every agent on the screen, this one's included. With
`--yield-ms`, input that comes right after any agent's counts as that agent's, and an agent that
waited in line for the screen checks once more that you are still away before it sends anything.
On X11, each named agent also gets a drawn cursor: an arrow in the agent's colour with a small
label holding its name. It moves to the point of each click, move, drag, scroll and click_element,
and to the field set_value writes to. It is only a picture: the real pointer still does the
clicking, and the overlay window takes no input and is left out of `list_windows`. It needs a
system Python with PyGObject (the same one `ui_tree` uses); if that is missing the cursor turns
itself off and `doctor` says so. The colour is picked from the name; set it with
`--cursor-color '#RRGGBB'`, or turn the cursor off with `--no-cursor`.
In a test on Xvfb, two agents typing 40 words each into two windows got their own text intact
in 4.1 s. With no lock, both windows came out with letters from both agents.
With two agents typing 15 words each and no `ui_tree`, comparing outside the lock cut the time an
action holds it from 0.48 s to 0.13 s, and both finished in 7.4 s instead of 14.4 s (2026-11-07).
Because another agent may act while this one compares, a `screen_changed` can now include that
agent's drawn cursor or its window moving over this one.
#### An own pointer and keyboard for each agent (X11)
`ccdesk mcp --agent bot-1 --own-pointer` gives the agent its own pointer and keyboard focus, so it
does not use yours and does not wait for the other agents. At startup it runs
`xinput create-master ccdesk-bot-1`, which adds a master pointer and keyboard (with their XTEST
devices) that only this agent drives. Moves, clicks, drags, scrolling, typing and key presses go
through them; your pointer stays where it is and your keyboard focus does not change. It
needs `xinput`, libX11, libXi and libXtst, and it is X11 only. It is refused without `--agent` and
on other backends. Names must be unique on a screen: a second server with the name of a running
one refuses to start ("use another --agent name"). The devices are removed when the server exits,
also on SIGTERM and SIGINT. After a `kill -9` or a crash, the leftovers of dead agents (the ones
whose name is not in use) are removed at the next own-pointer start or by `ccdesk doctor`, which
lists what it removed.
Window managers that grab clicks can move your keyboard focus when the agent clicks. bspwm does
this on every unfocused window, so with bspwm each agent button press runs with
`click_to_focus` set to `none` for a moment (under a lock shared by all agents on the screen) and
the saved value is put back right after; `doctor` or the next start restores it if a process died
in between. Other window managers are not handled, and `doctor` says so: an agent click there may
move your keyboard focus.
`activate_window` sets the agent's keyboard focus and leaves your window manager alone. Only a
window that is not on screen (minimized, or on another desktop) goes through the normal
activation, which does move your focus once; the answer says so in `note`. A window another
window covers still receives no clicks: the click is refused, as before. The focus check
compares against the agent's own focus and the click check against the window under the agent's
pointer. Each own-pointer agent has its own input lock, so two of them type at the same time.
`--yield-ms` is ignored: X counts every agent's input, own-pointer input included, as activity
(it also resets screensavers and the idle timer other agents' `--yield-ms` reads), so own-pointer
agents would wait on each other. The drawn cursor follows the agent's pointer.
Characters that have no key in your keymap are typed through a spare keycode that is remapped
for a moment, as xdotool does. `doctor` shows the device ids, or why they are unavailable.
#### A named private screen for several agents
Each `--headless` server starts its own private screen, so two of them never see each other's
windows. To share one private screen, start it on its own and have the servers join it:
```sh
ccdesk screen start work [--size 1600x900] [--wm CMD] # foreground; Ctrl-C or `screen stop` ends it
ccdesk mcp --screen work --agent bot-1
ccdesk mcp --screen work --agent bot-2
ccdesk screen list
ccdesk screen stop work
```
`screen start` prints how to watch it (`x11vnc -display :N -localhost`); `ccdesk peek` also
lists named screens. `--screen` turns on `launch` like `--headless` and cannot be combined with
it. A server that exits leaves the screen running. `ccdesk install --screen work` registers the
server with that flag. The state file is `$XDG_RUNTIME_DIR/ccdesk/screens/NAME.json` (mode 0600,
in a 0700 directory) and holds the display, size, window manager, pid and only the variables a
joining server needs.
### Firefox for Claude, without the mouse (opt-in)
For web pages, ccdesk can hand off to Mozilla's
[firefox-devtools MCP server](https://github.com/mozilla/firefox-devtools-mcp). It drives
Firefox over WebDriver BiDi: it reads the page as text (a small page's snapshot is a few dozen
tokens) and clicks inside it, so your mouse and keyboard stay yours. Needs Node.js for `npx`.
```sh
ccdesk install --firefox # also registers firefox-devtools (npx, pinned, --tool-preset slim)
```
Claude then has **its own Firefox**: the server opens it the first time Claude needs a page,
with a profile kept in `~/.local/share/ccdesk/firefox`. Sign in there once to the sites Claude
should use (only those, not your mail or bank); the logins stay for next time. Its window
has an orange strip on top and an orange active tab, so you can tell it from yours (the
profile's `chrome/userChrome.css`). Your own Firefox is never restarted or flagged. That browser tells sites it is automated
(`navigator.webdriver`), so some (Cloudflare, Akamai) may challenge it.
To let Claude use **your** Firefox and the logins it already has instead:
```sh
ccdesk install --firefox mine
ccdesk firefox # start your Firefox with --marionette --remote-debugging-port
ccdesk firefox --off # afterwards: start it normally again
```
Firefox only takes the flags when it starts, so `ccdesk firefox` asks you to quit a running
Firefox first (Ctrl+Q; session restore brings your tabs back); it never quits it for you.
While the flags are on, sites can see the automation in your everyday browsing, and any
program on this PC can drive your browser through the local port. `ccdesk doctor` warns
while it is on.
## Tools
| Tool | What it does |
| --- | --- |
| `screenshot(region?, path?, scale=1, window?)` | Saves a PNG and shows it in the answer, with `{path, width, height, approx_tokens}`, so seeing it takes no second request. With `path` it only saves it, and a PNG over 3.5 MB stays a file with a note. `region` is `"x,y,w,h"`; region results also include `origin`. `window` (id or title) captures just that window: a crop of the screen, so whatever covers it shows too, and it refuses a window that is minimized or on another desktop. `scale=0.5` halves each side (needs ImageMagick). |
| `click(x, y, button="left", count=1, force=false, expect_title?, show_change=false)` | Moves the pointer and clicks. `count=2` double-clicks. `show_change=true`: see [Without a tree](#without-a-tree). |
| `move(x, y)` | Moves the pointer (hover). |
| `drag(from_x, from_y, to_x, to_y, duration_ms=400, force=false, show_change=false)` | Left-button drag along a smooth path. |
| `scroll(x, y, dx, dy, force=false, show_change=false)` | Wheel clicks with the pointer at (x, y). `dy > 0` scrolls down. |
| `type_text(text, submit=false, replace=false, force=false, expect_title?, show_change=false)` | Types into the focused widget. `replace=true` presses ctrl+a first; `submit=true` (or a trailing `\n`) presses Return after a short pause. |
| `press_key(key, modifiers=[], confirm=false, force=false, expect_title?, show_change=false)` | `"Return"`, `"ctrl+l"`, or `key="t", modifiers=["ctrl","shift"]`. Shortcuts that move focus (`ctrl+t`, `ctrl+l`, `alt+d`, `F6`...) wait a moment so the next `type_text` lands in the new place. |
| `list_windows()` | Returns `id`, `title`, `class`, `geometry`, `desktop` and `active` for each window, plus `current_desktop`. |
| `activate_window(title_or_id)` | Raises and focuses a window by id or by partial title. Later input is refused if it would reach another window. |
| `manage_window(title_or_id, action, x?, y?, width?, height?)` | `move`, `resize`, `minimize`, `maximize` or `restore` a window. Returns `before`, `after`, `on_show` and `changed`, read back from the window system, so a request the window manager ignored shows as `changed: false`, as does a window that was already like that. |
| `list_monitors()` | Each monitor's name and box, plus the size of the whole screen. |
| `clipboard(action="get"\|"set", text?)` | Reads or replaces the clipboard. |
| `wait(seconds)` | Sleeps, up to 300 s. Right after an action, says `wait_for_change()` would have been the better call. |
| `wait_for_change(region?, timeout=10, settle_ms=500)` | Waits (up to 300 s) until the screen differs from how it looked right after the last action, then until it is still for `settle_ms`. Returns `changed`, `settled`, `waited_ms` and `changed_area`. Needs ImageMagick. |
| `doctor()` | Reports the session type, tools found, screen size, scale and policy. |
| `ui_tree(window?, query?, page=false, max_nodes=300, shot_fallback=true)` | The window's buttons, fields, checkboxes, links and text as compact lines from the accessibility tree (AT-SPI), with values, states and screen boxes. |
| `click_element(ref, button="left", count=1, force=false, expect_title?)` | Clicks an element from the latest `ui_tree()`; refuses if it changed or left the screen. A single left click on a button, checkbox, radio, link, tab or menu item runs its accessibility action when it has one, so it needs no pointer, scaling or free line of sight (the result says `activated`). Fields and everything else get a real click at the centre (`clicked`), and so does any click with `expect_title`, which has to check the window that receives it. |
| `set_value(ref, text, force=false)` | Replaces the value of a field, slider or spin button in one call and reads it back: success means the app holds it (`verified`). Refuses password fields. When the app ignores or refuses the write and keeps its old value, as Firefox and Chromium web forms do, it clicks the field, selects all and types the text, then reads it back (`typed: true`). |
| `batch(steps, show_change=false)` | Runs 1 to 20 input steps (`click_element`, `set_value`, `type_text`, `press_key`, `click`, `scroll`, `move`, `drag`, `activate_window`, `wait` up to 5 s) in one call: the seat is taken once, every step keeps its own checks (destructive keys still need `confirm=true`) and audit entry, and only the final state is observed (`changes` or `screen_changed`). All steps are validated before the first one runs; it stops at the first failure and reports `done`, `steps` and `failed`. Use it to fill a form from one `ui_tree()`. |
| `launch(command)` | Starts an app without a shell. Only enabled with `--allow-launch`. |
Coordinates are screen pixels with the origin at the top left, the same space as a
full-screen screenshot.
## Demo
This transcript is a real stdio session against `ccdesk mcp`, recorded under `xvfb-run`
while driving the Tk test app in `tests/tk_app.py`. Each `>` line is a tool call and each `<`
line is the server's response. The pixel coordinates were found in the screenshot. In
Claude Code, Claude does the same thing by reading the PNG.
```text
> list_windows()
< {"windows": [{"id": "0x00200029", "title": "ccdesk-test", "class": "tk.Tk", "geometry": {"x": 40, "y": 50, "width": 420, "height": 420}, "desktop": null}]}
> activate_window(title_or_id='ccdesk-test')
< {"activated": {"id": "0x00200029", "title": "ccdesk-test", "class": "tk.Tk", "geometry": {"x": 40, "y": 50, "width": 420, "height": 420}, "desktop": null}}
> screenshot()
< {"path": "/home/hermes/.cache/ccdesk/screenshots/screen-20261019-200511-959.png", "width": 1280, "height": 1024}
> click(x=249, y=79)
< {"clicked": {"x": 249, "y": 79}, "button": "left", "count": 1}
> screenshot(region='99,49,300,110')
< {"path": "/home/hermes/.cache/ccdesk/screenshots/screen-20261019-200514-075.png", "width": 300, "height": 110, "origin": {"x": 99, "y": 49}, "note": "pixel (px,py) in this image is screen (origin.x+px, origin.y+py)"}
> click(x=249, y=179)
< {"clicked": {"x": 249, "y": 179}, "button": "left", "count": 1}
> type_text(text='hello from Claude')
< {"typed_chars": 17}
> press_key(key='Return')
< {"pressed": "Return"}
> press_key(key='w', modifiers=['ctrl'])
< ERROR Error executing tool press_key: Ctrl+w can close windows or quit applications; take a screenshot to verify the target, then call again with confirm=true
> launch(command='xterm')
< ERROR Error executing tool launch: launch is disabled: restart the server with `ccdesk mcp --allow-launch`
```
When the session ended, the app reported `label: "submitted:hello from Claude"`. In Claude
Code a typical request looks like *"open Firefox, go to news.ycombinator.com and tell me the
top story"*. The skill makes Claude find or focus Firefox, use `ctrl+l` for the address bar,
type the URL, wait, screenshot and read the page.
## Token cost
Screenshots are most of what a ccdesk session costs. The model pays about `width*height/750`
input tokens per image it reads (each result includes `approx_tokens`), and the image stays in
context for every later turn. Measured on a real Play Console session: a 1920x1080 monitor
capture is ~2,800 tokens, and 17 of them were about 70% of the context the session used.
| Capture | Size | Tokens |
| --- | --- | --- |
| one monitor, full scale | 1920x1080 | ~2,765 |
| one monitor, `scale=0.6` | 1152x648 | ~995 |
| one monitor, `scale=0.5` (still readable) | 960x540 | ~691 |
| region around a form field | 600x300 | ~240 |
The cheapest way to look is not a picture at all. `ui_tree()` reads the accessibility tree
(AT-SPI, what screen readers use) and returns the on-screen elements as text: a form is
usually 100 to 600 tokens, and on a test form a full fill-and-save took two `ui_tree()` calls
of ~150 tokens each and no screenshots. Clicks land on the element's real box instead of an
estimate from pixels.
Apps only publish the tree when accessibility is on when they start:
```sh
gsettings set org.gnome.desktop.interface toolkit-accessibility true # GTK apps and Firefox
# or start them with GNOME_ACCESSIBILITY=1; Chromium/Electron: --force-renderer-accessibility
```
Outside GNOME (bspwm, i3, sway...) that is often not enough: the session's accessibility bus
has its own switch, `org.a11y.Status IsEnabled`, that nothing turns on, and Firefox reads it
at startup. `ccdesk doctor` shows it as `a11y bus: off` and prints the fix:
```sh
busctl --user set-property org.a11y.Bus /org/a11y/bus org.a11y.Status IsEnabled b true
# then restart Firefox; the setting lasts until logout, so add the line to .xinitrc or bspwmrc
```
When `ui_tree()` finds no window, its error names the window that is missing from the bus,
what is on the bus, and this fix, so Claude can ask you to apply it. Together with that text,
`ui_tree()` attaches a picture of the window (at most about 1500 tokens), so the model does not
need another request for `screenshot()`. It does the same when the tree has fewer than 3 lines and
none with a button, field or other role to act on (an Electron or canvas window). Not with
`query=`, and `shot_fallback=false` turns it off.
The tree is read by `ccdesk/a11y_helper.py`, which runs under a system Python with PyGObject
(`python-gobject` on Arch, `python3-gi` on Debian, plus `at-spi2-core`), so ccdesk itself keeps
no compiled dependencies. `CCDESK_A11Y_PYTHON` picks another interpreter. `ccdesk doctor`
lists the apps that publish a tree. The helper stays alive for the session (one JSON request
per line on stdin), so a read does not pay for a Python start and an AT-SPI handshake every
time; it exits when ccdesk does.
After the first `ui_tree()`, actions (`click`, `click_element`, `type_text`, `press_key`)
return `changes`: only the lines of that window that changed, such as
`e5 check box "I accept the terms" [checked]`. The model verifies from that instead of
reading the tree or the screen again. Refs stay stable across calls. The helper watches the
window for up to 0.3 s and answers the moment a line changes, so a fast app reports its effect
at once instead of after a fixed pause: on the bench's GTK form a click went from 605 ms to
22 ms (`bench/RESULTS.md`).
Measured end to end (`bench/`, same form task, Claude Code's own usage numbers, 3 runs each):
no ccdesk 89k input tokens, ccdesk with screenshots 118k, ccdesk with `ui_tree` and `changes`
63k (−29%), 57k with `--always-load` (−37%). Without ccdesk Claude already screenshots just
the window, so the saving is mostly fewer round trips, not smaller pictures.
Without a tree (apps that don't publish one, or the bus switched off), `click`, `type_text`,
`press_key`, `scroll` and `drag` compare a 1/8-size grayscale copy of the receiving window
from before and after the action and return `screen_changed` and, if it changed,
`changed_area`. A blinking text caret doesn't count. "Save did nothing" is then one call
instead of a wait and two screenshots, and `wait_for_change()` replaces fixed `wait()`s for
pages and dialogs. Each comparison costs about 0.2 s per capture (ImageMagick does the
shrinking; without it the fields are left out).
<a id="without-a-tree"></a>`show_change=true` on those same actions also puts a capture of
`changed_area` (padded 24 px, scaled down to about 600 tokens at most) in the answer as
`change_shot`, so the check needs no screenshot of its own. And when a screenshot taken right
after an action is more than four times the size of what changed, or a `wait()` of a second
or more follows an action, the result carries a `hint` naming the cheaper call. In a real
session without a tree (filling three web forms of the same shape) most of the 117 screenshots were
half-scale full-monitor checks of a click that changed a few hundred pixels, at ~700 tokens
each, and 86 fixed `wait()`s did what `wait_for_change()` does.
The skill tells Claude to orient at `scale=0.5`, then act and verify with small full-scale
regions, and to pull long page text through the clipboard instead of reading it from pixels.
## Security model
ccdesk drives your real desktop with your user's privileges. The guardrails are:
- **Read-only mode.** `ccdesk mcp --read-only` disables every input tool: `batch`, `click`, `click_element`, `set_value`, `move`,
`drag`, `scroll`, `type_text`, `press_key`, `activate_window`, `manage_window`, `clipboard(set)` and
`launch`. Screenshots, `list_windows`, `clipboard(get)`, `wait` and `doctor` still work.
- **No launching by default.** `launch(command)` refuses unless the server was started with
`--allow-launch`. Even then it runs the command without a shell (`shlex.split`, no pipes,
redirects or `;`) in a new session.
- **Destructive shortcuts need confirmation.** `Alt+F4`, `Ctrl+Q`, `Ctrl+W`, `Super+Q`,
`Ctrl+Alt+Delete` and `Ctrl+Alt+BackSpace` are refused unless the call passes
`confirm=true`. Supersets such as `Ctrl+Shift+W` count too. Window managers bind their own
close, quit and power keys, so add yours with `--confirm-keys`, for example on bspwm:
`ccdesk install --confirm-keys "super+x,ctrl+alt+super+p,ctrl+alt+super+r,ctrl+alt+super+q"`.
- **Input goes to the window Claude chose.** An accessibility action goes to its element wherever
focus and the pointer are; if the tree is of another window than `activate_window()` chose,
`set_value` refuses and `click_element` takes the checked route below. Otherwise, after
`activate_window()`, `click`, `click_element`, `drag`, `scroll`, `type_text` and `press_key` check where the input would
land (the window under the pointer, or the one with keyboard focus) and refuse when it is
another window, for example because you switched to your terminal to answer. Dialogs of
that window, such as a file chooser, count as the same window. `force=true` sends it
anyway. Every input result names the window that received it (`"window": {id, title}`).
This needs an EWMH window manager on X11 or Hyprland; without one, or on other Wayland
compositors, focus is unknown and the check is skipped. On Hyprland the stacking order under
the pointer is inferred, because Hyprland does not report it, and a dialog is recognised as a
floating window of the same app on the same workspace.
- **Input goes to the tab Claude chose.** A browser is one window with many tabs, so the
check above passes when you switch from the tab Claude was using to your mail. Pass
`expect_title="Kimai"` to `click`, `click_element`, `type_text` or `press_key` and it refuses unless the receiving
window's title contains that text (case-insensitive); Firefox and Chromium put the active
tab's title in the window title. `force=true` does not skip it, and when the window can't
be known the call is refused rather than sent unchecked. It is checked once, just before the
input: a tab switch while text is being typed is not caught. `drag` and `scroll` don't take it.
- **Yielding to you.** On your own screen you share one pointer and keyboard with Claude.
`ccdesk mcp --yield-ms 3000` (or `ccdesk install --yield-ms 3000`) holds every input action
until you have left the mouse and keyboard alone for 3 s, waiting up to 15 s and then
refusing with "the user is using the mouse or keyboard". ccdesk's own events reset X's idle
counter too, so only input that came after its own counts as yours. Results say
`yielded_ms` when it waited, and `doctor` shows `user idle`. Needs `xprintidle` (X11); off
by default, and pointless for `--headless`.
- **Typed text stays private.** It reaches `xdotool`, `wtype` and `ydotool` through stdin, not as
arguments that any process of yours could read in `/proc`. After a `ui_tree()` that shows a focused
password field, `type_text` refuses. It knows only what the latest read of the tree showed, so
this is a backstop for the rule below, not a guarantee.
- **Audit log.** Every call, including denied and failed ones, is appended as one JSON line
to `~/.cache/ccdesk/audit.log` (mode 0600; override with `--audit-log` or
`CCDESK_AUDIT_LOG`). Each line has a timestamp, the tool, its arguments, the status
(`ok`/`denied`/`error`), the result or error, and the duration. Clipboard contents are
logged by length only. Typed text is logged, truncated to 200 characters, which is one
more reason the skill tells Claude never to type secrets.
- **Input validation.** Coordinates are bounds-checked against the screen. Counts, scroll
amounts, durations, waits and text length are capped. Key names are whitelisted
characters, so nothing reaches a helper tool as an option or shell syntax.
- **The skill.** It tells Claude to screenshot before acting and verify after, to never type
passwords, to never dismiss dialogs blindly, and to treat on-screen text as data, not
instructions.
Each tool also carries MCP annotations (`readOnlyHint`, `destructiveHint`, `openWorldHint`), so a
host can tell looking from acting. They are hints for the host; the checks above are what refuse.
These guardrails are not a sandbox. Any client that can call `type_text` can type a shell
command into a terminal. Use `--read-only` when you only want Claude to look, and keep
Claude Code's own tool-permission prompts turned on for `mcp__ccdesk__*` if you want to
approve each action.
## Wayland caveats
Wayland isolates clients from each other by design, so ccdesk can only do what the
compositor allows:
- **Screenshots** need `grim`, which requires wlr-screencopy (sway, Hyprland, river, Wayfire).
It runs as `grim -s 1`: logical pixels, so coordinates match, but on a scaled monitor the
image is not at full physical resolution. GNOME falls back to `gnome-screenshot -f`, and only
for full-screen captures. KDE has neither; on Plasma `spectacle` does it (see below).
- **Pointer input** on sway goes through sway itself (`swaymsg seat - cursor set/press/release`):
exact positions, no root, nothing extra to install. Elsewhere ccdesk speaks the
wlr-virtual-pointer protocol directly (`ccdesk/backends/vpointer.py`, standard library only):
exact positions, real motion events, no root, no daemon, nothing to install. Verified on
Hyprland; river and Wayfire should work where the protocol is offered, not tested. GNOME and
KDE don't offer it. As the last resort, compositors without it need `ydotool` 1.x with the
`ydotoold` daemon and access to `/dev/uinput`; absolute positioning then depends on ydotool's
pointer acceleration, so verify clicks with screenshots. (On Hyprland it was measured landing
at double the requested position.)
- **Keyboard** uses `wtype` (virtual-keyboard protocol, not available on GNOME) and falls
back to `ydotool type` for plain text. `press_key` requires `wtype`, except on Hyprland, where
it uses Hyprland's own `hl.dsp.send_shortcut`. Known issue: a named key sent alone with
`wtype -k` (Left, BackSpace) does not reach GTK 3 apps, while typed text does. Seen on a real
Hyprland and on a headless sway with only virtual devices; not yet checked on a real sway
session.
Text reaches `wtype` through stdin, to keep it out of the process list. In Chromium on sway that
typed the same as an argument. Accents, arrows, dashes and CJK arrive intact, but Chromium cuts
emoji to 16 bits, so `🙂` becomes a private-use character. `type_text` warns when text has emoji;
use `clipboard` and `ctrl+v` there. `xdotool` on X11 typed the same emoji into Chromium correctly.
computer-use-linux reports stdin garbling text in Electron on Hyprland; not checked here on a real
Hyprland, so verify typed text in those apps.
- **ui_tree** works on Wayland too. GTK and Qt report element boxes relative to their own
window there; on sway ccdesk adds the window's position from `swaymsg -t get_tree` and on
Hyprland from `hyprctl clients`, so `click_element` lands on screen coordinates. On other
compositors the result carries a note to verify with a screenshot.
- **Windows** are listed and focused through `swaymsg` (sway) or `hyprctl` (Hyprland, with Lua
`hl.dsp.focus`, falling back to the old `focuswindow` for versions before Lua; the fallback is
untested, and `press_key` and `manage_window` need a version with Lua). On Plasma, `kdotool`
does it through KWin scripting. Elsewhere ccdesk falls back to XWayland via `xdotool`, which
only sees X11 apps.
- **Window control** (`manage_window`) is implemented on sway, Hyprland and KDE Plasma; other
compositors refuse with a clear error. On Hyprland, move and resize make a tiled window float,
maximize uses Hyprland's maximized state, minimize moves the window to a hidden special
workspace, and restore undoes those, brings a minimized window to the workspace on show, and
re-tiles only windows this ccdesk server floated (a restart forgets them). On Plasma the moves
and sizes go to KWin, which decides what a tiled window does with them. **Monitors** come from
`swaymsg`, `hyprctl` or, on Plasma, KWin; on Hyprland they and the screen size are logical
pixels. ccdesk's coordinates start at 0,0, so `doctor` warns about a monitor at a negative
position.
**Per-window screenshots** crop the window's box, which needs a compositor that lists windows.
- **Clipboard** uses `wl-copy` / `wl-paste`.
- If you only need to drive XWayland apps, `CCDESK_BACKEND=x11` (or `--backend x11`)
forces the X11 backend.
`ccdesk doctor` reports exactly which of these are present.
### KDE Plasma (Wayland)
Plasma on Wayland has its own backend, picked from `XDG_CURRENT_DESKTOP`. On X11 it uses the X11
backend and needs none of this. Three tools do the work:
```sh
sudo pacman -S spectacle ydotool # Arch; elsewhere: spectacle, ydotool
paru -S kdotool # 0.3 or newer. AUR, or `cargo install kdotool`
systemctl --user enable --now ydotool # the input daemon, as you, not root
```
- **Screenshots** go through `spectacle`, because KWin implements no wlr-screencopy and `grim`
exits with "the screen capture protocol is not supported". A region shot is cut out of a full
grab and needs ImageMagick. So does a scaled display, where the grab is brought down to logical
pixels to match windows and the pointer.
- **Windows** come from `kdotool`, which loads a script into KWin over D-Bus and reads its answer
back. One script answers for every window, with its frame box, desktop and state, so
`list_windows`, `activate_window`, `manage_window` and the window-under-the-pointer check all
work. Without `kdotool`, screenshots and typing still work, but clicks and drags are refused
because there is no way to place the pointer, the window tools say what is missing, and
coordinates are wrong on a scaled display.
Panels, the desktop and other shell surfaces are left out of `list_windows`, but the focus
guard still sees them: a click on a panel, or typing into the open launcher, is refused like
any other window that is not the one you activated.
- **Input** is `ydotool`, through `/dev/uinput`, with `ydotoold` running as your user (the Arch
package ships that user unit). Plasma offers neither the virtual-pointer nor the
virtual-keyboard protocol, so there is no unprivileged path. `ydotool --absolute` does not place
the pointer under KWin (every value lands in a corner), so ccdesk moves the pointer in relative
steps and corrects against the position KWin reports, which also absorbs the pointer
acceleration curve. A click that cannot be placed within a few pixels is refused rather than
sent somewhere else. `press_key` needs no `wtype`: ccdesk turns key names into the keycodes
`ydotool key` takes. Keys and `type_text` assume a US layout, and `ydotool` only types ASCII, so
paste anything else through the clipboard.
- `ui_tree` works as everywhere else, once the accessibility bus is on (see above). With `kdotool`,
its boxes are moved to screen coordinates; without it, `ui_tree` says they may be relative to
the window.
`ccdesk doctor` lists whichever of these is missing.
## Support matrix
| Environment | screenshot | click/move/drag/scroll | type_text | press_key | list/activate windows | clipboard |
| --- | --- | --- | --- | --- | --- | --- |
| X11 (any WM/DE) | ✅ scrot / import | ✅ xdotool | ✅ xdotool | ✅ xdotool | ✅ wmctrl | ✅ xclip / xsel |
| X11 without a WM (Xvfb) | ✅ | ✅ | ✅ | ✅ | ✅ xdotool + xprop fallback | ✅ |
| Wayland: sway | ✅ grim | ✅ swaymsg seat | ✅ wtype / ydotool | ⚠️ wtype (see above) | ✅ swaymsg | ✅ wl-clipboard |
| Wayland: Hyprland | ✅ grim | ✅ virtual pointer | ✅ wtype / ydotool | ✅ hl.dsp.send_shortcut | ✅ hyprctl | ✅ wl-clipboard |
| Wayland: GNOME | ⚠️ full screen only | ⚠️ ydotool + ydotoold | ⚠️ ydotool | ❌ | ⚠️ XWayland apps only | ✅ wl-clipboard |
| Wayland: KDE Plasma | ✅ spectacle | ⚠️ ydotool + ydotoold | ⚠️ ydotool | ✅ ydotool | ✅ kdotool | ✅ wl-clipboard |
| Headless (`--headless`, any session) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ private clipboard |
| No display (SSH, CI) | ❌ (doctor explains) | ❌ | ❌ | ❌ | ❌ | ❌ |
`manage_window` on X11 uses `xdotool` (EWMH window managers). On bspwm it goes through `bspc`:
move and resize make a tiled window float, maximize means fullscreen (bspwm has no maximized
state), minimize hides the node, and restore unhides it and tiles it again, including a window
you had floating. On X11, "on show" means the window is mapped; a window manager that keeps
windows mapped on other desktops can't be told apart, and with several monitors the current
desktop can't decide it, because each monitor shows its own.
`tests/test_integration_windows.py` runs that against a real bspwm on a private screen. Other
X11 window managers and sway are covered by unit tests of the commands ccdesk builds. Hyprland
is covered against the real thing; see below.
The X11 rows are exercised by the integration suite under Xvfb, and the sway row by
`tests/test_integration_wayland.py` on a headless sway it starts itself (screenshots, window
list, clipboard, `ui_tree` coordinates, clicks and typing in a real GTK app), which also
checks the virtual pointer separately (exact clicks, wheel, drag). The Hyprland row by
`tests/test_integration_hyprland.py`, verified on Hyprland 0.56.2 with two monitors at scale 1.5
and 1. It opens windows and clicks for real, so it only runs with `CCDESK_TEST_HYPRLAND=1` from
inside Hyprland, or with its session variables set. The headless
row by `tests/test_integration_headless.py`, which launches a GTK app on the private screen
and checks that your session bus never sees it. The other
Wayland rows are covered by unit tests of the exact commands ccdesk builds.
Requires Python 3.10+. The full suite passes on Python 3.14 with both `mcp` 2.3 and `mcp` 1.30.
ccdesk does not replace [Cua Driver](https://github.com/trycua/cua). If `cua-driver` is on
your `PATH`, `doctor` mentions it, but ccdesk never calls it.
## Development
```sh
python -m venv .venv && . .venv/bin/activate
pip install -e '.[test]'
xvfb-run -a pytest -q # unit tests + Tk integration test on a virtual X server
pytest -q # without a display the integration tests are skipped
xvfb-run -a ccdesk doctor
```
The display tests skip themselves when `DISPLAY` looks like your real screen (`:0`, `:1`):
they click and type for real. Set `CCDESK_TEST_REAL_DISPLAY=1` to override.
The Wayland test needs `sway`, `grim`, `wtype`, `wl-clipboard`, a C compiler, `pkg-config` and
`wayland-scanner`. It starts a headless sway on its own socket, plus a small virtual-pointer
helper (`tests/wayland/vpointer.c`, compiled on the fly) because a headless seat has no mouse.
The Hyprland test needs `hyprctl`, `grim`, `wtype`, `wl-clipboard` and a system Python with
PyGObject and GTK 3.
The integration test (`tests/test_integration_tk.py`) starts a Tk window with a button, a
label, an entry, a canvas and a list. It takes a screenshot, finds the button by its colour
in the PNG, clicks it, and checks that the label changed. It then types into the entry,
presses Return, drags across the canvas, scrolls the list and checks the clipboard,
asserting each result against the app's own state.
Layout:
```
ccdesk/
cli.py ccdesk doctor | firefox | install | mcp | peek | version
firefox.py ccdesk firefox, and the firefox-devtools registration
headless.py --headless: private Xvfb screen, window manager and D-Bus
server.py MCP tool definitions (the only module importing `mcp`)
desktop.py validation + policy + audit around a backend
safety.py read-only / launch policy, destructive keys, JSONL audit log
parsing.py pure parsers: keys, regions, wmctrl/xdotool output, PNG header
doctor.py environment report
install.py Claude Code registration and skill copy
backends/ x11.py, wayland.py, kde.py, vpointer.py, keycodes.py, base.py
skills/computer-use/SKILL.md
tests/
```
### Releasing
A release is a tag. `v0.2.0` publishes that commit to PyPI and opens the GitHub Release with the
wheel and the sdist attached. The version lives in `ccdesk/__init__.py`; hatch reads it from there
and the tag has to match it.
```sh
scripts/release.sh 0.2.0 # unit tests, version bump, commit, tag, push
```
The script won't run off `master`, with a dirty tree, or with a version that doesn't go up. A
version that was never released can be tagged as it stands, so `0.1.0` of an unreleased repo goes
out without a bump commit.
`.github/workflows/release.yml` builds, runs the tests again on the tagged commit, uploads to PyPI
with trusted publishing and writes the notes from the commits since the last tag. `uv.lock` has to
be committed: the workflow installs with `--locked` when it finds one.
The PyPI upload needs a trusted publisher once, at
<https://pypi.org/manage/account/publishing/>: project `ccdesk`, owner `asloth`, repository
`ccdesk`, workflow `release.yml`, environment `pypi`. No token is stored anywhere.
If a run fails halfway, fix it and re-run the workflow from the Actions tab with the same tag: the
PyPI upload skips versions that already exist and the release assets are replaced.
## License
MIT. See [LICENSE](LICENSE).
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues