kde-desktop-mcp

A screenshot-free desktop control MCP server for KDE Plasma (Wayland), Hyprland and Windows. It lets MCP clients such as Claude Code read the inside of applications through the accessibility tree and act on elements directly. Written in Rust; a single executable.

Verified on a real KDE Plasma desktop. Hyprland was verified in a nested session inside KDE. On Windows, keyboard, mouse and screenshots were tried under Wine; tree reading (tree, find, get_text, focus, click) has not yet been tried on real Windows. Please share results from tools/mcp_smoke_test.py.

How it works

Instead of looking at the screen, Claude reads the elements of a window as a list like [55] button "Save" and acts on an element by its number.

The code lives in rust/src/: main.rs holds the MCP protocol, the allowlist, element ids and the tools; render.rs the tree and search output; backend.rs defines the platform interface. Platform code is in linux/ (KDE and Hyprland share an AT-SPI layer) and windows/. Claude sees the same tools on every platform.

The desktop is chosen automatically: Windows on Windows; on Linux, Hyprland if HYPRLAND_INSTANCE_SIGNATURE is set, otherwise KDE. To choose by hand on Linux, set DESKTOP_MCP_BACKEND to kde or hyprland.

On Hyprland the active window is matched to its AT-SPI application by process id (pid), so the toolkit's focus flag does not have to be trusted.

focus first asks the accessibility layer for focus. If the app refuses (common with Qt), it clicks the centre of the element for real.

Why it is fast:

• It talks to AT-SPI over D-Bus directly, without libatspi, and pipelines requests: all attributes of an element, and every level of the tree, are requested at once, so the time is bound by the app's response speed rather than by round-trip latency.

• On Windows a window's whole subtree, with the properties needed, arrives in a single UI Automation cache request (BuildUpdatedCache).

• KWin scripts go over D-Bus instead of qdbus6 processes, Hyprland commands over Hyprland's socket instead of hyprctl processes.

• Start-up takes 2 ms; the virtual keyboard and mouse are prepared in parallel at start-up.

• On KDE the waits were measured and shortened (uinput events are processed in order); focus remembers focus methods that do not work in an app and skips them; type_text starts no process; screenshots come straight from KWin instead of spectacle.

• Output is short, so Claude reads less: one space of indent per level, sibling leaves with the same role and actions share one line (list item ×3: [8] "a" [9] "b" [10] "c"), and the "Set Focus" action Qt gives almost every element is not listed. A Dolphin + Kate tree went from 15,182 to 7,092 characters.

• In browsers too: Firefox's "clickAncestor" action on every element and the embedded-object placeholder (U+FFFC) of web pages are not listed; real action names are read instead of Firefox's broken batched names (;;). tree(document=true) shows only the active tab's page (found in one call through the window's EMBEDS relation). The same Firefox page went from 5,943 to 1,421 characters.

• In Chromium and Electron apps, the showContextMenu action every element carries is not listed either, and the generic doDefault of their frame, panels and pages counts as implied: it no longer keeps nameless panels as lines. A Ferdium (WhatsApp) tree went from 19,906 characters, cut at 300 nodes behind six nested panels, to 7,759 for the whole window; with interactive: true from 19,924 to 2,586, the chat list's rows included (see below). act without action never picks showContextMenu: an element with no other action is an error that says to use click, or action: "showContextMenu" when the menu is what is wanted.

• Actions a role implies are left out (a link's jump, a button's press, a tab's switch, a web section's click); a button or link without any says actions=none. Nameless, text-free wrappers (landmarks, forms, scroll panes) are left out with their children lifted, as are empty nameless leaves (separators, images inside buttons) and labels around a same-named button. A Google results page in Firefox went from 6,976 to 5,702 characters. A static text or label leaf that only repeats what its parent's line shows (a heading's or link's own text node, a status bar's count) is left out of tree, and of find when the parent is listed as high; asked for with role, it is listed. A YouTube page's tree(document=true) went from 3,792 to 3,327 characters. get_text in Chromium no longer reads such leaves a second time after the text of the element that holds them (a WhatsApp page: 3,035 instead of 5,010 characters, the same text), and the U+FFFC in a container's text goes to the child object it stands for (an avatar, an emoji image) rather than to a text leaf whose words the text already holds.

• On Linux, a find, tree of an app, or element picked by app + query reuses the app's tree read by an earlier call while the app provably did not change: it sent no AT-SPI change event since (children, name, text, any state, visible data, windows), no app or window came or went, no tool sent input or activated a window, the user did not touch the keyboard or mouse, and the read is under 2 s old. Only apps known to send these events (Qt, Firefox, Chromium, or any app once it sent one) and no page still loading. An element taken from such a read is checked to still be there (one call) before it is used. A get_text by query in Firefox went from 8.7 ms to 1.1 ms. Clicking by query also measures the element in the same round trip as the lookup, from the window found in the read.

• apps lists blocked apps by name on one line and skips untitled windows.

• An AT-SPI call waits at most 2 s; an app that does not answer cannot lock the server. Actions that open a menu (Qt does not reply until the menu closes) count as sent after 400 ms.

• Tool calls are handled concurrently: a pending wait_for or a slow app does not hold up ping and other calls. If Claude cancels a call (notifications/cancelled), it is stopped. Tools that send keyboard or pointer input still run one at a time, so keystrokes never interleave.

• Popup menus and overlays of web pages are part of tree/find output. Firefox does not mark these layers as SHOWING; containers that are visible and lie inside the window are entered as well, and so are elements without a layout box (display: contents, -1 extents): YouTube wraps its whole page in one, which hid every element of the page from find and query. Content scrolled out of view is skipped. Example: Gemini's upload menu ("Dosya yükleyin") is now found without a screenshot.

• Matching (find, query, wait_for) compares names and text case-folded (Turkish İ/ı as i), with whitespace runs, no-break spaces and the placeholders of web containers as one space and soft hyphens and zero-width characters left out; up to 1000 characters of an element's text are searched. When nothing matches that way, an element whose name or text holds every word of the query, in any order and accents aside, does ("cozum yontemleri" for "Çözüm Yöntemleri"). When still nothing visible matches, find (and the error of a tool given query) lists up to five matches scrolled out of view in the web page, with ids (act and get_text work on them as they are; act with action: "scroll_to" brings one into view for click), else up to five visible elements with the closest names (no matches; closest: …), so the next call can use an id instead of a screenshot.

• Read tools (apps, tree, find, get_text, wait_for, wait_idle, screenshot, pause, clipboard_get, windows, doctor) carry MCP's readOnlyHint; Claude Code runs several of them in parallel when asked for in one message. The others carry destructiveHint: true for close_window and the input tools (which can do whatever the user could), false for activate_window, move_window and launch.

Related MCP server: blade-computer-use

On-screen indicator

While the agent uses the keyboard or mouse, the screen edges glow orange, a label at the top says what it is doing (e.g. "Claude · typing") and a ring marks each pointer target. All of them are layers that pass clicks through to the window below (wlr-layer-shell; KDE Plasma and Hyprland). They disappear 2.5 s after the last action and are hidden before a screenshot is taken. The server starts the indicator on the first action as desktop-mcp --indicator; it exits with the server. To turn it off, set the environment variable DESKTOP_MCP_INDICATOR=0 in the MCP configuration.

When the user takes over

The agent and the user share one keyboard and mouse. Input tools (act, set_text, focus, click, drag, scroll, key, type_text, mouse, activate_window, file_dialog, move_window, launch, and any tool given window) wait for the user:

• Pause. While the user typed or moved the pointer in the last 1.5 s, they fail with paused: the user is using the keyboard ... without sending anything; the agent retries once the user stops. User input during a batch, a drag or a repeated key stops the remaining input (interrupted: ...). Read tools and the clipboard tools keep working.

• Stop shortcut: Meta+Alt+Esc. It ends a running drag or batch and refuses every input tool with stopped by the user until the agent calls activate_window on its own (not as a batch step and not through window); the agent is told to ask you first. The indicator shows "paused" or "stopped" when it refuses. (Plasma 6 already uses Meta+Esc for System Monitor.)

How it is seen (doctor shows it under "takeover detection"):

• With read access to /dev/input (the user is in the input group), the server reads the real keyboards and pointers there: read-only, never grabbed, its own desktop-mcp-* devices left out. This is exact, and the stop chord is seen on every compositor. Note that the input group lets every program of the user read all keystrokes.

• Otherwise it follows the compositor's idle notifications (ext-idle-notify-v1, KDE and Hyprland). They count the agent's own input too, so user input is only noticed outside of it (between steps, or when it lasts longer than the agent's).

• KDE registers the stop shortcut with KGlobalAccel: it appears under System Settings > Shortcuts > Desktop MCP, where it can be changed. On Hyprland the stop shortcut needs /dev/input.

Settings (environment variables in the MCP configuration): DESKTOP_MCP_TAKEOVER_MS (default 1500; 0 turns the pause off) and DESKTOP_MCP_STOP_KEY (default meta+alt+esc, key names as for the key tool; off turns it off). Not available on Windows yet.

Tools

• apps, tree, find, get_text: reading and searching. With document: true, tree lists only the document of the active window (in a browser, that tab's page without tabs and toolbars). get_text on a web page or another container returns the text of everything in it in reading order (web containers' own text is only a placeholder per child element). It returns at most limit characters (default 10,000, about 2,500 tokens) from offset (default 0); a cut reply ends with … cut at character N; get_text with offset=N reads on. Whole web pages used to come back as 20,000 characters while the part wanted was near the top. A tree/find line shows up to 80 characters of an element's text with its line breaks (text="Shopping list\nmilk\nbread\n"), so a short document's lines and final newline are visible without get_text; … after the closing quote marks a text that goes on.

• Large trees: tree(interactive: true) lists only elements one can operate (actions, editable/checkable/selectable/expandable, and on Linux focusable named table rows, cells, list and tree items: a web app's chat list whose rows say only clickAncestor/showContextMenu) under their nearest named ancestor; on a GitHub issues page in Firefox it was 15,339 instead of 24,687 characters and still listed all 259 links, buttons and fields. A reply is capped at max_chars (default 20,000) and max_nodes; a cut one ends with … N more nodes; continue with tree(…, after=ID), which lists the rest after that element (ids are stable, so pages join up exactly). screenshot: true adds the image of the app's window when it is in front.

• set_text reads back what it wrote: the reply says "verified", or it fails saying what the field holds (a length limit, reformatting). Firefox's input fields report accessibility writes as successful and do nothing, so when single-line text does not take, it focuses the element, selects all with Ctrl+A and types it. An editable element without a text interface (Qt 6's editable combo boxes, such as a KDE file dialog's name field) is typed into the same way and read back from its name. Password fields cannot be read back.

• scroll: turns the wheel over an element (a list, a page, a panel).

• Verified actions of act (one tool rather than new ones: act is loaded up front, so they cost no tool-search turn, and the schema grew by about 570 characters instead of three new tools of 1,700 each). Each reads the state back and replies with it (Toggle -> checked (verified)), or fails saying what it sent and what it reads instead (sent Toggle, but it reads unchecked afterwards); one already in the asked state sends nothing and says so (already checked; nothing sent), so it is safe to repeat:

  • action: "check" / "uncheck": a check box, check menu item, radio button (only checked: another one of its group unchecks it), toggle button or switch, or anything the platform marks checkable. Runs its check, uncheck, toggle, press or click action. A checkable list or tree row (Qt's item views, GTK's toggle cells) only through an action named check/uncheck, or toggle outside Qt (Qt's Toggle selects the row); otherwise focus through accessibility and Space (full tier). "toggle" flips it, or the expansion of an element that is expandable, not checkable, and has no action named Toggle (one that has it, a Qt row, runs that action as before).

  • action: "expand" / "collapse": a tree item, disclosure button, accordion or combo box through an action named for it (expand, collapse, GTK's expand or contract); buttons and combo boxes may use their press. A list or tree row is never pressed for it (that opens a file in Dolphin): an element with no such action (Qt's tree items) gets focused through accessibility and sent Right or Left, which needs the full tier; when it cannot be focused that way the reply says to click it and press the key.

  • option: "UTF-8" (with action: "select", or alone): picks the entry with that label (matched like menu labels: case, & markers, ellipses aside; a unique part is enough) in a combo box, list, tab list or menu button, from its entries in the tree even while it is closed. In order, each only while the one before is not verified: the list's AT-SPI Selection (SelectChild, after ClearSelection in a multi-selection list; skipped in Firefox and Chromium pages, whose scripts would not see it), the entry's own select/click/press action (not Qt's Toggle, which only marks a row), then for a closed combo box (with the full tier) focus through accessibility and as many Down/Up presses as entries lie between the shown one and the wanted one. Verified by the value a combo box is named after (Qt, GTK), else the entry's selected state (Firefox); a list where other entries stay selected too says how many. A control that lists no entries is opened through its action only when it opens a popup (a combo box, a menu, a button marked as having a popup or with a Show Menu/open/expand action) and the entry is taken from the popup menu; anything else (a plain button, a row) fails with nothing sent.

  • action: "scroll_to": brings an element scrolled out of view (an id from the off-screen list of a miss) into view through AT-SPI Component.ScrollTo (on the element, else its nearest ancestors); when that is missing or does not show it, turns the wheel over its scrollable container (5 notches a turn, at most 10, toward the element) until it is showing, and says that the pointer moved. The wheel turns over the part of the container inside its window (in a web page over the page, not a <ul> or <table> in it), and stops when the user takes over. An element that is hidden (not visible: in a closed popup or a collapsed branch; a Qt row scrolled out of a showing list counts as scrolled away) gets no wheel.

  None of them opens a Firefox menu or <select> popup through accessibility (see menu): such a call fails with nothing sent and says to click the control. Their key fallbacks stop when the user takes over, and the act call's confirmation covers them. On Windows (roles compared without spaces: checkbox, listitem), check/uncheck/expand/collapse use the UI Automation Toggle and ExpandCollapse actions and select the entry's SelectionItem; Selection and ScrollTo are not supported there (untested on Windows). Verified so far with unit tests only; not yet tried against Kate, Dolphin or Firefox.

• Modifier clicks: click, mouse and scroll take modifiers (ctrl, shift, alt, meta), pressed before the pointer action and released in reverse order afterwards, also when it fails (ctrl+click adds to a selection, shift+click selects a range, ctrl+scroll zooms). clicks: 2 is a double and clicks: 3 a triple click (select a line); button is left, right or middle.

• hold: presses a key, chord or mouse button and leaves it down across calls (down: false releases it; without key/button it releases everything). Every input reply ends with what is still held (; held: shift, left). Everything held is released when a call fails, when a call is cancelled (notifications/cancelled) and when the server exits (end of input, SIGTERM/SIGINT/SIGHUP); on Linux the kernel also releases the keys of the virtual devices when the process dies. Meant for modifiers and buttons: a held letter auto-repeats like a real key.

• cursor_position: where the cursor is, in the coordinates mouse takes (KDE: workspace.cursorPos through a KWin script; Hyprland: cursorpos; Windows: GetCursorPos). Read-only.

• drag: presses, moves in steps and releases (drag and drop, sliders, selections). Each end is an element id (from_id/to_id), a screen coordinate, or with image: true a pixel of the last screenshot.

• act, set_text, focus, click: act on an element. The element is picked by its tree/find number (id) or in one call by app + query (+ role): a visible element whose name equals query, else one whose name or text contains it. Window, tab and page titles (frame, window, page tab, and a web page's document) count only when nothing else matches or role names that role; wait_for skips them unless role names them. Last come elements holding every word of the query, accents aside.

• Ties: when several elements match equally well (the best of those tiers), these tie-breaks apply in turn, while the matches are in several windows of the app: (1) a window whose title contains the call's window argument beats the others (before the active window, whose ACTIVE state can lag just after window raised another); (2) the app's active window beats its background windows; (3) a dialog (dialog, alert, file chooser) beats the app's other windows, and of stacked dialogs the one opened last (listed last) wins. Then, wherever they are: (4) an element one operates (a button, entry, list item: an interactive role or state) beats one that only shows the name (a label, a heading, a list named the same); (5) for input tools, an element with an on-screen area beats one without (zero size), whether its window is in front or not (checked for up to 12 candidates). A tie left after that is refused by the input tools (act, click, focus, set_text, scroll), which send nothing and list at most 6 candidates with ids, so the next call passes one id (or role, or window):

  ambiguous: 2 elements in 'firefox' match 'Kapat'; nothing sent. pass one id:
  [3] push button "Kapat" at 3440,0
  [4] push button "Kapat" at 3440,22

  The position is shown when the window is in front; the window title is added when the candidates are in several windows. Read tools take the first: get_text notes it ([5] … (first of 2 matches):), wait_for matches on it, find lists them all. Matches are counted over one read of the app, made afresh for a refusal.

• Effect and observe on input tools (act, menu, close_window, click, key, type_text, mouse, drag, scroll, set_text): with observe (and by default for act, menu and close_window) the reply ends with an effect: line comparing the active window before and after the input (its title, the focused element, and on Linux the app's accessibility change events): changed, no visible change, or window changed to '<title>' (a dialog opened, a window closed). It waits for the app to react up to settle_ms (default 150), returning as soon as it reported a change; the fingerprint itself costs about 1 ms. A canvas that draws without accessibility events may report no visible change. observe adds a view in the same reply: focus (the focused element), tree (compact tree of the active window, or of app's, at most 6000 characters) or screenshot (inline image). Without observe, the other input tools still note, without waiting, when another window is in front after the input (; active window now '<title>') or came to the front since the previous input (; before it, '<title>' had come to the front: a dialog that opened after that reply); about 3 ms per input.

• What an input opened (Linux): after act, menu, close_window and the input tools given observe, and after click, key, mouse and type_text without it, the reply lists the menus, popup menus, combo box lists, dialogs and alerts of allowed apps that opened, one line each, with their entries' ids, so the next call acts on them directly:

  opened: popup menu (8 entries): [51] "Kopyala" [52] "Yapıştır" [53] check menu item "Satır Sarma" {checked} …
  opened: dialog 'Değişiklikleri kaydet?': "Belge değiştirildi." [60] push button "Kaydet" [61] "Kaydetme" [62] "İptal"

  A role is named where it changes (a menu's menu items and a list's list items go without); a dialog's buttons come first, after its first label (the question it asks), those of its last button group (OK, Save, Cancel) before the others, then its other controls; its lists, tables, trees, documents and tool bars are not read. A KDE file dialog that file_dialog fills is named as opened: file dialog 'Open File' (use file_dialog), without entries. At most 12 entries and about 600 characters a line (the rest is counted: … +5 more), 3 surfaces a reply; a surface listed once in a call (by close_window) is not read or listed again in its effect line. What showed after a reply (a dialog a button opened a few ms after the click) is listed by the next input's reply as before it, opened: …; what one reply listed is not again. A surface is listed only when its app's name reads as an allowed app's. The AT-SPI events object:state-changed:showing and window:create say what showed since the input: when none came, this costs nothing (no round trip); otherwise the role of each shown element is read (one round trip for all of them), and only menus, dialogs and lists under a combo box are read further. A right click (click, mouse), a click on a combo box or menu entry, and keys that usually open a menu or dialog (menu, shift+f10, f10, alt+a letter, alt+down, alt+f4, ctrl+o/s/p/w/q, ctrl+shift+s) wait up to 150 ms for something to show, ending as soon as it did; other inputs only list what showed already. When registering those events failed, those likely openers and act, menu and close_window look at the active app's top-level windows instead, and a surface one such look listed is not read or listed again while it stays open. Not on Windows: UI Automation's roles do not name menus and dialogs this way (a dialog is a window), so nothing is looked at or waited for there. Only menus already open are read: nothing here opens a menu through accessibility (Firefox can crash when one does).

• Failures say what is on screen: when find finds nothing, or act, click, focus, get_text, set_text, scroll, drag, wait_for, wait_idle, menu, key, type_text or mouse fails (or a batch step fails and the batch does not end with screenshot or tree), the reply ends with now: active window '<title>' [app]; focus: <element>, so the next call needs no screenshot to see why. Titles in such notes (now:, activated '…', active window now '…') are cut at 80 characters like element names, with the … after the closing quote so the quoted part still works as window, and a focus line leaves out the focused state. Not for refusals by the allowlist, confirmations, an ambiguous target or the user taking over. A miss by query also lists off-screen matches or the closest names with ids (see find).

• Stale ids: every use of an id reads the element's role and name again (one round trip). If the element is gone, or its role or name changed (the page was redrawn, the object path now belongs to another element), nothing is done and the reply is stale id N (was <role> "<name>", now ...); call tree/find again. For elements whose name follows their content or title (windows, tabs, documents, labels, status bars, text fields, combo boxes, editable elements) and for unnamed elements, only the role is compared.

• wait_for: waits until an element appears or disappears (checks every 50 ms, at most 10 minutes); for dialogs opened by a key or click, a long load or a build. A longer timeout is cut to 10 minutes and the reply says so. state (enabled, focused, checked, selected) waits until the element has that state (with gone, until it no longer does). window instead of query waits for a window of an allowed app whose title or app name contains it to open, or with gone to close (KDE and Hyprland window lists, re-read when AT-SPI reports a window change).

• wait_idle: waits until an app's accessibility tree has not changed for quiet_ms (default 300) and no browser page is still marked busy (loading from the network): after opening a URL or a link, use it instead of a fixed pause. The reply names the app's window title (a browser's page title); a timeout says when a page was still loading.

• pause: waits a fixed time (ms, at most 10 minutes); for waiting on time only, not on a visible change.

• windows: lists windows of allowed apps with position, size, state (active, maximized, minimized), output and virtual desktop in screen coordinates, after the screen layout (name, area, scale) and the virtual desktops (KDE: number, name, the current one; Hyprland: the regular workspaces and the monitor each is on) (KDE and Hyprland).

• move_window: moves, resizes, maximizes, minimizes or restores a window, and with desktop (a number or name as windows lists them) moves it to that virtual desktop (KDE: the window is then on that one only) or workspace (Hyprland: silently, the view stays; a new number makes the workspace); waits (at most 1 s) until the window has its new geometry and reports the real result (KDE and Hyprland; see the Hyprland limitations).

• close_window: closes the one window whose title or app name contains query (several matches: an error listing them) as its close button does (KWin closeWindow(), Hyprland closewindow), so an app with unsaved changes still asks; the reply then says the window is still open and lists the app's dialog (the save prompt) with its buttons' ids (showing dialog '…': [60] push button "Save" [61] "Discard" …). Always needs the user's consent, as tools on the confirm: list do (see Tiers, confirmation and password fields).

• menu: runs a menu bar entry by its labels in one call: path is a list (["Dosya", "Farklı Kaydet…"]) or one string ("Dosya > Farklı Kaydet"). Labels are matched case-insensitively (Turkish dotted and dotless i alike), without &/_ accelerator markers, a shortcut after a tab or a trailing ellipsis; an exact label wins, else a unique part of one. A label that is not there is also tried against KDE's English action names (file, file_save_as, edit_copy), so "File > Save As" works in a Turkish Kate (it saved two turns per task in the e2e benchmark). Without path it lists the top level, and a path ending at a submenu lists that submenu's entries with ids (read-only where the app keeps closed submenus in the tree). Qt and GTK apps keep closed submenus' entries in the tree, so nothing is opened. KDE apps that replace the menu bar with a hamburger button ("Menüyü Aç", which has no accessibility action and no children) keep the hidden menu bar in the tree, and its entries work: menu finds it too (Kate, Dolphin). Firefox fills a menu only while it is open; menu does not open Firefox menus itself (doing so through accessibility hung or crashed a fresh Firefox in tests), so open the menu with key first.

• launch: starts an allowed app (app, e.g. kate) with an optional args list (no shell; its .desktop file first, else the program on PATH) and waits until its window opens (wait_ms, default 15 s, at most 60 s; 0 does not wait). Prints the window as windows does; also finds launches handed to an already running instance (firefox, kate). Window tracking works on KDE and Hyprland.

• Terminal windows: windows also lists each terminal tab's pts device, foreground command and directory; activate_window with terminal: "pytest" brings forward the terminal running that command (or in that directory). A terminal with several tabs comes forward on whichever tab it shows. The terminal must be in the allowlist (e.g. konsole).

• file_dialog: fills in the file Open/Save dialog in front: path (absolute or ~/…) is written into its file name field, the default button (Open, Save) is pressed and the dialog's closing is awaited (Qt 5's name field and Qt 6's editable combo box alike). Works on an allowed app's own KDE dialog and on the portal dialog (xdg-desktop-portal-kde) a browser opens for uploads and downloads; a portal dialog is filled only when the parent window KWin reports for it belongs to an allowed app, and a dialog without a reported parent is refused. The portal is not added to the allowlist; no other element of the dialog is reachable. For action: "open" the file must exist, for "save" its folder (KDE only).

• doctor: checks the setup in one report: version, allowlist, which running apps are allowed, the session, AT-SPI, the virtual keyboard and mouse, KWin scripting, screens and the screenshot path.

• key, type_text: keyboard. The reply names the element that has focus afterwards; type_text warns when focus is not on an editable field (KDE). In a batch, a step whose focus is the one the previous report described says focus: [id] (unchanged) instead of describing it again. A menu or dialog the key opened is listed with ids (see "What an input opened").

• activate_window: brings a window to the front; fails when no window of an allowed app matches.

• window on input tools (key, type_text, click, mouse, scroll, focus, drag, set_text, act, screenshot): a title or app-name substring, matched as by activate_window. When the window in front is not a matching window of an allowed app, that window is activated first (the reply starts with activated '<title>'; ), in the same critical section as the input, so no other call takes the focus in between; when it already is, nothing extra happens. Without window nothing is activated.

• clipboard_get, clipboard_set: read or write the clipboard as text (Linux only). Because the clipboard can hold anything the user copied anywhere (passwords included), both work only while a window of an allowed app is active. clipboard_get returns at most limit characters (default 20,000; the reply says when it was cut); if the clipboard holds an image or files rather than text, it only names the types, and it does not read content a password manager marked as a password (x-kde-passwordManagerHint).

• mouse, screenshot: fallback only. screenshot returns the image inside the reply, shrinks it so its longer side is 1024 px by default (max_size) and reports which screen area it shows. The inline image is a JPEG (quality 80, full colour resolution, so small text stays sharp) when that takes at most three quarters of the PNG's bytes, else a PNG: a 1024x576 YouTube window took 132 KB instead of 670 KB, a Confluence page 94 KB instead of 246 KB; flat windows stay PNG. Image tokens depend on the size only (about width x height / 750: 786 at 1024x576), so the default size stays: at 896 px a 1080p window's 13 px text shrinks to 6 px. An inline screenshot leaves no file behind and names none; inline: false saves a PNG and returns only its path; given image_x/image_y, mouse converts a pixel of the image to a screen coordinate (window position and display scaling, e.g. 125 %, included). Zoom: region: [x, y, w, h] in pixels of the last screenshot captures again and returns only that part, cropped before shrinking, at up to full resolution (max_size still applies); image_x/image_y then refer to the zoomed image.

• batch: runs several tools in order in one call and stops at the first error; reduces Claude's number of turns. A step with "optional": true (typically a wait_for) may fail without stopping the batch: it is reported as [n] tool failed (optional): … and the remaining steps still run, so a final observation step (find, screenshot) is not lost. A step's id/*_id argument "$N" is the element step N produced (find's first hit, wait_for's match, the target of a tool given one). A small procedure fits in one batch, with plain JSON control flow:

  • "if": {"step": N, TEST} on a step runs it only when the test holds on earlier step N; otherwise it is reported as [n] skipped (if step N …). TEST is one of "ok": bool (the step succeeded), "found": bool (it succeeded and produced an element; find had a match) or "contains": "text" (its reply or error, case-insensitive). A skipped step meets no condition.

  • "on_fail": [steps] on a step runs those steps (reported as [n.on_fail] …) only when the step failed; the batch then goes on. Not after a refusal by the allowlist (or a tier, or a password field), a confirmation the user declined or was not asked for, or the user taking over: then [n.on_fail] not run (refused). Such a refusal also ends a repeat, even in an optional step.

  • {"repeat": {"steps": [...], "until": TEST, "max": 10}} runs its steps (reported as [n.k] …, last run only) until TEST holds on the last of them, checked after each run; it fails after max runs (1 to 20, default

    10. or once it ran 10 minutes, and when a step in it fails that is not optional. "$N" and conditions on a repeat mean its last step's result in the latest run. Its input steps without observe (and any such step a condition tests) wait for the app to react, up to 150 ms (ending at its first change events), before their reply is read: the focus a key reports is the one it moved to.

  • {"stop_if": {"step": N, TEST}} ends the batch there, successfully. Steps are numbered at the top level only; steps inside repeat and on_fail take only tool, args and optional (no nesting). The whole batch is checked before anything runs (unknown tools or keys, conditions and "$N" that do not name an earlier step or can never hold, max out of range): a mistake fails it with not run (nothing sent): [n] …. Its input lock, the takeover check, the allowlist and observe work per step as in a plain batch. Example, scroll Dolphin's view of a folder "Reports" (the view is named after the folder) until a file shows, then click it, once the view stopped moving: [{"repeat": {"steps": [{"tool": "scroll", "args": {"app": "dolphin", "query": "Reports", "role": "list"}}, {"tool": "wait_idle", "args": {"app": "dolphin", "quiet_ms": 150}}, {"tool": "find", "args": {"app": "dolphin", "query": "report.pdf"}}], "until": {"found": true}}}, {"tool": "click", "args": {"id": "$1"}}].

The initialize reply carries instructions (about 3 KB): the workflow, the targeting rules shared by the tools above, the allowlist and which tools to load together. Clients that load tool schemas on demand, such as Claude Code, show it at session start before any schema is loaded; the per-tool descriptions only point to it.

Allowlist

Re-read on every call; a default list is created on first run.

• KDE and Hyprland: ~/.config/desktop-mcp/allow.txt, one AT-SPI application name per line (kate, firefox…). Window tools (windows, move_window, activate_window) match a window by the AT-SPI application name of its process (pid), so apps whose window class differs are found too (Ferdium: electron). Apps without accessibility match by window class (like xterm).

• Windows: %APPDATA%\desktop-mcp\allow.txt, one program name per line without .exe (notepad, firefox, chrome…).

Rules:

• Elements of apps not in the list cannot be read or acted on.

• Keyboard and mouse input is sent only when the window in front belongs to an app in the list.

• activate_window only brings windows of listed apps to the front.

• launch refuses an app not in the list before starting anything.

• file_dialog fills in the dialog of the (unlisted) portal only when it was opened for a listed app; a dialog opened by another app is refused.

Use the apps tool to see the right names.

Tiers, confirmation and password fields

An app can be limited to a tier with app:tier; a plain name means full, so files without tiers keep working unchanged:

firefox                 # full: everything
konsole:view            # read tools only
dolphin:click           # no typing
confirm: close_window, key:konsole
passwords: keepassxc

The tier is checked against the app a call reaches: the target element's app for element tools, the active window's app for keyboard and pointer input (both for click, scroll and focus). An app listed twice gets the lower tier; an unknown tier counts as view. A refusal names the tier.

confirm: lists tools (tool) or tools in one app (tool:app) that need the user's consent per call. When the client supports MCP elicitation the server asks the user through it; otherwise the call is refused unless it carries confirm: true (the agent is told to ask the user first).

Password fields (AT-SPI role password text, UIA IsPassword) never show their text in tree, find or wait_for, and get_text refuses them. set_text, and type_text while one has focus, refuse them too unless the app is on a passwords: line (passwords: all for every app) or the server runs with DESKTOP_MCP_ALLOW_PASSWORDS=1. doctor shows the tiers, the confirm list and the password setting.

Installation

Every platform needs Rust. After installing, open a new Claude Code session and check with /mcp that the desktop server is connected.

KDE Plasma (Wayland)

Requirements:

• KDE Plasma 6, Wayland session

• System packages (Arch): at-spi2-core, libkscreen, wl-clipboard, spectacle

• KWin 6.8 or later for fast screenshots. On older KWin, screenshot falls back to spectacle; enabling KWin's ScreenShot2 interface takes installing a .desktop file by hand once (details in the SHOT_NOT_AUTHORIZED comment in rust/src/linux/kwin.rs)

• Write access for the user on /dev/uinput (getfacl /dev/uinput)

git clone https://github.com/ahmetadakli/kde-desktop-mcp.git
cd kde-desktop-mcp/rust
cargo build --release
claude mcp add --scope user desktop -- "$PWD/target/release/desktop-mcp"

Hyprland

Requirements:

• A Hyprland session (the movecursor dispatcher)

• System packages (Arch): at-spi2-core, wl-clipboard; recommended: wtype (typing without touching the clipboard), grim (fallback screenshots)

• A working AT-SPI bus in the session. Qt apps may need env = QT_LINUX_ACCESSIBILITY_ALWAYS_ON,1 in the Hyprland configuration.

• Write access for the user on /dev/uinput

The installation commands are the same as for KDE. Start Claude Code from inside the Hyprland session so HYPRLAND_INSTANCE_SIGNATURE reaches the server.

Windows

Requirements: Windows 10 or 11. In PowerShell:

git clone https://github.com/ahmetadakli/kde-desktop-mcp.git
cd kde-desktop-mcp\rust
cargo build --release
claude mcp add --scope user desktop -- "$PWD\target\release\desktop-mcp.exe"

Building for Windows on Linux also works (mingw-w64-gcc and rustup target add x86_64-pc-windows-gnu, then cargo build --release --target x86_64-pc-windows-gnu).

Speed: tools loaded up front

Claude Code defers MCP tools behind a tool search, which costs a model turn at the start of every desktop task. The server marks the ten tools nearly every task starts with (batch, find, act, activate_window, key, type_text, wait_for, wait_idle, screenshot, apps) with _meta: {"anthropic/alwaysLoad": true}, so Claude Code loads them with the session (about 11.5k characters of schema); the others stay deferred. To load every tool up front instead, add "alwaysLoad": true to the server's entry in the MCP config.

Speed: skipping the permission prompt

Claude Code sends every MCP call that is not in its allow list to a prompt or, in auto mode, to its safety classifier; that adds roughly 0.7–2.5 s per call. If you trust all of the server's tools, add one rule to ~/.claude/settings.json:

{ "permissions": { "allow": ["mcp__desktop"] } }

Testing

Unit tests need no real desktop (the pure part of the Windows logic runs on Linux too):

cd rust && cargo test --release

tools/mcp_smoke_test.py tests end to end on the real desktop over the real MCP protocol (Linux, Kate): it opens the editor with a temporary file, finds the element, refocuses it after focus moved elsewhere, types text with Turkish characters through batch, tries rapid consecutive typing and key repeat, saves with Ctrl+S, reads the file back from disk, launches Kate with launch, and checks that the allowlist blocks an unknown app. It prints the duration of every call. Only the Python standard library is needed. Do not touch the keyboard and mouse while it runs.

python3 tools/mcp_smoke_test.py rust/target/release/desktop-mcp

tools/mcp_bench.py measures one or more servers with the same calls and prints a table of median ms (the window must be open):

python3 tools/mcp_bench.py --app kate --window notes.txt \
    --server "Rust=rust/target/release/desktop-mcp"

End-to-end task benchmark

Tool calls take milliseconds; what makes a desktop task slow is the number of model turns and the size of what the model reads. tools/e2e_bench.py measures that on real tasks: it runs Claude Code headless (claude -p, only this server's tools; built-in tools, the user's settings, plugins, hooks and skills left out) once per task × model × server binary and records model turns, desktop tool calls, wall time up to the final result split into startup, model and tool time (calls of one turn that run at once count once), the exit after the result on its own, reply characters and images, tokens, the model actually used, and whether the task got done. default in --models means Claude Code's own default model, not the one in your settings. The tasks in tools/e2e_tasks.json each open their own window on a fresh temp file (Kate: append a line and save, Save As to a new path, search and replace, read a value; Firefox: fill and submit a local form in a new window; Dolphin: create a folder), state the goal as a user would, check the outcome on disk, in the final answer or with a read-only query, and close only what they opened (the PID their setup recorded, or the one window carrying the run's unique tag).

python3 tools/e2e_bench.py --list
python3 tools/e2e_bench.py --tasks kate_append,kate_replace --models default,claude-haiku-5-5 \
    --server "old=/path/to/old/desktop-mcp" \
    --server "new=rust/target/release/desktop-mcp" --repeat 3 --results /tmp/e2e

It prints a Markdown table of medians and writes results.json (with every tool call, its time and reply size), table.md and each run's raw stream to --results (default: a new temp dir; never inside the repository). The agent types into the windows the tasks open (each run makes its window the active one first): do not use the keyboard or mouse while it runs. Firefox and Dolphin open the task's window in their running instance; close your own windows of them first, or pass --exclusive to skip those runs (each run notes other open windows of its app). Ctrl+C stops the agent with its whole process group before cleanup. python3 -m unittest tools/test_e2e_bench.py tests the measuring and the process handling without a desktop.

Measurements: KDE speed-up

tools/mcp_bench.py, KDE Plasma 6.8, Kate; median of 5 rounds (ms). "Before" is 2f2e5df (the version that removed Python), "After" this version:

The ~2 ms in activate_window is reading the active window title so act can report a window change. The real gain lies beyond milliseconds: short output, targeting with app + query in one call, wait_for and batch reduce the number of Claude's turns and the text it reads.

Measurements: compared with the old Python version

The server used to be written in Python; its last state is commit 4cd2632 in the git history. The Rust version offers the same tools with the same output: 1,499 lines of output on KDE and 2,025 on Hyprland were compared with the Python version, with no difference. The values were taken with tools/mcp_bench.py applying the same call sequence to both servers in turn; median of 5 rounds (ms, in a Kate window, same machine).

KDE Plasma 6 (Wayland):

Hyprland 0.56 (nested session inside KDE):

Windows (Notepad inside Wine 11.20; not a real Windows measurement):

Wine's UI Automation client does not support cache requests, tree walking and most properties (E_NOTIMPL), so tree, find, get_text, focus and click do not work under Wine on either server; the apps and activate_window figures in the table are for returning without finding a window. Most of the difference in the input steps comes from Rust's shorter waits.

The gap grows with larger trees: a 2,000-element tree of Dolphin including hidden elements takes 507 ms in Python and 48 ms in Rust. screenshot time is mostly the spectacle and grim programs themselves.

In nested Hyprland, Ctrl/Alt shortcuts from uinput get lost on their way through the outer KWin (on both servers), so no shortcut tests were run there. In a real Hyprland session uinput goes straight to Hyprland.

Known limitations

KDE:

• The server turns on the accessibility bus at run time. Apps started before that need one restart.

• Chromium, Chrome and Electron apps show their tree only when started with --force-renderer-accessibility.

• type_text pastes through the clipboard into fields other than Qt's multi-line editors (such as Kate). It first saves the clipboard text and puts it back once the target app has read the paste (seen through Wayland data-control). The typed text does not enter Klipper's history (it is offered with the password hint). The clipboard is not put back when it held an image, files or text over 4 MiB, nor when the app does not read the paste within 1 s (so that a late reader does not paste the user's clipboard); the reply says so in both cases. Rich text (HTML) comes back as plain text only.

• Direct insertion differs slightly from pasting: replacing selected text is two undo steps, with multiple cursors only the primary cursor gets the text, overwrite mode and Kate's vi mode are bypassed, and auto-indent and bracket completion do not run as they would on a paste.

Hyprland:

• Client-side shadows of GTK apps can shift click and focus clicks by a few pixels.

• Two windows of the same process with the same title cannot be told apart.

• Without wtype, type_text pastes through the clipboard as on KDE and puts the clipboard text back afterwards.

• Hyprland has no minimize: move_window with state: "minimize" moves the window to the special workspace special:minimized, and windows reports windows on a special workspace no monitor shows as minimized. restore brings it back to the workspace its monitor shows (not necessarily the one it came from).

• Maximize is Hyprland's maximized fullscreen mode (fullscreenstate 1 1); windows reports any fullscreen mode as maximized. Hyprland applies fullscreen states only to the focused window, so maximize and restore of a maximized window also focus it.

• Moving or resizing a tiled window makes it floating first (setfloating); it stays floating afterwards.

• Hyprland 0.57 takes only Lua dispatchers over its socket (hyprctl dispatch X became hl.dispatch(X)). The server notices the first Lua error and from then on sends the Lua form of every dispatcher it uses (hl.dsp.focus, hl.dsp.window.move/resize/float/fullscreen_state/ close, hl.dsp.cursor.move); verified against a nested Hyprland 0.57. Named workspaces have no number there; windows lists them by name.

Windows:

• Input cannot be sent to windows running as administrator from a server that is not (Windows UIPI).

• Microsoft Store apps may show up as applicationframehost.

• Key names follow the US keyboard layout positions; use type_text for text.

• In very large windows (browsers, Word) tree and find can be slow because they fetch the whole subtree at once.

All:

• Games and canvas-based interfaces mostly show up empty in the accessibility tree.

Security

The server runs with your user's permissions. The allowlist is a safety layer, not operating-system level isolation. Think twice before adding terminals, the file manager or system settings to the list.

License

MIT

Inline screenshots are encoded with the jpeg-encoder crate; this software is based in part on the work of the Independent JPEG Group.