perch
Controls apps on the GNOME desktop: lists open windows and their workspaces, reads a window's accessibility tree as compact element references, captures window screenshots, and performs click, type, key, scroll and drag steps. Window listing and focusing on GNOME Wayland go through a small GNOME Shell extension installed with a setup command.
Controls apps on Hyprland. Window screenshots and keyboard input can be kept in the background so a window on a hidden workspace never comes to the front, letting an agent drive an app without disturbing the user's screen (toggleable via COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND).
Controls apps on the KDE desktop: enumerates and targets windows, reads their buttons, fields and text as accessibility elements, takes per-window screenshots, and sends input steps to a chosen window.
Controls apps on the Sway compositor: window listing and targeting, accessibility tree reads, window screenshots, and click/type/key input steps.
Handles Wayland desktops for desktop control, including window targeting, screenshots and input, alongside X11, so an agent can see and act on windows under Wayland compositors.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@perch list my open windows and tell me which one has focus"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
perch
A small MCP server for desktop control. It lets an AI agent see and use the apps on your desktop with five compact tools, and keeps what it adds to the agent's context small.
perch does no desktop work itself. On Linux it runs computer-use-linux, which handles accessibility trees, screenshots, Wayland and X11 input, and window targeting on GNOME, KDE, Hyprland, Sway and others. perch shows the agent a shorter interface over it. Windows and macOS are planned.
Tools
Tool | Does |
| Open windows: id, app, title, focus, workspace |
| A window's buttons, fields and text as compact lines: |
| A picture of one window (or the screen), or a PNG file |
| Whether accessibility, input and the window list work, and what blocks them |
| Steps in one window: click, type, key, set, action, scroll, drag, focus, wait |
What it does for the agent:
e-refs.
read_windowlists elements ase34;act_windowturns them into the server's element index and snapshot.Screenshot pixels.
[x, y]inact_windoware pixels of the window's lastread_screenshot; perch converts them.Your screen stays yours where it can. Clicking an element whose primary action is click, press or toggle runs that action without focusing the window, on every desktop. On Hyprland, keys and window screenshots also stay in the background, so a test app on a hidden workspace never comes to the front; elsewhere both bring the window forward. Other elements (a table cell that expands or edits) are refused with their action names, to use as an
actionstep. Mouse steps need an explicitfocusstep, and keys for the focused window are refused if you moved the focus.Reused window ids. If a window closed and another app got its id,
act_windowrefuses instead of acting in the wrong app.Typed text is never echoed in results.
Size, measured against computer-use-linux 0.7.53 on Thunar:
computer-use-linux | perch | |
Tool list | 19 tools, 67.8 KB | 5 tools, 6.1 KB |
Server instructions | 7.6 KB | none |
One window's tree | 31.5 KB | 1.1 KB |
Related MCP server: planchette
Use
{
"mcpServers": {
"perch": { "command": "npx", "args": ["-y", "@redfoxxo/perch@1"] }
}
}Node.js 22.12 or newer. On Linux, computer-use-linux comes with perch (its binary downloads on first use) and needs the setup its README describes: accessibility on, and an input backend.
Variable | Default | Effect |
|
| Hyprland: keys and window screenshots without focusing the window. |
Other COMPUTER_USE_LINUX_* variables pass through unchanged.
On GNOME Wayland, computer-use-linux lists and focuses windows through a small Shell extension. Install it with the version perch runs, then log out and back in:
npx -y @redfoxxo/perch@1 setup-window-targetingDevelop
npm ci
npm test
npm run typecheck
npm run build # build/index.js is the serverLicense
MIT
Available Tools
5 toolsact_windowADestructive
Run steps in one window, in order, stopping at the first that fails: click, type, key, set, action, scroll, drag, focus, wait. Targets are e-refs from read_window or [x, y] pixels of its read_screenshot; drag takes pixels of a whole-screen read_screenshot. Element clicks, set and action never bring the window to the front; keys and typing don't on Hyprland, elsewhere they focus it. Mouse steps ([x, y] clicks, right or double clicks, scroll, drag) need a focus step first, which shows the window to the user. focused: true sends keys to the focused window, refused if the focus moved. then: tree (default after read_window) or screenshot (brings the window forward except on Hyprland) shows the result.
Inputs: steps (required), window (required), then.
read_screenshotA
Picture of one window (or the whole screen without window), longest side max_edge px. A window is captured in the background on Hyprland; elsewhere it comes to the front. Costs image tokens: prefer read_window. With file, saves a PNG there instead.
Inputs: file, window, max_edge.
read_statusARead-only
Whether desktop control works here: accessibility, input, windows, and what blocks it.
Inputs: no named parameters.
read_windowARead-only
A window's buttons, fields and text as compact lines, e.g. [e34 button "Save" disabled]. Act on them with act_window using the e-refs; a new read_window (of any window) replaces them.
Inputs: window (required).
read_windowsARead-only
Open windows, one per line: id, app, title, focused, workspace.
Inputs: no named parameters.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 5 tools
read_window (element tree of one window) vs read_screenshot (image) is cleanly separated, and act_window is clearly the write-side counterpoint. The only real risk is read_window vs read_windows, where the plural name could be misread as 'read contents of all windows' rather than 'list open windows', though the description disambiguates.
All names are snake_case verb_noun (read_window, read_windows, read_screenshot, read_status, act_window), with a natural read/act split that maps to inspect vs. manipulate. No mixed conventions or vague verbs like process/run.
Five tools is tight and well-scoped: four read-side probes (element tree, window list, screenshot, capability status) plus one consolidated act_window that absorbs click/type/key/scroll/drag/focus/wait as steps. Nothing feels redundant or absent on count grounds.
The lifecycle of inspect-then-act on existing windows is fully covered, including a preflight read_status and post-action 'then' verification. Gaps remain around launching apps/closing windows and clipboard access, so an agent can inspect and drive open windows but cannot start a program itself.
Maintenance
Related MCP Connectors
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
Connect ChatGPT, Claude, Cursor or VS Code to your Windows PC: files, terminal, browser, apps.
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Eyes and hands on real Windows PCs — observe, click, type via Glasswarp API.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables comprehensive Windows desktop automation including screen capture, OCR text extraction, mouse/keyboard control, window management, process control, and clipboard operations through 25+ tools for AI agents.38 PyPI8MIT
- AlicenseNot gradedqualityCmaintenanceEnables coding agents to control other application windows by providing primitives for clicking, typing, capturing screenshots, and window management. Supports macOS, Windows, and Linux.MIT
- AlicenseNot gradedqualityBmaintenanceLets AI agents see and control desktop applications through the accessibility layer, enabling clicking, typing, scrolling, dragging, and window/app management across macOS, Windows, and Linux entirely on the local machine.25MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to automate desktop operating systems and applications through UI automation, screenshots, and input simulation.1MIT