perch
Perch is an MCP server for desktop control, letting an AI agent inspect open windows and act on them through five tools.
List open windows with id, app, title, focus, and workspace.
Read a window’s buttons, fields, and text as compact e-ref lines for targeting.
Capture a window or the whole screen as an image, optionally saving a PNG and limiting image size.
Check whether accessibility, input, and window listing work and what blocks them.
Run ordered actions in a window: click e-refs or screenshot pixels, type text, press keys/chords, set fields or sliders, invoke named actions, scroll, drag, focus, and wait.
Return an updated tree, screenshot, or nothing after actions.
Keep desktop interaction in the background where supported, and use screenshots/pixel clicks for apps without accessibility.
Supports Linux and Windows; macOS is not supported yet.
Controls apps on the GNOME desktop: lists open windows and their workspaces, reads a window's accessibility tree as compact element references, captures window screenshots, and performs click, type, key, scroll and drag steps. Window listing and focusing on GNOME Wayland go through a small GNOME Shell extension installed with a setup command.
Controls apps on Hyprland. Window screenshots and keyboard input can be kept in the background so a window on a hidden workspace never comes to the front, letting an agent drive an app without disturbing the user's screen (toggleable via COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND).
Controls apps on the KDE desktop: enumerates and targets windows, reads their buttons, fields and text as accessibility elements, takes per-window screenshots, and sends input steps to a chosen window.
Controls apps on the Sway compositor: window listing and targeting, accessibility tree reads, window screenshots, and click/type/key input steps.
Handles Wayland desktops for desktop control, including window targeting, screenshots and input, alongside X11, so an agent can see and act on windows under Wayland compositors.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@perch list my open windows and tell me which one has focus"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
perch
A small MCP server for desktop control. It lets an AI agent see and use the apps on your desktop with five compact tools, and keeps what it adds to the agent's context small.
perch does no desktop work itself. On Linux it runs computer-use-linux (accessibility trees, screenshots, Wayland and X11 input, window targeting on GNOME, KDE, Hyprland, Sway and others). On Windows it runs the Windows runtime of open-computer-use (UI Automation, window messages, window capture). Both come with perch, behind the same short interface. macOS is not supported yet.
Requirements
Node.js 22.12 or newer.
Linux: x64 or arm64 with glibc (the builds computer-use-linux ships), in a desktop session. Accessibility on, and an input backend, as computer-use-linux's quick start describes. Its binary downloads on first use if npm skipped its install script.
GNOME Wayland: computer-use-linux lists and focuses windows through a small Shell extension. Install it with the version perch runs, then log out and back in:
npx -y @redfoxxo/perch@1 setup-window-targetingWindows: nothing else to install. Run perch in the signed-in desktop session, not over SSH or as a service.
read_status tells what is missing on a desktop, and how to fix it.
Related MCP server: planchette
Setup
perch runs over stdio; npx -y @redfoxxo/perch@1 starts it.
Claude Code:
claude mcp add perch -- npx -y @redfoxxo/perch@1On native Windows, start npx through cmd /c:
claude mcp add perch -- cmd /c npx -y @redfoxxo/perch@1.
opencode (opencode.json):
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"perch": { "type": "local", "command": ["npx", "-y", "@redfoxxo/perch@1"] }
}
}Other clients (mcpServers JSON):
{
"mcpServers": {
"perch": { "command": "npx", "args": ["-y", "@redfoxxo/perch@1"] }
}
}Environment variables (Linux):
Variable | Default | Effect |
|
| Hyprland: keys and window screenshots without focusing the window. |
| the client's name | Name computer-use-linux shows on screen while it acts |
Other COMPUTER_USE_LINUX_* variables pass through to computer-use-linux
unchanged. On Windows, perch starts the runtime with
__COMPAT_LAYER=HighDpiAware unless you set that variable, so window
positions and UI Automation coordinates are both in pixels on scaled
displays.
Tools
Tool | Does |
| Open windows: id, app, title, focus, workspace |
| A window's buttons, fields and text as compact lines: |
| A picture of one window (or the screen), or a new PNG file |
| Whether accessibility, input and the window list work, and what blocks them |
| Steps in one window: click, type, key, set, action, scroll, drag, focus, wait |
What perch does for the agent:
e-refs.
read_windowlists elements ase34;act_windowturns them into the server's element index and snapshot. A newread_windowreplaces them.Screenshot pixels.
[x, y]inact_windoware pixels of the window's lastread_screenshot; perch converts them, and refuses points outside that image.Your screen stays yours where it can. Clicking an element whose primary action is click, press or toggle (Windows: invoke, select or toggle) runs that action without focusing the window. Other elements (a table cell that expands or edits) are refused with their action names, to use as an
actionstep. On Hyprland and Windows, keys, typing and window screenshots stay in the background too; elsewhere they bring the window forward. Mouse steps need an explicitfocusstep, and keys for the focused window are refused if the focus moved.Reused window ids. If a window closed and another app got its id,
act_windowrefuses instead of acting in the wrong app.Typed text is never echoed in results. Long text and field values in trees are cut at 1,000 characters.
Size, measured against computer-use-linux 0.7.53 on Thunar:
computer-use-linux | perch | |
Tool list | 19 tools, 67.8 KB | 5 tools, 4.8 KB |
Server instructions | 7.6 KB | none |
One window's tree | 31.5 KB | 1.1 KB |
Limitations
Apps without accessibility (old toolkits, Wine, games, canvases) show no tree; use
read_screenshotand[x, y]clicks there.Drags on Linux take pixels of a whole-screen screenshot; on Windows, pixels of the window's screenshot, like clicks.
Windows, background input: keys and typing are posted to the window as messages. Classic Win32 fields take them; some newer apps (Chromium, Electron, WinUI) ignore them, so the step says to check the result.
focused: trueafter afocusstep sends real keystrokes instead.Windows, classic controls come forward when UI Automation acts on them: in CI, a WinForms button and check box (click,
action) and Notepad's text field (set) all brought their window to the front. The step then says "the app came to the front".Windows, trees show no disabled or checked state: the runtime doesn't report them.
Windows, refused apps: open-computer-use won't touch terminals, PowerShell, password managers or Windows Security. Apps running as administrator can be read but not clicked or typed into unless perch runs as administrator too.
Windows, speed: the runtime runs every operation in a new process, so a crashing capture can't take perch down. On GitHub's Windows runner, reads took 0.1 s and
act_windowcalls 0.5–1.2 s.Install size: npm installs both runtimes (about 40 MB) on every platform, since open-computer-use's package doesn't declare its platforms.
Development
See DEVELOPMENT.md.
License
MIT
Available Tools
5 toolsact_windowADestructive
Run steps in one window, in order; stops at the first that fails. Targets: e-refs from read_window, or [x, y] pixels of this window's last read_screenshot (drag: of a whole-screen read_screenshot; Windows: of the window's). Element clicks, set and action don't bring the window to the front (Windows: classic controls may, and the step says so). Keys and typing stay in the background on Hyprland and Windows (some Windows apps ignore them: check); elsewhere they focus the window. Mouse steps ([x, y] clicks, right or double clicks, scroll, drag) need a focus step first, which shows the window to the user. focused: true sends real keys (any layout and characters) to the window after a focus step; refused if the focus moved. then: tree (default after read_window), screenshot (raises the window except on Hyprland and Windows) or none.
Inputs: steps (required), window (required), then.
read_screenshotA
Picture of one window (or the whole screen without window), longest side max_edge px. A window is captured in the background on Hyprland and Windows; elsewhere it comes to the front. Costs image tokens: prefer read_window. With file, saves a PNG there instead.
Inputs: file, window, max_edge.
read_statusARead-only
Whether desktop control works here: accessibility, input, windows, and what blocks it.
Inputs: no named parameters.
read_windowARead-only
A window's buttons, fields and text as compact lines, e.g. [e34 button "Save" disabled]. Act on them with act_window using the e-refs; a new read_window (of any window) replaces them.
Inputs: window (required).
read_windowsBRead-only
Open windows, one per line: id, app, title, focused, workspace.
Inputs: no named parameters.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 5 tools
read_windows (enumerate all windows) and read_window (inspect one window's element tree) are the only pair at risk of confusion, and their descriptions make the plural/singular distinction workable. read_screenshot, read_status, and act_window each target clearly different things: pixels, capability diagnostics, and mutation.
All five names follow a strict verb_noun pattern, with read_* for observation tools and act_window for the sole mutating tool. The read/act split is semantically meaningful rather than an arbitrary deviation.
Five tools is tightly scoped to the domain: enumerate, inspect, screenshot, check capability, and act. Each tool occupies a necessary slot in the perceive-then-act loop, with no redundant or filler entries.
The read-and-act lifecycle for existing windows is fully covered, including status checks and image fallback. Minor gaps remain: there is no way to launch an application or explicitly close a window outside of clicking an in-app control, which an agent may need for common desktop workflows.
Maintenance
Related MCP Connectors
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
Connect ChatGPT, Claude, Cursor or VS Code to your Windows PC: files, terminal, browser, apps.
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Eyes and hands on real Windows PCs — observe, click, type via Glasswarp API.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables comprehensive Windows desktop automation including screen capture, OCR text extraction, mouse/keyboard control, window management, process control, and clipboard operations through 25+ tools for AI agents.55 PyPI8MIT
- AlicenseNot gradedqualityCmaintenanceEnables coding agents to control other application windows by providing primitives for clicking, typing, capturing screenshots, and window management. Supports macOS, Windows, and Linux.MIT
- AlicenseNot gradedqualityBmaintenanceLets AI agents see and control desktop applications through the accessibility layer, enabling clicking, typing, scrolling, dragging, and window/app management across macOS, Windows, and Linux entirely on the local machine.27MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to automate desktop operating systems and applications through UI automation, screenshots, and input simulation.1MIT