desktop-control
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
desktop-control — MCP server (screenshot + keyboard/mouse + accessibility + voice cmds/STT + voice synth/TTS) [[JUST For Fun]]
Lets an MCP client (e.g. Claude Code) drive a GUI: capture the screen, then move/click/type and — optionally — target controls by name via the OS accessibility layer.
Cross-platform: Windows (pyautogui + UI Automation) and Linux
(X11 via pyautogui; Wayland via grim/gnome-screenshot + ydotool; AT-SPI
accessibility). OS specifics are isolated in backends.py; server.py is the
shared tool surface. See DESIGN.md for the design, ARCHITECTURE.md for the
big picture (desktop-control + voice + bridge), and DEPLOY.md for a
step-by-step WSL2 + Windows install.
1. Install (on the target machine)
Python 3.10+ on the machine whose desktop you want to drive.
python -m pip install -r requirements.txtPlatform extras:
Platform | Needed | How |
Windows | UI Automation |
|
Linux X11 | input + capture | works out of the box (pyautogui + mss) |
Linux Wayland | capture |
|
Linux Wayland | input |
|
Linux | accessibility | distro pkg |
Wayland blocks synthetic input by design. The Wayland path is best-effort (
drag/scrollunsupported via ydotool; keys limited to a common keymap). Prefer an X11 session, or rely on the AT-SPI tools (ui_tree/ui_click).
Related MCP server: vision-input
2. Run modes
The server must run on the machine being driven. Two ways to wire it to Claude Code:
stdio (client launches it). From WSL targeting Windows, point the command at
python.exeso the Windows Python runs. On native Linux/Windows, usepython/python3.SSE (you launch it):
python server.py --sse→http://127.0.0.1:8000/sse, then connect the client to that URL.
WSL2 users: WSLg can't see the Windows desktop, so to drive Windows run the server on Windows (via
python.exeor SSE). SeeWSL2.mdfor ready configs and the voice-in-WSL2 path.
# stdio
claude mcp add desktop-control -- python3 /path/to/server.py # Linux
claude mcp add desktop-control -- python.exe 'C:\path\to\server.py' # Windows / from WSL
# sse
claude mcp add --transport sse desktop-control http://localhost:8000/sseSee mcp.json.example for project-scope .mcp.json blocks.
3. Tools
Tool | Purpose |
| platform, session, real & image size, scale, accessibility |
| PNG of the screen, scaled to |
| mouse |
| wheel scroll |
| keyboard |
| pause (≤30s) |
| accessibility tree dump / click by name (UIA or AT-SPI) |
| resumable human-guided flows |
| pause for the user and capture their response |
Coordinate model
Screenshots are scaled so the longest side ≤ MCP_DESKTOP_MAX_DIM (default
1280). All click/move/drag coordinates are in the returned screenshot's pixel
space; the server converts them to real pixels. Call screen_size() for the
mapping. For pixel-fragile UIs, prefer ui_tree + ui_click.
4. Configuration (env)
Variable | Default | Effect |
|
| max screenshot side (px) |
|
| if |
|
| inter-action delay used by pyautogui |
|
|
|
|
| comma-separated tokens treated as confirmation in guided flows |
|
| comma-separated tokens treated as rejection in guided flows |
5. Voice-driven loop (optional)
See voice/README.md for prerequisites, configuration, and the local/offline
voice loop:
wake word / speech capture
STT (Whisper)
agent round-trip
TTS playback
6. Limits
Multi-monitor: primary/virtual screen.
Wayland input is restricted (see §1); X11 or AT-SPI recommended.
Latency: each step is a screenshot→decide→act round-trip.
7. Voice (optional) — voice/
Offline voice command (STT) and spoken responses (TTS): a voice MCP
server (speak / listen / transcribe_file) plus a hands-free loop
(wake word → record → agent → speak). Engines: faster-whisper + pyttsx3/Piper,
all local. See voice/README.md.
8. WSL2 → Windows delegation (optional) — bridge/
ask_windows_agent lets a WSL2 client delegate a task to a Windows
agent (Claude/OpenAI/Mistral/Copilot; CLI or API), so the Windows side can drive
desktop/voice while WSL2 orchestrates. The legacy tool name ask_windows_claude
remains available for compatibility. See bridge/README.md and WSL2.md.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Control a real browser from AI agents: navigate, click, type, screenshot, read the page.
Connect ChatGPT, Claude, Cursor or VS Code to your Windows PC: files, terminal, browser, apps.
Give your agent eyes: screenshots of any web page it can see, pages as markdown, and PDFs.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables computer control via mouse, keyboard, OCR, and screen/window management, similar to Anthropic's computer-use.MIT
- FlicenseNot gradedqualityDmaintenanceEnables AI to capture screenshots and control mouse and keyboard for automated desktop interaction.-
- FlicenseNot gradedqualityDmaintenanceEnables AI assistants to control Windows GUI by listing and focusing windows, capturing element snapshots via UIA/OCR/CDP, performing clicks/inputs/scrolls, verifying changes, waiting for screen updates, taking screenshots, and obtaining visual descriptions.2-
- AlicenseNot gradedqualityCmaintenanceEnables AI hosts to drive cross-platform desktop GUI automation and browser control, providing tools to read, click, type, send shortcuts, screenshot, and verify GUI elements and web pages.3,776 npmMIT