Universal Computer Control MCP
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Universal Computer Control MCP take a screenshot, find the Save button, click it, and verify the file saved"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Universal Computer Control MCP
Give your AI agent eyes and hands on any computer. Observe → understand → act → verify. No browser MCPs. No Playwright. Just real control.
A cross-platform computer control MCP server that lets an AI agent operate a Windows or Linux computer like a human: observe the screen → understand the UI → choose an action → perform it → observe again → verify → recover if needed.
The system deliberately has no dependency on Chrome MCP, browser MCP, Playwright MCP or any other external MCP server. Browsers, IDEs, office suites, terminals, file managers, Electron apps and canvas-heavy applications are all just GUI applications. Multiple backends live inside the server and the best available mechanism is chosen automatically per action.
AI Agent / LLM
| MCP (stdio)
v
+------------------------------+
| Universal Computer MCP Server |
+---------------+--------------+
v
+------------------------------+
| Computer Control Engine | <- usable WITHOUT MCP
+---------------+--------------+
| | |
v v v
Accessibility Vision/OCR Physical Input
Layer Layer Layer
| | | | | |
UIA AT-SPI OCR OpenCV Mouse Keyboard
(Windows)(Linux) Tesseract VLM PyAutoGUI OS APIsHighlights
29 stable MCP tools under the
computer.namespace — capabilities, not implementation details. Nopyautogui_click, noocr_click, noplaywright_click.Interaction hierarchy per action: Windows UIA → Linux AT-SPI → semantic element → OCR text → image template → vision-language model → coordinates → PyAutoGUI. Every fallback is automatic.
computer.observeis the core tool: one structured snapshot with screen geometry, active window, cursor, OCR'd text and normalized UI elements (stableelement_Nids within a cycle).
Related MCP server: OpenSky
Requirements
Mandatory | Optional (recommended) | |
Python | 3.11+ | |
Core |
| |
Input |
| |
Windows |
| |
Linux |
| |
Screenshots |
| |
OCR |
| |
Vision |
| |
VLM |
|
Installation
Windows 10/11
git clone https://github.com/adkdev200/universal-computer-control.git && cd universal-computer-control
powershell -ExecutionPolicy Bypass -File scripts\install_windows.ps1Manual steps:
Install Python 3.11+ (
winget install -e --id Python.Python.3.12), tick Add to PATH.python -m venv .venv && .venv\Scripts\pip install -e ".[input,windows,ocr,vision,screenshot,vlm]"Install Tesseract OCR:
winget install -e --id UB-Mannheim.TesseractOCR(setocr.tesseract_cmdinconfig.yamlif it is not onPATH).Run:
.venv\Scripts\universal-computer-control(alias:.venv\Scripts\ucc)
Windows UI Automation needs no extra permissions. If you run the server elevated, non-elevated windows may refuse automation (UIA integrity levels).
Linux (X11 or XWayland)
git clone https://github.com/adkdev200/universal-computer-control.git && cd universal-computer-control
bash scripts/install_linux.sh --yes # add --no-sudo to skip system packages
./.venv/bin/ucc-doctor # verify every backend, exact fix hintsThe installer is fully non-interactive with --yes: it installs system
packages (wmctrl/xdotool/xclip/scrot/tesseract/at-spi2-core), all Python
extras, and a ready ~/.universal-computer/config/config.yaml with OCR,
template matching and the VLM provider enabled. ucc-doctor then probes every
backend live and prints copy-paste fix commands for anything missing. Note:
even on machines with no wmctrl/xdotool/xclip installed, the Linux window
backend now works through a built-in python-Xlib (EWMH/ICCCM) fallback - the
system tools only make it more robust.
Manual steps:
Python 3.11+ and system packages:
sudo apt install python3 python3-venv python3-pip \ wmctrl xdotool xclip scrot \ tesseract-ocr at-spi2-corepython3 -m venv .venv && ./.venv/bin/pip install -e ".[input,linux,ocr,vision,screenshot,vlm]"Run from a graphical session:
./.venv/bin/universal-computer-control(alias:./.venv/bin/ucc)
AT-SPI requirements (accessibility): at-spi2-core must be running (it is
on desktop distros) and for GNOME/GTK apps enable
gsettings set org.gnome.desktop.interface toolkit-accessibility true.
X11 vs Wayland: X11 is fully supported. On Wayland-with-XWayland, input
and screenshots work through XWayland but foreign window management may be
restricted by the compositor. On pure Wayland (no DISPLAY), the window
backend disables itself with a warning; AT-SPI still works for cooperating
toolkits. See docs/TROUBLESHOOTING.md.
Registering the server with an MCP client
claude_desktop_config.json (see examples/mcp-config.example*.json):
{
"mcpServers": {
"universal-computer": {
"command": "/absolute/path/to/universal-computer-control/.venv/bin/python",
"args": ["-m", "universal_computer"],
"env": {}
}
}
}Any MCP client that speaks stdio works the same way. The engine is also a plain library — no MCP required:
from universal_computer import ComputerControlEngine, load_config
computer = ComputerControlEngine(load_config())
obs = await computer.observe("normal")
await computer.click("Continue")
await computer.type("hello")Tools (MCP API)
Group | Tools |
Observation |
|
Mouse |
|
Keyboard |
|
Windows |
|
Clipboard |
|
System |
|
Admin |
|
computer.observe returns a normalized snapshot:
{
"ok": true,
"id": "obs_17",
"level": "normal",
"screen": {"width": 1920, "height": 1080, "monitors": [...]},
"active_window": {"title": "Google Chrome", "application": "chrome.exe", "process_id": 1234},
"cursor": {"x": 812, "y": 530},
"elements": [
{"id": "element_1", "role": "button", "name": "Continue",
"bbox": [1050, 700, 1170, 750], "source": "uia", "confidence": 0.99}
],
"ocr": [
{"text": "Continue", "bbox": [1050, 700, 1170, 750], "confidence": 0.97}
]
}computer.backend_status reports per-backend health:
{
"pyautogui": {"available": true, "healthy": true},
"uia": {"available": true, "healthy": true},
"atspi": {"available": false, "healthy": false,
"error": "pyatspi import failed (...); install python3-pyatspi"},
"ocr": {"available": true, "healthy": true}
}Configuration
config.yaml (see config.example.yaml) plus UCC_ environment overrides
with __ nesting:
UCC_ENGINE__VERIFICATION_MODE=strict
UCC_SECURITY__MODE=standard
UCC_SECURITY__ALLOWED_COMMANDS='["ls", "git status", "xdotool"]'
UCC_LOGGING__LEVEL=DEBUG
UCC_VISION__VLM__ENABLED=trueConfigurable: observation level & caching, verification mode
(off/basic/strict), PyAutoGUI pauses/failsafe, OCR provider & language,
vision threshold, template directory, VLM endpoint (key read from the
environment variable named in vision.vlm.api_key_env), security mode,
allowlists, rate limits, timeouts, persistence retention, log levels and
per-backend priorities (backends.priorities: {pyautogui: 5}).
Security model
Mode |
|
|
| allowed (still rate-limited); | allowed (still rate-limited) |
| requires | allowed unless a non-empty |
| only explicitly allowlisted; | exact allowlist match required |
Unquoted shell operators (;, |, &, `, <, >, $) only pass the
allowlist when an entry matches the full command string — a bare
program-name entry (git, python, ...) is not enough.
Always on: rate limiting (max_actions_per_minute), command timeouts,
credential redaction in logs and command output, no clipboard/typed-text
logging, close_window/run_command/launch_application confirmation gates,
and the emergency stop — computer.emergency_stop() halts every subsequent
action (checked before each attempt) and is persisted as a stop file so it
survives restarts and can be triggered externally.
Extending
New OCR: implement
universal_computer.vision.base.OCRProvider(detect_text(image) -> list[OCRResult]) and register it inVisionServices.from_config.New vision model: implement
VisionLanguageModel.locate_element(async, description →VisionResultwith bbox). The OpenAI-compatible implementation shows the JSON contract.New platform (macOS): subclass the
backends.baseABCs (InputBackend,ScreenshotBackend,AccessibilityBackend,WindowManagementBackend,ClipboardBackend,ApplicationBackend) and register them incore/bootstrap.py. Nothing else changes — the engine only knows capabilities.Template images: put PNGs in
vision.template_dir; refer to them by name fromcomputer.find_visualor as click targets.
Development
pip install -e ".[dev]"
pytest tests # 160 unit + integration tests, no display needed
ruff check src testsscripts/e2e_live_test.py is an end-to-end harness that boots its own
Xvfb + openbox session, creates real X11 windows, and exercises every
backend live (screenshots, OCR, window ops via both the tool paths and the
python-Xlib fallback, OpenCV template matching, the VLM pipeline against a
mock OpenAI-compatible endpoint, clipboard, input, launch, emergency stop):
python scripts/e2e_live_test.py # full stack
python scripts/e2e_live_test.py --no-tools # zero X11 tools: Xlib fallbackUnit tests run on any machine (OS-specific pieces are mocked). The
integration test boots the real server over stdio. See
docs/ARCHITECTURE.md for the module map and docs/TROUBLESHOOTING.md for
platform-specific fixes.
Contributors & Acknowledgments
Z.ai (GLM) — co-developer: MCP server, cross-platform backends, vision pipeline, python-Xlib window fallback and
ucc-doctordiagnosticsClaude Code (Anthropic) — co-developer: code generation, testing and plug-and-play hardening
@adkdev200 — creator & maintainer
This project is built collaboratively with AI pair-programming: the majority of the codebase was written and reviewed by Z.ai's GLM and Anthropic's Claude Code under the direction of the maintainer.
Available Tools
29 toolscomputer.backend_statusA
Report health/availability of every backend (input, screenshot, accessibility, window management, clipboard, OCR, vision) plus security settings and emergency-stop state.
Inputs: no named parameters.
computer.clickA
Click a UI element. Target can be: visible text ('Continue'), an element id from computer.observe ('element_17'), or coordinates ({'x': 500, 'y': 300}). The engine resolves it via accessibility first, then OCR, then vision - and verifies the result.
Inputs: target (required), button, verify.
computer.close_windowA
Close a window gracefully (WM_CLOSE). Requires confirm=true; modal confirmation dialogs remain visible for the agent to handle.
Inputs: window (required), confirm.
computer.double_clickB
Double-click a UI element (same flexible target rules as computer.click).
Inputs: target (required), verify.
computer.dragC
Drag from one target to another (text, element ids or coordinates).
Inputs: end (required), start (required), duration_s.
computer.emergency_stopA
EMERGENCY STOP: immediately halt all automation (checked before every action and attempt). Persists until reset_emergency_stop.
Inputs: no named parameters.
computer.find_textA
Find text anywhere on screen (OCR + accessibility) with fuzzy matching. Returns matches with bounding boxes so you can click them. Tolerates case, arrows and small typos ('Continue' matches 'Continue →').
Inputs: query (required), limit, threshold.
computer.find_visualA
Find a UI element visually: by image template name (from the configured template directory) or by natural-language description via the optional vision model. Returns a bounding box.
Inputs: template, threshold, description.
computer.focus_windowA
Focus a real OS window by (partial) title or application name, e.g. 'Chrome'. Works without any application-specific integration.
Inputs: window (required).
computer.get_active_windowA
Get the title, application and process of the currently focused window.
Inputs: no named parameters.
computer.get_clipboardA
Read the clipboard text.
Inputs: no named parameters.
computer.get_ui_treeB
Get the accessibility tree of the active (or a named) window as a nested structure of roles, names and bounding boxes. Uses Windows UI Automation or Linux AT-SPI; returns an explanatory error when no accessibility backend is available.
Inputs: window, max_depth.
computer.hotkeyA
Press a key combination. Pass a string like 'ctrl+s' or a list like ['ctrl', 'shift', 't'].
Inputs: keys (required).
computer.launch_applicationA
Launch a GUI application ('chrome', 'code', a .desktop entry, or a full path). Subject to security.allowed_applications, rate limiting and security.confirmation_required (confirm=true by default).
Inputs: command (required), confirm.
computer.list_windowsA
List all top-level OS windows with title, application, pid and state.
Inputs: no named parameters.
computer.maximize_windowC
Maximize a window by title/application.
Inputs: window (required).
computer.minimize_windowC
Minimize a window by title/application.
Inputs: window (required).
computer.move_mouseB
Move the mouse cursor to a target (text, element id or coordinates).
Inputs: target (required).
computer.observeA
Observe the current computer state: screen geometry, active window, cursor, OCR'd on-screen text and accessibility UI elements. Levels: 'minimal' (fast), 'normal' (screenshot+OCR+accessibility), 'full' (adds focused element details). This is the primary tool for understanding the screen before acting.
Inputs: level.
computer.pressC
Press a keyboard key ('enter', 'tab', 'escape', 'f5', ...).
Inputs: key (required).
computer.reset_emergency_stopA
Clear a previously triggered emergency stop.
Inputs: no named parameters.
computer.restore_windowC
Restore a minimized/maximized window.
Inputs: window (required).
computer.right_clickC
Right-click a UI element (same flexible target rules as computer.click).
Inputs: target (required), verify.
computer.run_commandA
Run a shell command (security.allowlist applies; see computer.backend_status). Requires confirm=true. Output is truncated and credential-redacted.
Inputs: command (required), confirm, timeout_s.
computer.screenshotA
Take a screenshot of the whole virtual desktop, a specific monitor or a [left, top, right, bottom] region. Saves it to the persistence directory and returns the path.
Inputs: region, monitor, return_image.
computer.scrollA
Scroll the mouse wheel. Positive amount scrolls up, negative down. Optionally at specific coordinates (x, y).
Inputs: x, y, amount.
computer.set_clipboardA
Write text to the clipboard (useful as a typing fallback).
Inputs: text (required).
computer.typeA
Type text at the currently focused input. Handles Unicode and multiline text (clipboard fallback used automatically). The typed content is never logged.
Inputs: text (required), verify.
computer.wait_forA
Wait until something changes on screen. Expectations: 'any_change', 'window' (value = window title substring), 'text' (value = OCR text substring). Use after actions to await UI reactions instead of blind sleeps.
Inputs: value, timeout_s, expectation.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 29 tools
Most tools have clearly distinct purposes (click vs drag vs scroll, find_text vs find_visual, the many window operations are each specific). The only mild overlap is between computer.observe, computer.screenshot and computer.get_ui_tree, but observe's description explicitly positions it as the primary screen-understanding tool, keeping confusion low.
Every tool follows a strict namespace.verb_noun snake_case pattern (computer.click, computer.focus_window, computer.list_windows, computer.emergency_stop). No camelCase or mixed conventions anywhere, so the surface is highly predictable.
29 tools is on the high side, but the domain (full desktop automation spanning input, OCR/vision, accessibility, window management, clipboard, app launching and shell) legitimately needs this breadth and each tool maps to a distinct capability. Slightly heavy rather than gratuitous.
Coverage is comprehensive: observation/screenshot/UI-tree, the full input set (click variants, type, hotkey, scroll, drag), window lifecycle (list/focus/minimize/maximize/restore/close), clipboard read/write, app launch, shell execution, backend health, and safety controls (emergency stop/reset). No obvious dead ends for the stated purpose.
Maintenance
Related MCP Connectors
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
Control a real browser from AI agents: navigate, click, type, screenshot, read the page.
A cloud Linux desktop for your AI: browse, click, type, read the screen and run commands.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Related MCP Servers
- AlicenseAqualityCmaintenanceCross-platform desktop automation MCP server that lets AI agents capture screenshots, run OCR with UI-element classification, control mouse/keyboard, and launch programs on Linux, macOS, and Windows.202MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to automate desktop operating systems and applications through UI automation, screenshots, and input simulation.1MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to inspect and operate Windows applications through structured UI Automation maps, humanized mouse and keyboard input, and on-demand screenshots or OCR via MCP. It supports locating elements by role, name, or query; clicking, typing, scrolling, window management, event watching, and chained multi-step actions without relying on image analysis.MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to observe and operate real desktops, mobile devices, and browser pages through screenshots, native accessibility trees, and web automation, including clicking, typing, hotkeys, and UI lookup across macOS, Linux, Windows, Android, and iOS.7Apache 2.0