Skip to main content
Glama
Universal Computer Control MCP

Universal Computer Control MCP

by TNTXZ

Universal Computer Control MCP

Give your AI agent eyes and hands on any computer. Observe → understand → act → verify. No browser MCPs. No Playwright. Just real control.

Platform Python License Tests MCP

A cross-platform computer control MCP server that lets an AI agent operate a Windows or Linux computer like a human: observe the screen → understand the UI → choose an action → perform it → observe again → verify → recover if needed.

The system deliberately has no dependency on Chrome MCP, browser MCP, Playwright MCP or any other external MCP server. Browsers, IDEs, office suites, terminals, file managers, Electron apps and canvas-heavy applications are all just GUI applications. Multiple backends live inside the server and the best available mechanism is chosen automatically per action.

                AI Agent / LLM
                       | MCP (stdio)
                       v
        +------------------------------+
        | Universal Computer MCP Server |
        +---------------+--------------+
                        v
        +------------------------------+
        |   Computer Control Engine    |  <- usable WITHOUT MCP
        +---------------+--------------+
            |            |             |
            v            v             v
     Accessibility   Vision/OCR    Physical Input
       Layer            Layer          Layer
      |      |         |      |       |       |
     UIA   AT-SPI     OCR  OpenCV   Mouse  Keyboard
    (Windows)(Linux) Tesseract  VLM  PyAutoGUI  OS APIs

Highlights

  • 29 stable MCP tools under the computer. namespace — capabilities, not implementation details. No pyautogui_click, no ocr_click, no playwright_click.

  • Interaction hierarchy per action: Windows UIA → Linux AT-SPI → semantic element → OCR text → image template → vision-language model → coordinates → PyAutoGUI. Every fallback is automatic.

  • computer.observe is the core tool: one structured snapshot with screen geometry, active window, cursor, OCR'd text and normalized UI elements (stable element_N ids within a cycle).

Related MCP server: OpenSky

Requirements

Mandatory

Optional (recommended)

Python

3.11+

Core

mcp, pydantic, PyYAML, Pillow

Input

PyAutoGUI ([input])

Windows

pywinauto, pygetwindow, psutil ([windows])

Linux

psutil ([linux]), system wmctrl, xdotool, xclip

Screenshots

mss ([screenshot]), scrot on X11

OCR

pytesseract + system tesseract-ocr ([ocr])

Vision

opencv-python, numpy ([vision])

VLM

httpx ([vlm]) + an OpenAI-compatible endpoint

Installation

Windows 10/11

git clone https://github.com/adkdev200/universal-computer-control.git && cd universal-computer-control
powershell -ExecutionPolicy Bypass -File scripts\install_windows.ps1

Manual steps:

  1. Install Python 3.11+ (winget install -e --id Python.Python.3.12), tick Add to PATH.

  2. python -m venv .venv && .venv\Scripts\pip install -e ".[input,windows,ocr,vision,screenshot,vlm]"

  3. Install Tesseract OCR: winget install -e --id UB-Mannheim.TesseractOCR (set ocr.tesseract_cmd in config.yaml if it is not on PATH).

  4. Run: .venv\Scripts\universal-computer-control (alias: .venv\Scripts\ucc)

Windows UI Automation needs no extra permissions. If you run the server elevated, non-elevated windows may refuse automation (UIA integrity levels).

Linux (X11 or XWayland)

git clone https://github.com/adkdev200/universal-computer-control.git && cd universal-computer-control
bash scripts/install_linux.sh --yes     # add --no-sudo to skip system packages
./.venv/bin/ucc-doctor                  # verify every backend, exact fix hints

The installer is fully non-interactive with --yes: it installs system packages (wmctrl/xdotool/xclip/scrot/tesseract/at-spi2-core), all Python extras, and a ready ~/.universal-computer/config/config.yaml with OCR, template matching and the VLM provider enabled. ucc-doctor then probes every backend live and prints copy-paste fix commands for anything missing. Note: even on machines with no wmctrl/xdotool/xclip installed, the Linux window backend now works through a built-in python-Xlib (EWMH/ICCCM) fallback - the system tools only make it more robust.

Manual steps:

  1. Python 3.11+ and system packages:

    sudo apt install python3 python3-venv python3-pip \
                     wmctrl xdotool xclip scrot \
                     tesseract-ocr at-spi2-core
  2. python3 -m venv .venv && ./.venv/bin/pip install -e ".[input,linux,ocr,vision,screenshot,vlm]"

  3. Run from a graphical session: ./.venv/bin/universal-computer-control (alias: ./.venv/bin/ucc)

AT-SPI requirements (accessibility): at-spi2-core must be running (it is on desktop distros) and for GNOME/GTK apps enable gsettings set org.gnome.desktop.interface toolkit-accessibility true.

X11 vs Wayland: X11 is fully supported. On Wayland-with-XWayland, input and screenshots work through XWayland but foreign window management may be restricted by the compositor. On pure Wayland (no DISPLAY), the window backend disables itself with a warning; AT-SPI still works for cooperating toolkits. See docs/TROUBLESHOOTING.md.

Registering the server with an MCP client

claude_desktop_config.json (see examples/mcp-config.example*.json):

{
  "mcpServers": {
    "universal-computer": {
      "command": "/absolute/path/to/universal-computer-control/.venv/bin/python",
      "args": ["-m", "universal_computer"],
      "env": {}
    }
  }
}

Any MCP client that speaks stdio works the same way. The engine is also a plain library — no MCP required:

from universal_computer import ComputerControlEngine, load_config

computer = ComputerControlEngine(load_config())
obs = await computer.observe("normal")
await computer.click("Continue")
await computer.type("hello")

Tools (MCP API)

Group

Tools

Observation

computer.observe (levels: minimal/normal/full) · computer.screenshot · computer.get_ui_tree · computer.get_active_window · computer.find_text · computer.find_visual · computer.wait_for

Mouse

computer.click · computer.double_click · computer.right_click · computer.move_mouse · computer.drag · computer.scroll

Keyboard

computer.type · computer.press · computer.hotkey

Windows

computer.list_windows · computer.focus_window · computer.minimize_window · computer.maximize_window · computer.restore_window · computer.close_window (needs confirm=true)

Clipboard

computer.get_clipboard · computer.set_clipboard

System

computer.launch_application (allowlist + confirm=true) · computer.run_command (allowlist + confirm=true)

Admin

computer.backend_status · computer.emergency_stop · computer.reset_emergency_stop

computer.observe returns a normalized snapshot:

{
  "ok": true,
  "id": "obs_17",
  "level": "normal",
  "screen": {"width": 1920, "height": 1080, "monitors": [...]},
  "active_window": {"title": "Google Chrome", "application": "chrome.exe", "process_id": 1234},
  "cursor": {"x": 812, "y": 530},
  "elements": [
    {"id": "element_1", "role": "button", "name": "Continue",
     "bbox": [1050, 700, 1170, 750], "source": "uia", "confidence": 0.99}
  ],
  "ocr": [
    {"text": "Continue", "bbox": [1050, 700, 1170, 750], "confidence": 0.97}
  ]
}

computer.backend_status reports per-backend health:

{
  "pyautogui": {"available": true, "healthy": true},
  "uia":       {"available": true, "healthy": true},
  "atspi":     {"available": false, "healthy": false,
                "error": "pyatspi import failed (...); install python3-pyatspi"},
  "ocr":       {"available": true, "healthy": true}
}

Configuration

config.yaml (see config.example.yaml) plus UCC_ environment overrides with __ nesting:

UCC_ENGINE__VERIFICATION_MODE=strict
UCC_SECURITY__MODE=standard
UCC_SECURITY__ALLOWED_COMMANDS='["ls", "git status", "xdotool"]'
UCC_LOGGING__LEVEL=DEBUG
UCC_VISION__VLM__ENABLED=true

Configurable: observation level & caching, verification mode (off/basic/strict), PyAutoGUI pauses/failsafe, OCR provider & language, vision threshold, template directory, VLM endpoint (key read from the environment variable named in vision.vlm.api_key_env), security mode, allowlists, rate limits, timeouts, persistence retention, log levels and per-backend priorities (backends.priorities: {pyautogui: 5}).

Security model

Mode

run_command

launch_application

permissive

allowed (still rate-limited); allow_shell=true runs via the shell

allowed (still rate-limited)

standard (default)

requires security.allowed_commands and confirm=true

allowed unless a non-empty allowed_applications says otherwise, and confirm=true

strict

only explicitly allowlisted; allow_shell never bypasses the allowlist

exact allowlist match required

Unquoted shell operators (;, |, &, `, <, >, $) only pass the allowlist when an entry matches the full command string — a bare program-name entry (git, python, ...) is not enough.

Always on: rate limiting (max_actions_per_minute), command timeouts, credential redaction in logs and command output, no clipboard/typed-text logging, close_window/run_command/launch_application confirmation gates, and the emergency stop — computer.emergency_stop() halts every subsequent action (checked before each attempt) and is persisted as a stop file so it survives restarts and can be triggered externally.

Extending

  • New OCR: implement universal_computer.vision.base.OCRProvider (detect_text(image) -> list[OCRResult]) and register it in VisionServices.from_config.

  • New vision model: implement VisionLanguageModel.locate_element (async, description → VisionResult with bbox). The OpenAI-compatible implementation shows the JSON contract.

  • New platform (macOS): subclass the backends.base ABCs (InputBackend, ScreenshotBackend, AccessibilityBackend, WindowManagementBackend, ClipboardBackend, ApplicationBackend) and register them in core/bootstrap.py. Nothing else changes — the engine only knows capabilities.

  • Template images: put PNGs in vision.template_dir; refer to them by name from computer.find_visual or as click targets.

Development

pip install -e ".[dev]"
pytest tests            # 160 unit + integration tests, no display needed
ruff check src tests

scripts/e2e_live_test.py is an end-to-end harness that boots its own Xvfb + openbox session, creates real X11 windows, and exercises every backend live (screenshots, OCR, window ops via both the tool paths and the python-Xlib fallback, OpenCV template matching, the VLM pipeline against a mock OpenAI-compatible endpoint, clipboard, input, launch, emergency stop):

python scripts/e2e_live_test.py             # full stack
python scripts/e2e_live_test.py --no-tools  # zero X11 tools: Xlib fallback

Unit tests run on any machine (OS-specific pieces are mocked). The integration test boots the real server over stdio. See docs/ARCHITECTURE.md for the module map and docs/TROUBLESHOOTING.md for platform-specific fixes.

Contributors & Acknowledgments

  • Z.ai (GLM) — co-developer: MCP server, cross-platform backends, vision pipeline, python-Xlib window fallback and ucc-doctor diagnostics

  • Claude Code (Anthropic) — co-developer: code generation, testing and plug-and-play hardening

  • @adkdev200 — creator & maintainer

This project is built collaboratively with AI pair-programming: the majority of the codebase was written and reviewed by Z.ai's GLM and Anthropic's Claude Code under the direction of the maintainer.

Available Tools

29 tools
computer.backend_statusA

Report health/availability of every backend (input, screenshot, accessibility, window management, clipboard, OCR, vision) plus security settings and emergency-stop state.

Inputs: no named parameters.

computer.clickA

Click a UI element. Target can be: visible text ('Continue'), an element id from computer.observe ('element_17'), or coordinates ({'x': 500, 'y': 300}). The engine resolves it via accessibility first, then OCR, then vision - and verifies the result.

Inputs: target (required), button, verify.

computer.close_windowA

Close a window gracefully (WM_CLOSE). Requires confirm=true; modal confirmation dialogs remain visible for the agent to handle.

Inputs: window (required), confirm.

computer.double_clickB

Double-click a UI element (same flexible target rules as computer.click).

Inputs: target (required), verify.

computer.dragC

Drag from one target to another (text, element ids or coordinates).

Inputs: end (required), start (required), duration_s.

computer.emergency_stopA

EMERGENCY STOP: immediately halt all automation (checked before every action and attempt). Persists until reset_emergency_stop.

Inputs: no named parameters.

computer.find_textA

Find text anywhere on screen (OCR + accessibility) with fuzzy matching. Returns matches with bounding boxes so you can click them. Tolerates case, arrows and small typos ('Continue' matches 'Continue →').

Inputs: query (required), limit, threshold.

computer.find_visualA

Find a UI element visually: by image template name (from the configured template directory) or by natural-language description via the optional vision model. Returns a bounding box.

Inputs: template, threshold, description.

computer.focus_windowA

Focus a real OS window by (partial) title or application name, e.g. 'Chrome'. Works without any application-specific integration.

Inputs: window (required).

computer.get_active_windowA

Get the title, application and process of the currently focused window.

Inputs: no named parameters.

computer.get_clipboardA

Read the clipboard text.

Inputs: no named parameters.

computer.get_ui_treeB

Get the accessibility tree of the active (or a named) window as a nested structure of roles, names and bounding boxes. Uses Windows UI Automation or Linux AT-SPI; returns an explanatory error when no accessibility backend is available.

Inputs: window, max_depth.

computer.hotkeyA

Press a key combination. Pass a string like 'ctrl+s' or a list like ['ctrl', 'shift', 't'].

Inputs: keys (required).

computer.launch_applicationA

Launch a GUI application ('chrome', 'code', a .desktop entry, or a full path). Subject to security.allowed_applications, rate limiting and security.confirmation_required (confirm=true by default).

Inputs: command (required), confirm.

computer.list_windowsA

List all top-level OS windows with title, application, pid and state.

Inputs: no named parameters.

computer.maximize_windowC

Maximize a window by title/application.

Inputs: window (required).

computer.minimize_windowC

Minimize a window by title/application.

Inputs: window (required).

computer.move_mouseB

Move the mouse cursor to a target (text, element id or coordinates).

Inputs: target (required).

computer.observeA

Observe the current computer state: screen geometry, active window, cursor, OCR'd on-screen text and accessibility UI elements. Levels: 'minimal' (fast), 'normal' (screenshot+OCR+accessibility), 'full' (adds focused element details). This is the primary tool for understanding the screen before acting.

Inputs: level.

computer.pressC

Press a keyboard key ('enter', 'tab', 'escape', 'f5', ...).

Inputs: key (required).

computer.reset_emergency_stopA

Clear a previously triggered emergency stop.

Inputs: no named parameters.

computer.restore_windowC

Restore a minimized/maximized window.

Inputs: window (required).

computer.right_clickC

Right-click a UI element (same flexible target rules as computer.click).

Inputs: target (required), verify.

computer.run_commandA

Run a shell command (security.allowlist applies; see computer.backend_status). Requires confirm=true. Output is truncated and credential-redacted.

Inputs: command (required), confirm, timeout_s.

computer.screenshotA

Take a screenshot of the whole virtual desktop, a specific monitor or a [left, top, right, bottom] region. Saves it to the persistence directory and returns the path.

Inputs: region, monitor, return_image.

computer.scrollA

Scroll the mouse wheel. Positive amount scrolls up, negative down. Optionally at specific coordinates (x, y).

Inputs: x, y, amount.

computer.set_clipboardA

Write text to the clipboard (useful as a typing fallback).

Inputs: text (required).

computer.typeA

Type text at the currently focused input. Handles Unicode and multiline text (clipboard fallback used automatically). The typed content is never logged.

Inputs: text (required), verify.

computer.wait_forA

Wait until something changes on screen. Expectations: 'any_change', 'window' (value = window title substring), 'text' (value = OCR text substring). Use after actions to await UI reactions instead of blind sleeps.

Inputs: value, timeout_s, expectation.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.7/5.0

    Scored across 29 tools

    Disambiguation4/5

    Most tools have clearly distinct purposes (click vs drag vs scroll, find_text vs find_visual, the many window operations are each specific). The only mild overlap is between computer.observe, computer.screenshot and computer.get_ui_tree, but observe's description explicitly positions it as the primary screen-understanding tool, keeping confusion low.

    Naming Consistency5/5

    Every tool follows a strict namespace.verb_noun snake_case pattern (computer.click, computer.focus_window, computer.list_windows, computer.emergency_stop). No camelCase or mixed conventions anywhere, so the surface is highly predictable.

    Tool Count4/5

    29 tools is on the high side, but the domain (full desktop automation spanning input, OCR/vision, accessibility, window management, clipboard, app launching and shell) legitimately needs this breadth and each tool maps to a distinct capability. Slightly heavy rather than gratuitous.

    Completeness5/5

    Coverage is comprehensive: observation/screenshot/UI-tree, the full input set (click variants, type, hotkey, scroll, drag), window lifecycle (list/focus/minimize/maximize/restore/close), clipboard read/write, app launch, shell execution, backend health, and safety controls (emergency stop/reset). No obvious dead ends for the stated purpose.

    Maintenance

    ActivitySlowing
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      C
      maintenance
      Cross-platform desktop automation MCP server that lets AI agents capture screenshots, run OCR with UI-element classification, control mouse/keyboard, and launch programs on Linux, macOS, and Windows.
      20
      2
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI agents to inspect and operate Windows applications through structured UI Automation maps, humanized mouse and keyboard input, and on-demand screenshots or OCR via MCP. It supports locating elements by role, name, or query; clicking, typing, scrolling, window management, event watching, and chained multi-step actions without relying on image analysis.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to observe and operate real desktops, mobile devices, and browser pages through screenshots, native accessibility trees, and web automation, including clicking, typing, hotkeys, and UI lookup across macOS, Linux, Windows, Android, and iOS.
      7
      Apache 2.0