Computer Control MCP (Enhanced)

Enhanced MCP server for full computer control: mouse, keyboard, screenshots, OCR, deep UI automation, semantic element actions, process management, filesystem watching, and accessibility-aware app launching. Built for AI agents that need to see, understand, and interact with desktop applications.

Enhanced fork of computer-control-mcp by AB498.

Quick Usage (MCP Setup Using uvx)

Running uvx computer-control-mcp-enhanced@latest for the first time will download Python dependencies (~70MB). Subsequent runs are instant.

{
  "mcpServers": {
    "computer-control": {
      "command": "uvx",
      "args": ["computer-control-mcp-enhanced@latest"]
    }
  }
}

Or install globally with pip:

pip install computer-control-mcp-enhanced
computer-control-mcp-enhanced

Related MCP server: AutoMac MCP

What's New (vs upstream)

This fork adds significant perception and automation capabilities for AI agents:

• UI Automation — Full Windows UI Automation (UIA) and Linux AT-SPI tree traversal with occlusion filtering, exposing interactive elements (buttons, text fields, menus) with absolute screen coordinates

• Deep UI Automation — Discover elements with stable refs, traverse the element tree (parent/child navigation), and perform semantic actions (toggle, select, invoke, set text/range values, expand/collapse, move/resize) plus native text manipulation (select by search/offset, cursor positioning, word/line queries, text bounding rectangles) — no coordinate math needed

• Combined perception (take_screenshot_full) — Image + OCR + UI automation in a single call with parallel execution, selectable via include_image/include_ocr/include_ui flags

• Region capture — All screenshot/OCR/UI tools accept a region=[x, y, w, h] parameter to capture arbitrary screen rectangles instead of full screen or full window

• Coordinate verification (capture_region_around) — Capture a small area around target coordinates with coordinate rulers showing real screen positions on edges, plus optional red circle marker. Agents read coordinates directly from rulers — no scale factor math needed, typically 1-2 steps instead of 5-6

• Wait & polling tools — wait_for_window (appear/disappear/active), wait_for_focused_element, wait_for_screen_change — synchronize with application state instead of blind delays

• Filesystem watching — Persistent directory watchers with event queues, or one-shot waits for file changes. Monitor builds, downloads, or any filesystem activity

• Process & system management — kill_process, list_processes, get_system_info — full process lifecycle and system diagnostics

• Accessibility-aware app launching (launch_app) — Launch apps with the right accessibility flags for maximum UI element exposure (e.g. --force-renderer-accessibility for Chromium, ACCESSIBILITY_ENABLED=1 for VS Code on Linux)

• Change detection — Pixel diff, OCR diff, and UI diff tools to detect what changed on screen between actions

• Screenshot optimization — Prescaling to agent-friendly sizes, WebP/JPEG compression, grayscale/BW modes for token savings

• Performance — Parallel OCR tiling for full-screen captures, app_filter to skip irrelevant window trees in UI automation (~18s to ~0.2s for targeted windows), role-based heuristics instead of expensive COM pattern queries

Features

• Full mouse control (click, move, drag, button hold)

• Keyboard input (type text, press keys, key combinations, hold keys)

• Screenshots of full screen, specific windows, or arbitrary regions

• OCR text extraction with absolute screen coordinates

• UI automation element detection (buttons, fields, menus, etc.)

• Deep UI automation with stable element refs, tree traversal, and semantic actions

• Semantic element actions: toggle, select, invoke, focus, expand/collapse, set text, set range value, move/resize

• Native text manipulation: select by offset or search, read selections, cursor positioning, word/line queries, text bounding rectangles

• Table/grid data reading with headers and paging support

• Programmatic container scrolling (page/line/percent) with scroll position queries

• View switching, virtualized item realization, drag info, and hyperlink navigation

• Window management (list, activate, wait for appear/disappear/active, fuzzy/regex matching)

• Process management (list, kill) and system diagnostics (CPU, memory, disk, OS)

• Filesystem watching (persistent watchers with event queues, one-shot file change waits)

• Screen change detection (pixel, OCR, and UI automation diffs)

• Accessibility-aware app launching for better UI element exposure

• Coordinate verification with ruler overlays and visual markers for precise clicking

• Image optimization (format, quality, color mode, prescaling)

• GPU-accelerated window capture via WGC (Windows only)

• Clipboard operations

• Action batching via perform_actions

Available Tools

Mouse Control

Keyboard Control

Screenshots & Perception

Screen Change Detection

Text Interaction

Cursor & Position

Window & App Management

Deep UI Automation — Discovery

Deep UI Automation — Semantic Actions

Deep UI Automation — Tables, Scrolling & Advanced

Process & System Management

Filesystem Watching

Utilities

Region Capture

All screenshot, OCR, UI automation, and change detection tools accept a region parameter:

# Capture just the top-left quadrant
take_screenshot_full(region=[0, 0, 960, 540])

# OCR only within a specific area
take_screenshot_with_ocr(region=[100, 200, 400, 300])

# Activate a window, then capture a sub-region of it
take_screenshot_full(title_pattern="Notepad", region=[1300, 280, 200, 100])

# Detect changes only within a region
check_screen_changed(region=[500, 500, 400, 300])

This lets an agent start with a full-screen capture, identify the area of interest, then focus subsequent captures on just that region to save tokens.

Coordinate Verification Workflow

AI agents often misjudge coordinates from prescaled screenshots. The capture_region_around tool solves this with coordinate rulers — real screen coordinates drawn directly on the image edges:

# Capture a region with coordinate rulers (enabled by default)
capture_region_around(x=935, y=630, radius=80, ruler_tick_interval=25)
# → Zoomed-in view with X coordinates on top edge, Y coordinates on left edge
# → Agent reads target's exact position from the rulers — no math needed
# → Click the coordinates directly

How it works:

1. Agent takes a full screenshot and roughly estimates the target area

2. Calls capture_region_around — rulers on edges show real screen coordinates

3. Agent reads the target's exact coordinates from the ruler grid lines

4. Clicks — typically accurate on the first attempt

Optional: add mark_center=True to also draw a red circle marker at the estimated position for visual confirmation before clicking.

This replaces the old multi-iteration guessing loop with a 1-2 step process.

Accessibility-Aware App Launching

The launch_app tool automatically applies the right accessibility flags per app family:

Example: Microsoft Edge exposes 90 UI elements normally vs 199 elements with the accessibility flag (2.2x increase).

# Launch Chrome with accessibility enabled
launch_app(command=["google-chrome", "https://example.com"])

# Preview what would happen without launching
launch_app(command=["code", "."], dry_run=True)

Deep UI Automation

Go beyond coordinate-based clicking — interact with UI elements semantically using stable element refs.

Discovery: Find elements by name, role, or text content across any window. Results include stable refs that can be passed directly to action tools.

# Find all toggle switches in Windows Settings
find_ui_elements(title_pattern="Settings", role_filter="push button", text_filter="Night light")

# Get the element under specific coordinates
get_element_at_point(x=500, y=300)

# Navigate the element tree
get_element_children(element_ref={...})
get_element_parent(element_ref={...})

Semantic Actions: Toggle switches, select tabs, fill text fields, adjust sliders — all without calculating coordinates.

# Toggle a switch
toggle_element(element_ref={...})

# Set a text field directly
set_element_text(element_ref={...}, text="Hello World")

# Adjust a slider
set_element_range_value(element_ref={...}, value=75)

# Select a tab or list item
select_element(element_ref={...})

# Move/resize a window via UI automation
set_element_extents(element_ref={...}, x=100, y=100, width=800, height=600)

Text Manipulation: Select, navigate, and query text within document/edit elements — no keyboard simulation needed.

# Find and select a substring
select_text_by_search(element_ref={...}, search_text="brown fox")

# Select by character offsets
select_text_range(element_ref={...}, start=4, end=19)

# Read what's currently selected
get_text_selection(element_ref={...})  # → {"selections": [{"text": "brown fox"}]}

# Get/set cursor position
get_text_caret_offset(element_ref={...})  # → {"offset": 10}
set_text_caret_offset(element_ref={...}, offset=50)

# Get word or line at a position
get_text_at_offset(element_ref={...}, offset=4, unit="word")  # → {"text": "quick "}

# Get screen coordinates for a text range
get_text_bounds(element_ref={...}, start=4, end=19)  # → bounding rectangles

Element refs survive minor UI changes (scrolling, focus shifts) but need re-discovery if the window title changes or the element tree restructures.

Built-in Agent Guide

The MCP includes a built-in skill guide that any AI agent can retrieve at runtime:

get_agent_guide()
# → {"guide_path": "/path/to/AGENT_GUIDE.md", "message": "Read the file at guide_path..."}

Returns the file path to a comprehensive markdown guide (~50KB). The agent reads the file at that path to learn best practices, tool selection, workflows, and troubleshooting for all 85 tools. No external documentation or agent skill files needed — the guide ships with the pip package.

Filesystem Watching

Monitor directories for file changes — useful for watching build output, downloads, or log files.

# Persistent watcher
start_file_watch(paths="C:/project/dist")
# ... trigger a build ...
get_file_watch_events(watch_id="...")  # → created, modified, deleted events
stop_file_watch(watch_id="...")

# One-shot wait
wait_for_file_change(paths="C:/Users/me/Downloads", timeout_ms=30000)

Requires the watchdog Python library (pip install watchdog).

Configuration

Custom Screenshot Directory

{
  "mcpServers": {
    "computer-control": {
      "command": "uvx",
      "args": ["computer-control-mcp-enhanced@latest"],
      "env": {
        "COMPUTER_CONTROL_MCP_SCREENSHOT_DIR": "C:\\Users\\YourName\\Pictures\\Screenshots"
      }
    }
  }
}

Automatic WGC for Specific Windows

For GPU-accelerated windows that render black with standard capture:

{
  "env": {
    "COMPUTER_CONTROL_MCP_WGC_PATTERNS": "obs, discord, game, steam"
  }
}

Development

# Clone
git clone https://github.com/gzmagyari/computer-control-mcp.git
cd computer-control-mcp

# Install in dev mode (edits reflect immediately)
pip install -e .

# Run the server
computer-control-mcp-enhanced

# Run tests
python -m pytest

# Build
pip install hatch
hatch build

License

MIT

Based on computer-control-mcp by AB498.