desktop-control-mcp

A Windows desktop automation server that exposes mouse, keyboard, screen capture, and AI-powered UI element detection through two interfaces:

• MCP server (mcp_server.py) — stdio transport, designed for use with Claude Code, Claude Desktop, and other MCP clients

• HTTP server (server.py + app.py) — a Flask API on http://localhost:7845, runnable from a system tray icon

Both interfaces share the same underlying controller.py (Windows automation) and ui_parser.py (vision models), so capabilities are identical.

Highlights

• AI vision-based clicking — UI elements are detected with Microsoft OmniParser v2 (YOLO icon detector + Florence-2 captioner + EasyOCR), so you can target buttons/fields by intent rather than by guessing pixel coordinates from a screenshot.

• DPI-aware — runs as PROCESS_PER_MONITOR_DPI_AWARE, so coordinates are always physical pixels regardless of Windows display scaling.

• Cursor in screenshots — the real Windows cursor bitmap is composited into every screenshot via the Win32 GDI API, so AI clients can see where the mouse is.

• Three screenshot modes — single compressed JPEG, annotated detection view, and time-spread burst capture (for observing animations or loading states).

• Smart keyboard dispatch — separate tools for literal text, single keys, and modifier combos so "ctrl+c" is never accidentally typed as text.

Related MCP server: desktop-automation-mcp

Installation

Requires Windows and Python 3.10+.

git clone https://github.com/ahmetdenizyilmaz/desktop-control-mcp.git
cd desktop-control-mcp
pip install -r requirements.txt

On first run, OmniParser v2 model weights (~1 GB) are downloaded from Hugging Face into the local cache. GPU is used automatically if a CUDA-enabled PyTorch is installed; otherwise CPU is used.

Running

As an MCP server (stdio)

Add an entry like this to your MCP client config (e.g. Claude Desktop / Claude Code):

{
  "mcpServers": {
    "desktop-control": {
      "command": "python",
      "args": ["C:\\path\\to\\desktop-control-mcp\\mcp_server.py"]
    }
  }
}

The server boots immediately; models load in a background thread so the first detect_ui_elements call waits for them but the rest of the tools are available right away.

As an HTTP server with tray icon

python app.py

A tray icon appears with Start / Stop / Quit. The Flask API listens on http://localhost:7845. See API_DOCS.md for endpoint details.

MCP tools

Screen info

Screenshots

Mouse

Keyboard

The mandatory click workflow

Coordinates are not stable across screen changes — every menu opens, every dialog appears, every scroll moves things. The server enforces this loop:

1. DETECT — detect_ui_elements returns the current element table.

2. FIND — match your target in the Label column.

3. CLICK — click_mouse(x, y) using the exact Center coordinates from the table.

4. VERIFY — take_screenshot to confirm the click landed.

5. RECOVER — if the click missed, re-detect (the screen may have changed) and retry.

Never guess coordinates from a raw screenshot. The element table from detect_ui_elements is the source of truth.

Example element table

  ID  Type   Conf   Center         Label
   1  text   0.98   ( 499,   55)   File  Edit  View
   2  icon   0.91   ( 674,  200)   Search button
   3  icon   0.87   (1200,  400)   Close

To click the search button: click_mouse(x=674, y=200).

HTTP API

When run as app.py, the Flask server on port 7845 mirrors the MCP tools:

Full request/response schemas are in API_DOCS.md.

File layout

mcp_server.py          FastMCP server — tool definitions and lifespan
controller.py          Win32 / pyautogui / mss core — screenshots, input, window info
ui_parser.py           OmniParser v2 loading + UI detection + image annotation
screenshot_manager.py  Disk-side screenshot cache (cleanup, naming, burst dirs)
overlay.py             On-screen overlay utilities
lock_manager.py        Concurrency lock for shared resources
server.py              Flask HTTP API
app.py                 Tray-icon launcher for the Flask server
templates/index.html   Web UI for the Flask server
requirements.txt       Python dependencies
API_DOCS.md            HTTP endpoint reference
run_agents.bat         Convenience launcher for Claude Code in this folder

Platform notes

• Windows only. The cursor compositing, DPI awareness, and active-window code use Win32 APIs directly.

• Primary monitor only. All coordinates are in the primary monitor's pixel space.

• First detection is slow. OmniParser model load + EasyOCR initialization can take 30–60 seconds on first call. Subsequent detections are fast (sub-second on GPU, a few seconds on CPU).