desktop-control-mcp

Claude Desktop Controller MCP

A Windows desktop automation server that exposes mouse, keyboard, screen capture, and AI-powered UI element detection through two interfaces:

MCP server (mcp_server.py) — stdio transport, designed for use with Claude Code, Claude Desktop, and other MCP clients
HTTP server (server.py + app.py) — a Flask API on http://localhost:7845, runnable from a system tray icon

Both interfaces share the same underlying controller.py (Windows automation) and ui_parser.py (vision models), so capabilities are identical.

Highlights

AI vision-based clicking — UI elements are detected with Microsoft OmniParser v2 (YOLO icon detector + Florence-2 captioner + EasyOCR), so you can target buttons/fields by intent rather than by guessing pixel coordinates from a screenshot.
DPI-aware — runs as PROCESS_PER_MONITOR_DPI_AWARE, so coordinates are always physical pixels regardless of Windows display scaling.
Cursor in screenshots — the real Windows cursor bitmap is composited into every screenshot via the Win32 GDI API, so AI clients can see where the mouse is.
Three screenshot modes — single compressed JPEG, annotated detection view, and time-spread burst capture (for observing animations or loading states).
Smart keyboard dispatch — separate tools for literal text, single keys, and modifier combos so "ctrl+c" is never accidentally typed as text.

Installation

Requires Windows and Python 3.10+.

git clone https://github.com/ahmetdenizyilmaz/desktop-control-mcp.git
cd desktop-control-mcp
pip install -r requirements.txt

On first run, OmniParser v2 model weights (~1 GB) are downloaded from Hugging Face into the local cache. GPU is used automatically if a CUDA-enabled PyTorch is installed; otherwise CPU is used.

Running

As an MCP server (stdio)

Add an entry like this to your MCP client config (e.g. Claude Desktop / Claude Code):

{
  "mcpServers": {
    "desktop-control": {
      "command": "python",
      "args": ["C:\\path\\to\\desktop-control-mcp\\mcp_server.py"]
    }
  }
}

The server boots immediately; models load in a background thread so the first detect_ui_elements call waits for them but the rest of the tools are available right away.

As an HTTP server with tray icon

python app.py

A tray icon appears with Start / Stop / Quit. The Flask API listens on http://localhost:7845. See API_DOCS.md for endpoint details.

MCP tools

Screen info

Tool	Description
`get_screen_size()`	Primary monitor resolution. Call once to learn the coordinate space.
`get_active_window()`	Title, position, and size of the currently focused window.

Screenshots

Tool	Description
`take_screenshot(quality=30)`	Compressed JPEG of the primary monitor. For observation only — do not derive coordinates from it.
`detect_ui_elements(confidence_threshold=0.5, full_response=False)`	Runs OmniParser + EasyOCR, returns an annotated image plus a table of `(id, type, confidence, center, label)` for every detected element. This is how you find coordinates to click.
`take_screenshot_burst(frame_count=10, duration_seconds=1.0)`	Captures N frames evenly spread over a duration. Useful for animations, loading spinners, or timing-sensitive UI.

Mouse

Tool	Description
`click_mouse(x, y, button="left")`	Single click. Always use coordinates from `detect_ui_elements`.
`double_click(x, y)`	Double-click.
`move_mouse(x, y)`	Move cursor without clicking.
`drag_mouse(x1, y1, x2, y2, button="left")`	Click-and-drag.
`scroll(x, y, direction, amount=3)`	Wheel scroll at a position (`up`/`down`/`left`/`right`).

Keyboard

Tool	When to use
`type_text(text)`	Plain text input (filenames, search queries, URLs, code). Never interprets `+` as a hotkey.
`press_key(key, presses=1)`	A single named key (`enter`, `tab`, `escape`, `f5`, arrow keys, `delete`, etc.).
`send_keys(text)`	Modifier+key combos only (`ctrl+c`, `alt+tab`, `win+d`, `ctrl+shift+s`).

The mandatory click workflow

Coordinates are not stable across screen changes — every menu opens, every dialog appears, every scroll moves things. The server enforces this loop:

DETECT — detect_ui_elements returns the current element table.
FIND — match your target in the Label column.
CLICK — click_mouse(x, y) using the exact Center coordinates from the table.
VERIFY — take_screenshot to confirm the click landed.
RECOVER — if the click missed, re-detect (the screen may have changed) and retry.

Never guess coordinates from a raw screenshot. The element table from detect_ui_elements is the source of truth.

Example element table

  ID  Type   Conf   Center         Label
   1  text   0.98   ( 499,   55)   File  Edit  View
   2  icon   0.91   ( 674,  200)   Search button
   3  icon   0.87   (1200,  400)   Close

To click the search button: click_mouse(x=674, y=200).

HTTP API

When run as app.py, the Flask server on port 7845 mirrors the MCP tools:

Endpoint	Purpose
`GET /health`	Liveness check.
`POST /screenshot`	Returns a base64 PNG of the screen.
`POST /move`	`{x, y}` — move cursor.
`POST /click`	`{x, y, button?}` — click.
`POST /keys`	`{keys}` — text or hotkey (auto-detected).
`POST /command`	Single bracket command like `[ClickMouse(500,300)]`.
`POST /actions`	List of bracket commands run sequentially with an optional `delay` between them.

Full request/response schemas are in API_DOCS.md.

File layout

mcp_server.py          FastMCP server — tool definitions and lifespan
controller.py          Win32 / pyautogui / mss core — screenshots, input, window info
ui_parser.py           OmniParser v2 loading + UI detection + image annotation
screenshot_manager.py  Disk-side screenshot cache (cleanup, naming, burst dirs)
overlay.py             On-screen overlay utilities
lock_manager.py        Concurrency lock for shared resources
server.py              Flask HTTP API
app.py                 Tray-icon launcher for the Flask server
templates/index.html   Web UI for the Flask server
requirements.txt       Python dependencies
API_DOCS.md            HTTP endpoint reference
run_agents.bat         Convenience launcher for Claude Code in this folder

Platform notes

Windows only. The cursor compositing, DPI awareness, and active-window code use Win32 APIs directly.
Primary monitor only. All coordinates are in the primary monitor's pixel space.
First detection is slow. OmniParser model load + EasyOCR initialization can take 30–60 seconds on first call. Subsequent detections are fast (sub-second on GPU, a few seconds on CPU).