computer-use-mcp

Give your coding agent hands. A free, open-source MCP server that lets a chat agent drive any Windows desktop app through window-relative, token-cheap tools - built for VS Code Chat agents, but it speaks plain MCP stdio and works with any host.

Vision is powered by the DeepSeek vision model. Screenshots never enter the model context by default; when the agent must actually see something, the look tool asks DeepSeek one narrow question (cached and opt-in - a key enables it, every other tool works without one).

Why it is different

Most "computer use" setups burn tokens on screenshots and guess absolute screen coordinates. This server inverts that:

• Window-relative coordinates, always. Every point is (x, y) inside the target window (0,0 = its top-left), in the same pixel space as the screenshots. No monitor arithmetic, no DPI surprises, no guessing which display is which.

• Screenshots are free. screenshot saves a PNG and returns a shot_id, the view size and a changed percentage. No image ever enters the model context unless look is called.

• Vision is opt-in, cached and DeepSeek-powered. look is the only costly tool: one narrow question to the DeepSeek vision model, optional region crop, results cached by image+question. Deterministic checks (read_pixel, read_text, ui_map) answer most questions for free.

• The server waits, not the model. wait_until blocks server-side on elements, window titles, pixels or a stable region - one tool call instead of a poll loop.

• Structured UI first, pixels last. ui_map / find_control / click_control / set_value drive named UI Automation controls; freehand strokes are the documented last resort.

• Safety rails on by default. Input is refused unless the target window is in front, every failure carries a stable error code and a suggested_recovery line, hold Ctrl+Alt+Q (or drop a STOP file) to abort, and every action is logged to logs/actions.jsonl.

Measured on DeepSeek's API, 60 vision requests cost on the order of $0.01 (≈44k tokens) - and a typical drawing session needs only ~5% of its calls to be vision; everything else is deterministic.

Related MCP server: windows-gui-mcp

What is in the box

The agent and the skills are part of the product: drop them into your project's .github/ folder and your own agent inherits the hard-won rituals (selection/floating-state handling, dialog flows, UIA-first routing, batching for speed).

See it in action: three Paint scenes drawn end-to-end by an agent with the bundled skill - demo 1 · demo 2 · demo 3.

Requirements

• Windows 10/11 x64 (the server uses Win32, GDI and UI Automation).

• Release ZIP: nothing else - computer-use-mcp.exe is self-contained.

• From source: Python 3.11+ (3.13 recommended).

• Optional: a DeepSeek API key for the look vision tool. Without it, look reports VISION_DISABLED and every other tool works unchanged.

Quick start

Option A - Windows release ZIP (recommended)

1. Download computer-use-mcp-<version>-windows-x64.zip from Releases and extract it anywhere.

2. Add the server to VS Code. Copy mcp.example.json into your project's .vscode/mcp.json (or merge the computer entry) and fix the two paths:

   {
     "servers": {
       "computer": {
         "type": "stdio",
         "command": "C:\\path\\to\\computer-use-mcp.exe",
         "envFile": "C:\\path\\to\\.env"
       }
     }
   }

3. Copy the agent and skills into your workspace so the chat agent can use them:

   xcopy /E /I <extracted>\.github\agents  .github\agents
   xcopy /E /I <extracted>\.github\skills  .github\skills

4. Restart VS Code, pick the computer-use agent in the chat picker, approve the computer/* tools (or set pre-approval via Chat: Manage Tool Approval), and ask:

   Target: Notepad. Type "Hello from the agent" into the document, then read it back and confirm.

Option B - from source

py -3.13 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt

.vscode/mcp.json in this repository already registers the server with the venv interpreter. In VS Code: MCP: List Servers → computer → Start, then use the computer-use agent.

Optional vision setup: copy .env.example to .env and fill in your key.

DEEPSEEK_API_KEY=sk-...

Verify the install without moving the mouse:

.\.venv\Scripts\python.exe scripts\smoke_test.py

Tools

Mutating tools answer with [outcome: executed|verified|partial|...] and, on failure, a suggested_recovery line (for example focus_target after a focus steal).

Cost ladder (cheapest question first)

1. read_pixel(x, y) - exact colour at a window-relative point (free).

2. read_text / ui_map - text and named controls without vision (free).

3. screenshot - changed: % versus the previous frame (free).

4. look - DeepSeek vision, only for "what is actually drawn" questions; narrow question, small region.

Agent and skills

The .github/ folder is the human side of the product:

• Agent (.github/agents/computer-use.agent.md): the operating manual. Least-fragile-route ladder, recon pass for unknown apps, dialog discipline, live/floating object commit rituals, batching for speed, and a symptom → action failure table. It pins the DeepSeek V4.1 Flash model for BYOK setups; edit the frontmatter to use a different model.

• Skills (.github/skills/<name>/SKILL.md): app-specific measured handbooks. The agent reads the matching skill before touching the GUI. Contributing one is only Markdown - see CONTRIBUTING.md.

How it works

flowchart LR
    A[Chat agent] -- "MCP stdio (JSON-RPC)" --> B[computer-use-mcp]
    B --> C[Win32: windows, GDI capture, input]
    B --> D[UI Automation: named controls]
    B -- "only look" --> E[DeepSeek vision - optional, cached]
    B --> F[(logs/actions.jsonl)]

• One synchronous action at a time: a middleware lock serializes tool calls so mouse/keyboard input never interleaves.

• The target window is re-resolved on every call (bounds, DPI, foreground state), so expected_shot_id can refuse an action when the screen changed under you.

• UI Automation runs on one long-lived COM thread (COM wrappers are finalized there - a Windows quirk that otherwise crashes long sessions).

Safety

The warning above is not boilerplate: this tool drives a real program with real data. Before the first longer run, make sure the target can tolerate mistakes (a scratch document, versioned files or backups), and stay present for the first sessions.

• Abort: hold Ctrl+Alt+Q during any action, or create a STOP file in the data directory (%LOCALAPPDATA%\computer-use-mcp for the packaged build, the repository root for source runs). Delete it to resume.

• Preflight: input is refused with ERROR[not_foreground] when the target window is not in front; app-owned dialogs and menus count as "in front".

• Bounds: points outside the target window are rejected; drags always release the mouse button, even mid-abort.

• Audit: every action is appended to logs/actions.jsonl.

• API key: lives only in your local .env (git-ignored, scanned in CI). Never commit it.

• Do not run the agent in autopilot/allow-all mode on a workstation with valuable data - use a VM or a dedicated session for untrusted content. See SECURITY.md.

Limits (v1)

• The capture copies a screen region: keep the target window visible (don't cover it).

• Elevated (UAC) windows cannot receive synthetic input (UIPI).

• One tool call at a time (synchronous); use run(steps) to batch known sequences.

• Single target window at a time; switching targets re-anchors all coordinates.

Contributing

This project is free and grows through contributions of two kinds - both are equally welcome:

• Build a skill for an app you know well. No server code required, just a SKILL.md with measured facts and a failure playbook. This is the highest-leverage contribution.

• Improve the server - new tools, better UI Automation coverage, reliability fixes. Plain Python, small surface, documented conventions.

See CONTRIBUTING.md for the development setup, the skill quality bar and the PR flow. Bug reports with a tool-call log excerpt from logs/actions.jsonl are gold.

Development and releases

# checks (also run in CI)
.\.venv\Scripts\python.exe -m ruff check .
.\.venv\Scripts\python.exe scripts\check_no_secrets.py
.\.venv\Scripts\python.exe scripts\mcp_handshake.py -- .\.venv\Scripts\python.exe server.py

# Windows release build (creates AND handshake-tests the executable)
.\build-windows-zip.ps1 -Clean

The version lives in version.py (single source of truth); releases are driven by a GitHub release: publishing v<version> runs the windows-zip-release workflow, which verifies the tag against version.py, builds the ZIP, attaches it to the release and confirms the asset. Maintainers: follow RELEASE-CHECKLIST.md.

server.py            MCP tools (mcp SDK: MCPServer)
actions.py           Windows primitives: windows, capture, input, UI Automation
vision.py            optional DeepSeek vision call with caching
version.py           package version (single source of truth)
scripts/             dev + release tooling (smoke tests, ZIP writer, secret scan)
.github/             agent, skills, workflows

Roadmap

• click_element(id) grounded on the latest vision analysis.

• PrintWindow-based capture for occluded windows.

• Macro recording: replayable, parameterized step sequences on top of run.

License

Apache-2.0. Free for personal and commercial use; contributions are accepted under the same terms.