computer-use-mcp

Let AI drive Windows desktop apps — UI Automation accessibility trees + SendInput + window capture

An MCP server that lets a model see structure and click precisely.

English · 中文

What it is

What makes Codex's computer use far more reliable than "screenshot + guess coordinates" is that it hands the model a UI Automation accessibility tree:

0 Window "Untitled - Notepad"
	1 MenuBar "Application"
		2 MenuItem "File" actions=[Expand]
		3 MenuItem "Edit" actions=[Expand]
	4 Edit "Text Editor" value="" focusable FOCUSED actions=[SetValue, SelectText]
	5 StatusBar "Ln 1, Col 1"

The model sees an indexed list of controls, not a picture it has to guess coordinates from. It can say "click element 2" instead of "click (412, 37) and hope that's the File menu".

This project is a clean-room reimplementation of the Codex computer-use tool surface (@oai/sky + the codex-computer-use native component). Only the publicly observable API semantics were studied; all code here is independently written and contains no OpenAI source, binary, or asset. The native layer is built on the .NET Framework UI Automation stack.

Related MCP server: win-nav

Features

• Accessibility-first: every control gets a stable index, so actions survive layout tweaks

• Captures occluded windows: tries PrintWindow first, falls back to activating and grabbing the screen

• Coordinate safety: screenshots carry a screenshot_id; coordinates are rescaled and re-based automatically, and a click that would land on the wrong window fails loudly instead of silently

• Unicode input: KEYEVENTF_UNICODE, so CJK and emoji work regardless of keyboard layout

• Zero third-party native dependencies: only assemblies shipped with .NET Framework — no NuGet, no Rust/C++ toolchain

• Testable: ships a WinForms test target with a known control layout and 30 end-to-end assertions

Architecture

┌─────────────────────────────┐
│  MCP client                  │   DeepSeek Harness / Claude Desktop / Cursor ...
└──────────────┬──────────────┘
               │ MCP over stdio (JSON-RPC)
┌──────────────▼──────────────┐
│  MCP server (Node/TypeScript)│   src/ — tool schemas, arg mapping, image encoding
└──────────────┬──────────────┘
               │ NDJSON over a child-process pipe
┌──────────────▼──────────────┐
│  Native helper (C#, .NET FW) │   native/windows/ — cu-helper.exe
│  ├ UI Automation → AX tree   │
│  ├ SendInput     → mouse/key │
│  ├ PrintWindow / → capture   │
│  │  BitBlt                   │
│  └ DWM           → geometry  │
└──────────────┬──────────────┘
               │ Win32
        ┌──────▼──────┐
        │ Target apps │
        └─────────────┘

Why two processes:

• UI Automation is a COM API and SendInput needs exact struct layouts — both are available dependency-free on .NET Framework, and awkward or impossible from Node.

• Node owns the MCP protocol and tool orchestration, where its ecosystem is strongest.

• The process boundary decouples the two: a helper crash yields a readable error and a restart instead of taking down the server.

Install

Platform support

The MCP layer and all tool definitions are platform-agnostic: a backend only has to ship an executable under native/<platform>/bin/ that speaks the same NDJSON protocol. Adding one requires no changes to src/. See architecture.md § 11.

Requirements (Windows backend)

Build

git clone <this-repo>
cd computer-use-mcp

npm install
npm run build          # TypeScript -> dist/
npm run build:native   # -> native/windows/bin/cu-helper.exe

Verify the native side:

node dist/index.js --check
# helper: ...\native\windows\bin\cu-helper.exe
# ping: ok (dpi awareness = 2)
# targetable windows: 5

DeepSeek Harness

DSH reads MCP servers from the cordis profile patch layer. Append this to %USERPROFILE%\.dsh\profiles\desktop\cordis.patch.yml:

- insert:
    - id: mcp-computer-use
      name: '@deepseek-ai/dsh-mcp-client'
      config:
        serverName: computer_use
        transport: stdio
        command: 'node'
        args:
          - 'C:\path\to\computer-use-mcp\dist\index.js'
        env:
          CU_HELPER_CACHE_DIR: 'C:\path\to\cache\computer-use'
        toolCallTimeoutMs: 60000
        failOnStartupError: false

Notes:

• The - insert: wrapper is required. A bare top-level id is treated as an override of an existing entry and silently ignored.

• serverName must match [A-Za-z0-9_-]; tools surface as mcp__computer_use__<tool>.

• No restart needed — the patch file is hot-reloaded.

Other MCP clients

{
  "mcpServers": {
    "computer-use": {
      "command": "node",
      "args": ["C:\\path\\to\\computer-use-mcp\\dist\\index.js"],
      "env": { "CU_HELPER_CACHE_DIR": "C:\\path\\to\\cache\\computer-use" }
    }
  }
}

Tools

Typical loop

1. list_apps                    -> pick the app and window_id
2. get_window_state(window_id)  -> indexed tree + screenshot
3. click(element_index=12)      -> index-based, layout-proof
4. get_window_state(window_id)  -> verify the effect

get_window_state is an expensive point-in-time snapshot: batch related actions, then capture again. Element indexes are valid only for the snapshot that produced them.

How it works

Tree and element indexes

get_window_state walks the window with UI Automation's ControlView, skips offscreen elements, assigns each visible element an incrementing index, and caches index → AutomationElement. A later click(element_index=N) resolves the cached element, reads its screen rectangle, and clicks its centre — no coordinates involved.

Line format:

<tab indent><index> <ControlType> "<name>" value="<value>" <state flags> actions=[<actions>]

State flags: disabled / focusable / FOCUSED / required. The actions list is exactly what perform_secondary_action accepts for that element.

Coordinates and scaling

Screenshots are scaled so the longest edge is at most 1600px; the returned scale is image pixels / physical pixels. Coordinates the model reads off the image are image pixels — the helper divides by scale and adds the window origin. Passing screenshot_id verifies the coordinates came from the current screenshot of that window.

Input safety

Before every input the helper:

1. brings the target window to the foreground (AttachThreadInput + SetForegroundWindow, which defeats the foreground lock);

2. verifies with WindowFromPoint that the point really belongs to the target — if a Start menu, overlay, or modal covers it, it returns point_obstructed instead of clicking the wrong thing;

3. rejects coordinates outside the window with point_outside_window.

Troubleshooting

Development

npm run build          # TypeScript
npm run build:native   # native helper
npm run typecheck

npm run test:mcp       # MCP protocol layer (15 tests, no GUI needed)
npm run test:e2e       # end-to-end GUI (30 assertions; briefly steals focus)

tests/e2e.ps1 compiles and launches tests/fixture/TestTarget.cs — a WinForms window with a fixed control layout — then performs real clicks, typing, scrolling, and error-path assertions.

The helper also self-checks standalone:

native\windows\bin\cu-helper.exe --selftest

Relationship to Codex computer use

This is an independent implementation. It does not include or redistribute any OpenAI code, binary, or asset. Reverse engineering was limited to understanding publicly observable tool semantics (tool names, parameter meanings, how indexes and screenshots behave) so that models can work in the same interaction pattern. The native layer is written against Microsoft's public UI Automation / SendInput / DWM APIs.

Documentation

• Architecture and design decisions — the trade-offs and why they were made

• Reverse-engineering report — Codex computer use internals, its tool surface and native protocol, and how this project differs

License

MIT