Computer Use MCP

See the desktop. Understand the interface. Act with evidence.

agent-computer-use-mcp gives an MCP Host a compact, observable way to operate local Windows applications. It combines semantic accessibility, secure screenshot delivery, local PP-OCRv6, reliable Unicode input, and explicit action verification behind a session-bound control lease.

Download the latest release · Read the changelog · Contribute

Preview: 0.x targets Windows x64 and is evolving quickly. Actions remain Host-approved, application-visible, and fail closed when evidence is insufficient.

From intent to verified action

Natural-language task
        ↓
Host selects Computer Use from its semantic capability description
        ↓
Acquire a session-bound desktop control lease
        ↓
Observe: accessibility first → OCR → secure image understanding when needed
        ↓
Act through a semantic element, focus receipt, or observation-bound coordinate
        ↓
Verify the outcome and release control on success, failure, or cancellation

The Agent sees four focused tools:

Four additional management tools—health, doctor, installation, and repair—stay with the Host and are not projected into the Agent's normal tool inventory.

Related MCP server: ClawdCursor

Why this implementation

• Semantic first, pixels when necessary. Accessible controls are faster and more reliable than guessed coordinates. OCR handles low-latency text; image understanding is reserved for layouts, icons, and complex scenes.

• Coordinates carry provenance. Screenshot dimensions, native-window scale, truncation, focus, observation identity, and expiry travel with the result.

• Actions explain themselves. Every successful mutation reports the target path, provider path, delivery mode, and any fallback reason.

• Chinese input is a first-class path. Coordinate-grounded Unicode entry uses verified native delivery, while semantic fields remain on the driver path.

• Screenshots are assets, not file paths. The Host receives controlled MCP image content; the Agent never needs access to a temporary local file.

• Control has a lifecycle. Leases are bound to the requesting Host session and are revoked on completion, cancellation, timeout, disconnect, or process shutdown.

• No hidden self-update. Startup is offline. Installation, upgrade, downgrade, and rollback are explicit Host or operator actions.

Install

XiaozhiClaw

Open Plugins → Connectors, install Computer Use, review the requested desktop permissions, and enable it. Health, diagnostics, installation details, and repair planning are available from the connector's settings panel.

Any standard MCP Host

Download agent-computer-use-mcp-<version>-win32-x64.tar.gz from the latest GitHub Release, verify it against SHA256SUMS.txt, and extract it. The executable is:

artifact/bin/agent-computer-use-mcp.exe

Example MCP configuration:

{
  "mcpServers": {
    "computer-use": {
      "command": "C:\\Tools\\agent-computer-use-mcp\\artifact\\bin\\agent-computer-use-mcp.exe",
      "args": []
    }
  }
}

The release contains the exact Windows x64 driver, native overlay, ONNX runtime, PP-OCRv6 model pack, inventory, checksums, licenses, and SBOM used by CI. It requires no npm install and performs no network access at startup.

Safety model

Computer Use is intentionally visible and bounded:

• the Host owns approval, process lifecycle, installation, and media assets;

• the Agent cannot acquire another Host session's lease;

• password, payment, credential, private, and denied-window policies fail closed;

• the branded frame and cursor are excluded from screenshots, OCR, observations, traces, and benchmark artifacts;

• uncertain mutations are never blindly replayed—they require a fresh observation;

• repair reports a plan and never downloads or installs implicitly.

Develop

Requirements: Windows x64, Node.js 24, npm 11, and .NET 10 SDK.

git clone https://github.com/astraclawteam/agent-computer-use-mcp.git
cd agent-computer-use-mcp
npm ci
npm test
npm run mcp

Protocol and packaging gates:

npm run phase:1.6
npm run phase:1.7
npm run phase:1.8
npm run artifact:windows:build -- --allow-network

Real desktop action gate:

npm run phase:1.4

The full test suite is deliberately serial. It covers MCP interoperability, leases, policy, perception, Unicode delivery, cleanup races, overlay exclusion, and immutable Windows artifact assembly.

Release

A vX.Y.Z tag must match package.json, point to a commit reachable from main, and have a matching CHANGELOG.md section. The release workflow then:

1. runs the full test and official MCP SDK gates;

2. assembles and verifies the Windows x64 SEA artifact;

3. writes SHA256SUMS.txt;

4. creates the GitHub Release and marks it Latest.

The workflow has no npm or Gitee credentials and publishes only verified GitHub Release assets. See the release pipeline specification and release gates.

Contributing

Real application edge cases are especially valuable: custom-drawn controls, tray restoration, high-DPI scaling, multilingual input, focus transitions, and post-action verification. Start with CONTRIBUTING.md, or open an app smoke report.

Security-sensitive findings should use GitHub private vulnerability reporting.

License

MIT