Computer Use MCP

See the desktop. Understand the interface. Act with evidence.

agent-computer-use-mcp gives an MCP Host a compact, observable way to operate local Windows applications. It combines semantic accessibility, secure screenshot delivery, local PP-OCRv6, reliable Unicode input, and explicit action verification behind a session-bound control lease.

Download the latest release · Read the changelog · Contribute

Preview: 0.x targets Windows x64 and is evolving quickly. Actions remain Host-approved, application-visible, and fail closed when evidence is insufficient.

From intent to verified action

Natural-language task
        ↓
Host selects Computer Use from its semantic capability description
        ↓
Acquire a session-bound desktop control lease
        ↓
Observe: accessibility first → OCR → secure image understanding when needed
        ↓
Act through a semantic element, focus receipt, or observation-bound coordinate
        ↓
Verify the outcome and release control on success, failure, or cancellation

The Agent sees two task-level tools:

The lifecycle tools (computer.acquire, computer.observe, computer.act, and computer.release) are workflow-internal. Health, doctor, installation, and repair are also Host-only.

That boundary is carried by the advertised MCP contract, not by a vendor convention. tools/list returns exactly computer.task and computer.message unless the process was launched in host mode, and tools/call rejects every name the surface did not advertise with the same error as a tool that does not exist. A standard MCP Host therefore enforces the boundary without knowing anything about this implementation:

default launch          → computer.task, computer.message
--tool-surface=host     → the full ten-tool Host surface

The default surface is not a reduced desktop engine. computer.task and the Host lifecycle tools consume the same versioned Scene, action implementation, and receipts; the default surface only keeps targeting and controller lifecycle inside the MCP server. A desktop control that is editable through Host mode must therefore also be offered as an opaque edit candidate through computer.task.

Host mode is selected only from launch arguments or the AGENT_COMPUTER_USE_TOOL_SURFACE environment variable, both owned by whoever spawns the process. Nothing the Agent sends over the transport can widen its own inventory. An unrecognized surface value fails at startup rather than quietly falling back.

Related MCP server: lowlevel-computer-use-mcp

Why this implementation

• Semantic first, pixels when necessary. Accessible controls are faster and more reliable than guessed coordinates. OCR handles low-latency text; image understanding is reserved for layouts, icons, and complex scenes.

• Coordinates carry provenance. Screenshot dimensions, native-window scale, truncation, focus, observation identity, and expiry travel with the result.

• Actions prove their postconditions. Navigation receipts expose the provider outcome and the newer Host Scene evidence that confirmed the target state, destination surface, or owned transient. Provider status alone cannot turn an unobserved click into success. When the first post-action Scene is inconclusive, the Host may perform one read-only related-surface capture; it never replays the click.

• Editing never implies submission. computer.task exposes a proven Editable as one atomic replace-all action, grounds it in the current screenshot, and commits only after exact-value read-back. It never exposes a send/submit control or presses a commit key. Contact selection, conversation verification, sending, and new-bubble proof remain exclusive to computer.message.

• Navigation delivery is grounded before mutation. For a Host-selected navigation control with current same-window semantic bounds, the Host chooses one foreground pointer delivery inside that owned rectangle. It does not invoke first and then retry by coordinate. The first post-action screenshot carries the control anchor; an in-window popup is merged into the Scene only when its OCR rows agree with an anchor-local changed region or an independently detected flat pixel surface. An already-open popup can therefore expose its owned navigation items without replaying its toggle.

• Chinese input is a first-class path. Coordinate-grounded Unicode entry uses verified native delivery, while semantic fields remain on the driver path.

• Screenshots are assets, not file paths. The Host receives controlled MCP image content; the Agent never needs access to a temporary local file.

• Control has a lifecycle. Leases are bound to the requesting Host session and are revoked on completion, cancellation, timeout, disconnect, or process shutdown.

• Task routing cannot escape the Host. A live generic task result carrying an opaque task token limits the next interaction step to computer.task; terminal results expose no continuation control. Shell, raw targeting, and lifecycle tools cannot replace an unresolved Host candidate. Same-package child processes inherit only their proven owning-window process identity. A navigation click whose Scene postcondition is not proven closes without replay, even if the provider reported delivery. Once issued, the opaque task token owns continuation scope; the model does not restate or paraphrase the application and goal on later steps. A mixed-layout application may contain a chat-like pane while the goal is ordinary navigation; computer.task may expose its proven editor only as a non-submitting edit candidate while hiding send controls, conversation targets, transcript, and descendants. Only an explicit recipient-plus-send request uses computer.message.

• Browser-native work keeps its native path. Chrome or Chromium web-page interaction belongs to a dedicated browser capability such as agent-browser when the Host provides one, and XiaozhiClaw's in-app preview belongs to preview_browser. Computer Use remains the visible native-desktop and operating-system UI path, plus the fallback when a browser-native path cannot complete the interaction.

• No hidden self-update. Startup is offline. Installation, upgrade, downgrade, and rollback are explicit Host or operator actions.

Install

XiaozhiClaw

Open Plugins → Connectors, install Computer Use, review the requested desktop permissions, and enable it. Health, diagnostics, installation details, and repair planning are available from the connector's settings panel.

Any standard MCP Host

Download agent-computer-use-mcp-<version>-win32-x64.tar.gz from the latest GitHub Release, verify it against SHA256SUMS.txt, and extract it. The executable is:

artifact/bin/agent-computer-use-mcp.exe

Example MCP configuration:

{
  "mcpServers": {
    "computer-use": {
      "command": "C:\\Tools\\agent-computer-use-mcp\\artifact\\bin\\agent-computer-use-mcp.exe",
      "args": []
    }
  }
}

That configuration gets the Agent surface: two task-level tools, no lifecycle or management tools in the model's inventory. Only add "args": ["--tool-surface=host"] if your Host itself owns user approval, controller lifecycle, and tool projection — it hands the model eight more tools, including direct desktop mutation and repair.

For the default Agent surface, start computer.task with applicationName and goal. Choose a returned opaque candidateId; when that candidate declares inputRequired: true, provide the exact text in the task-level text field. relevance: "route" describes a reversible path toward a target that is not yet visible; it is not an action name. Element ids, bounds, observations, and surface receipts intentionally remain Host-owned.

For a Host integration, calling computer.acquire without a selector returns fresh application and window candidates without starting control. The next call accepts exactly one selector: applicationToken restores the primary window, windowId selects that exact observed window, applicationName resolves one exact current product name, and target: "foreground" selects the current OS foreground window. Combining selectors returns window.selector_conflict with the selector meanings; the retired partial-title selector is rejected. computer.act accepts one canonical action object and uses action.value for set_value and type_text; task-level text and nested action.action envelopes are not aliases. A returned surfaceReceipt.id optionally binds an action to that latest single-use surface and expires when a new observation replaces it, the action consumes it, or the lease ends.

The release contains the exact Windows x64 driver, native overlay, ONNX runtime, PP-OCRv6 model pack, inventory, checksums, licenses, and SBOM used by CI. It requires no npm install and performs no network access at startup.

Safety model

Computer Use is intentionally visible and bounded:

• the Host owns approval, process lifecycle, installation, and media assets;

• the tool surface is chosen at launch and defaults to the Agent surface, so an unconfigured Host cannot expose lifecycle or management tools by accident;

• the Agent cannot acquire another Host session's lease;

• password, payment, credential, private, and denied-window policies fail closed;

• the branded frame and cursor are excluded from screenshots, OCR, observations, traces, and benchmark artifacts;

• uncertain mutations are never blindly replayed—they require a fresh observation;

• repair reports a plan and never downloads or installs implicitly.

Develop

Requirements: Windows x64, Node.js 24, npm 11, and .NET 10 SDK.

git clone https://github.com/astraclawteam/agent-computer-use-mcp.git
cd agent-computer-use-mcp
npm ci
npm test
npm run mcp

Protocol and packaging gates:

npm run phase:1.6
npm run phase:1.7
npm run phase:1.8
npm run artifact:windows:build -- --allow-network

Real desktop action gate:

npm run phase:1.4

The full test suite is deliberately serial. It covers MCP interoperability, leases, policy, perception, Unicode delivery, cleanup races, overlay exclusion, and immutable Windows artifact assembly.

Release

A vX.Y.Z tag must match package.json, point to a commit reachable from main, and have a matching CHANGELOG.md section. The release workflow then:

1. runs the full test and official MCP SDK gates;

2. assembles and verifies the Windows x64 SEA artifact;

3. writes SHA256SUMS.txt;

4. registers and signs the artifact through the pinned Hub publisher;

5. verifies the exact version repeatedly through the public Hub catalog;

6. creates the GitHub Release and marks it Latest.

The Hub registration job has only its dedicated release transport credentials; the final isolated GitHub job receives contents: write only after the public catalog gate passes. The workflow has no npm publishing credential and does not mutate Gitee source. See the release pipeline specification and release gates.

Contributing

Real application edge cases are especially valuable: custom-drawn controls, tray restoration, high-DPI scaling, multilingual input, focus transitions, and post-action verification. Start with CONTRIBUTING.md, or open an app smoke report.

Security-sensitive findings should use GitHub private vulnerability reporting.

License

MIT