# mcp-computer-use

Standalone [MCP](https://modelcontextprotocol.io/) server exposing the
`computer_use` tool verbatim — the same tool, same schema, same verdict
ladder as in [Hermes Agent](https://github.com/NousResearch/hermes-agent)'s
`tools.computer_use`. Backed by [cua-driver](https://github.com/trycua/cua).

> The only difference from the original: transport. The vendor ships it as
> an in-process tool; we ship it as an MCP stdio tool so any
> MCP-speaking client (Claude Desktop, Cursor, Cline, your custom CLI,
> etc.) can drive a real desktop through it. The tool keeps the
> vendor's own name — `computer_use` — which is safe: host agents like
> Hermes prefix MCP tools with the server name
> (`mcp__mcp_computer_use__computer_use`), so it never collides with
> their in-process `computer_use` tool. Override via
> `MCP_COMPUTER_USE_TOOL_NAME` (any `[a-z][a-z0-9_]*` identifier) if a
> specific deployment wants a different name.

## What you get

A single MCP tool named `computer_use` with 14 actions:

| Action | Description |
|---|---|
| `capture`    | Screenshot + accessibility tree (modes: `som` / `vision` / `ax`) |
| `click` · `double_click` · `right_click` · `middle_click` | Pointer input (`element` index or `coordinate`) |
| `drag`       | Pointer drag (`from_*` → `to_*`) |
| `scroll`     | Scroll direction + amount |
| `type`       | Type text (layout-safe, with input-method hooks) |
| `key`        | Key / shortcut combos (e.g. `cmd+shift+t`) |
| `set_value`  | Select value on a control (popup / slider) without native menu |
| `focus_app`  | Focus an app (optionally `raise_window`) |
| `list_apps` · `list_windows` | Inventory queries (real driver calls) |
| `wait`       | Poll (capped at 30s) |

Every **input action** returns a structured `verdict`:

```json
{
  "effect":     "confirmed" | "unverifiable" | "suspected_noop",
  "verified":   true|false,
  "escalation": { "recommended": "px"|"foreground"|"stop",
                   "reason":     "..." },
  "code":       "background_unavailable" | "foreground_unsupported" | null
}
```

The standard *escalation ladder* (element+background → re-capture →
coordinate+background → typed page → foreground → stop) is unchanged
from the vendor.

Input is **background-first** — no cursor steal, no focus steal, no
Space-switch. Foreground requires `delivery_mode=foreground` and
`bring_to_front=true`.

## Install

```bash
git clone https://github.com/phimage/mcp-computer-use
cd mcp-computer-use
python3.11 -m venv .venv
. .venv/bin/activate
pip install -e .

# sanity
python -m mcp_computer_use.server
```

Requires Python ≥ 3.11 and `cua-driver` on `PATH`:
`which cua-driver`. macOS needs *Accessibility* + *Screen Recording*
granted to the MCP client terminal (the vendor driver handles the rest).

## Wire it into an MCP client

See [`examples/claude_desktop.json`](examples/claude_desktop.json) for a
Claude-Desktop-style config. The canonical shape is:

```json
{
  "mcpServers": {
    "computer-use": {
      "command": "python",
      "args": ["-m", "mcp_computer_use.server"],
      "env": {
        "MCP_COMPUTER_USE_APPROVAL_MODE": "deny",
        "MCP_COMPUTER_USE_SESSION_KEY": "my-client"
      }
    }
  }
}
```

## Remote over HTTP — Hermes in Docker, server on the host

Because this tool drives the *host* desktop (real AX APIs, real `cua-driver`
process), the server has to run **on the host machine**, not inside the
container. The container (where Hermes lives) then reaches it over the
network. Two transports are offered:

| Transport | How to run | When |
|-----------|-----------|------|
| `stdio` (default) | `python -m mcp_computer_use.server` | Same machine, client spawns it as a subprocess (Claude Desktop, Cursor, Hermes on the host). |
| `http` (Streamable-HTTP) | `python -m mcp_computer_use.server --transport http` | **Remote client** — e.g. Hermes in a Docker container, or any client over the network. |

### 1) Run the server on the host side (HTTP mode)

```bash
# On the host Mac, in a terminal:
MCP_COMPUTER_USE_APPROVAL_MODE=auto \
MCP_COMPUTER_USE_SESSION_KEY=hermes-docker \
MCP_COMPUTER_USE_HOST=0.0.0.0 \
MCP_COMPUTER_USE_PORT=8723 \
.venv/bin/python -m mcp_computer_use.server --transport http
```

By default it binds `0.0.0.0:8723` with a stateless HTTP session — each
request is self-contained (no session id to keep), which is the right
posture for a shared host service. The endpoint is `http://<host>:8723/mcp`;
`/health` is a trivial liveness probe.

- `MCP_COMPUTER_USE_STATELESS=1` (default) — stateless; any client can hit it.
- `MCP_COMPUTER_USE_ALLOWED_HOSTS` / `MCP_COMPUTER_USE_ALLOWED_ORIGINS`
  — optional DNS-rebinding / origin allow-lists (comma-separated). Empty =
  protection disabled, same as the MCP SDK default.
- `MCP_COMPUTER_USE_APPROVAL_MODE` — `deny` / `auto` / `callback` as before;
  the gate still applies in HTTP mode (the decision is made in the server
  process on the host, not in the container).

### 2) Point the Hermes-in-Docker client at it

From **inside** the container, register the host endpoint as a remote MCP
server (Hermes supports `url`-based MCP servers natively):

```bash
# Replace `host.docker.internal` with whatever address the container uses to
# reach the host (Docker Desktop on Mac gives you host.docker.internal
# out of the box; on Linux you'll typically use the host's LAN/bridge IP).
hermes mcp add mcp-computer-use \
  --url "http://host.docker.internal:8723/mcp"
```

If Hermes prompts "does this server require authentication?", answer **No**
for the simple setup (our server has no auth layer by default; add a
reverse-proxy / API-key in front if you want a token).

Then the tool is exposed to the agent as
`mcp__mcp_computer_use__computer_use` — the *same* 23-parameter schema,
*same* 14 actions, *same* verdict ladder as the stdio transport. The only
difference is the path across the wire: `HTTP → /mcp → manager → Server →
vendored handle_computer_use → cua-driver on the host`.

### 3) What this topology gives you

- **One desktop, one driver.** All clients (Hermes-in Docker, Claude
  Desktop on the host, a browser-based MCP client) share the same physical
  desktop via the host `cua-driver`. Concurrent input will still interleave
  exactly like concurrent terminals; if you need true isolation, run one
  server per user/session (bind different ports, different `SESSION_KEY`s).
- **Approval stays local to the server.** The `deny` / `auto` / `callback`
  policy runs on the host process, not the container — so the container
  never needs to be able to call back, and approval is enforced even across
  remote hops.
- **No secrets in the container.** The container only holds the host's
  network address.

### Caveats

- The server must run on a host that actually has a desktop (macOS /
  Windows / Linux with cua-driver working) — a container without X11/Wayland
  is not a valid `cua-driver` target.
- HTTP mode defaults to `0.0.0.0`. If your host is on an untrusted network,
  set `MCP_COMPUTER_USE_ALLOWED_HOSTS=<container-egress-host>:*` to enforce
  the `Host` header, and consider TLS via a reverse-proxy (Caddy /
  `caddy_reverse_proxy`) for anything beyond a private LAN.
- `cua-driver` reads Accessibility / Screen Recording grants tied to the
  *terminal* on the host that started the server — grant those to that
  terminal.

## Approvals (the piece the protocol doesn't have)

The MCP spec has no first-class "ask the user" primitive, so the vendor's
approval machinery (`set_approval_callback` + `approve_once` /
`approve_session` / `always_approve` / `deny` / `timeout`) is surfaced
through an env-var policy:

| Mode | Effect |
|------|--------|
| `deny`    (default) | All destructive actions refused until an integrator installs a callback or sets `auto`. |
| `auto`     | All destructive actions auto-approved for the current session. Convenience for headless automation. |
| `callback` | The integrator must register a callback (see below); every destructive call goes through it. |

Set it via env: `MCP_COMPUTER_USE_APPROVAL_MODE=deny|auto|callback`.
The legacy `MCP_COMPUTER_USE_AUTO_APPROVE=1` is a shortcut for `auto`.

**Interactive approvals (callback mode)** — install your own callback
before constructing the server. Signature:

```python
from mcp_computer_use.approval import interactive_callback

def my_gate(action: str, args: dict, summary: str) -> str:
    # Return: "approve_once" | "approve_session" | "always_approve"
    #         | "deny" | "timeout"
    return "approve_session"

interactive_callback(my_gate)
```

See [`examples/interactive_approval.py`](examples/interactive_approval.py)
for a full working example.

**Safe actions** — `capture`, `wait`, `list_apps`, `list_windows` never
hit the gate. They are read-only and always permitted.

**Hard blocks** — a handful of dangerous patterns are refused
regardless of approval mode (e.g. `rm -rf /` typed via `type`,
`cmd+shift+backspace` empty trash, `cmd+option+shift+q` force-logout).
These come straight from the vendor's
[`_BLOCKED_KEY_COMBOS`](src/mcp_computer_use/tool.py) table.

## Tests

```bash
pip install -e ".[dev]"
pytest -v
```

36 tests cover:
- schema identity (the 23-param dict, the 14-action enum)
- renderer (OpenAI-style → MCP `ContentBlock`, including the
  multimodal `capture` shape)
- dispatch through the vendored handler (`_SAFE_ACTIONS`,
  `_DESTRUCTIVE_ACTIONS`, callback verdicts, blocked patterns)
- E2E over stdio using the official MCP Python client (`initialize`,
  `tools/list` with the default name, `MCP_COMPUTER_USE_TOOL_NAME`
  override + invalid-name rejection, `tools/call`, unknown-tool error)
- E2E over Streamable-HTTP using the official MCP Python client — the exact
  path a remote client (Hermes in Docker) uses: server launched as a
  subprocess on a real port, then `initialize`, `tools/list`, and a full
  `tools/call` that reaches the vendored handler across the wire

## Live smoke test

```bash
.venv/bin/python examples/live_e2e.py
```

Spawns the *real* server as a subprocess, talks to it through the
official MCP client, and drives the *real* cua-driver: `list_apps`,
`focus_app Finder`, `capture Finder` (saves the screenshot to
`~/Library/Caches/mcp-computer-use/cache/images/`).

## Project layout

```
src/mcp_computer_use/
├── __init__.py           # public API + bootstrap invocation
├── _bootstrap.py         # sys.modules alias map
├── _shims/               # one module per vendor-internal import path
│   ├── async_utils.py    # verbatim from hermes
│   ├── aux_client.py
│   ├── hermes_constants.py
│   ├── image_routing.py
│   ├── lazy_deps.py
│   ├── load_config.py
│   ├── model_tools.py
│   ├── models_dev.py
│   ├── subprocess_compat.py
│   ├── subprocess_env.py
│   ├── tool_approval.py
│   ├── tools_config.py
│   └── vision_tools.py
├── backend.py            # [verbatim] ComputerUseBackend ABC
├── cua_backend.py        # [verbatim] CuaDriverBackend + MCP-over-stdio client
├── schema.py             # [verbatim] COMPUTER_USE_SCHEMA
├── tool.py               # [verbatim] handle_computer_use + dispatch
├── vision_routing.py     # [verbatim]
├── approval.py           # [new] env-var policy adapter
├── server.py             # [new] mcp.server.Server + tool registration
└── __main__.py           # [new] `python -m mcp_computer_use.server`
tests/
├── conftest.py           # clean-env + no-backend fixtures
├── test_approval.py      # approval policy
├── test_dispatch.py      # vendor dispatch (safe / destructive / callback)
├── test_e2e_stdio.py     # full protocol via the mcp client
├── test_render.py        # renderer unit tests
└── test_schema.py        # schema identity
examples/
├── claude_desktop.json
├── interactive_approval.py
└── live_e2e.py
```

## License & attribution

- **mcp-computer-use** — MIT, © 2026 phimage
- **Vendored source** — `tools/computer_use/*` from
  [NousResearch/hermes-agent](https://github.com/NousResearch/hermes-agent),
  MIT © 2025 Nous Research. Unmodified (see `[verbatim]` tags above).
- **Driver** — [trycua/cua](https://github.com/trycua/cua),
  MIT © 2025 Cua AI, Inc.