cua — computer use for protoAgent

Lets the agent drive native GUI apps on this machine: snapshot a window's accessibility tree, then click / type / scroll by element — without pulling the app to the foreground while you're using the computer.

It wraps cua-driver (MIT) as a managed MCP server. The driver is a separate binary that runs out-of-process, so this plugin adds zero packages to protoAgent's venv and works in the frozen desktop build.

Design rationale, options weighed, and the evidence behind them: protoAgent ADR 0084.

Read this before you enable it

This is real control of your real desktop — your files, your logged-in sessions, your accounts.

protoAgent's safety fences do not apply to anything the agent does here. Not "apply weakly" — do not apply. The network egress allowlist lives inside the fetch_url tool; the filesystem fence (ADR 0007) is a path check inside fs_tools. Both are in-process checks in protoAgent's own Python tools, and a mouse click never reaches either. An agent that can open your browser can reach any site on the internet. An agent that can open Finder can read any file you can.

That's not a bug to be patched — it's what computer use is. protoAgent's posture (ADR 0071) is trust-and-consent, not sandboxing, and this plugin is that posture at full strength. Enable it deliberately.

If you want computer use with containment, that's a VM, not this plugin — see ADR 0084 D5 on the deferred cua-sandbox path.

Related MCP server: macinput

Setup

1. Install the driver. It is not installed for you, and this plugin never installs it — the probe just reports it missing until you do.

/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"     # macOS / Linux
irm https://cua.ai/driver/install.ps1 | iex                        # Windows

Lands in ~/.local/bin by default (--bin-dir or CUA_DRIVER_BIN_DIR to change). Uninstall: curl -fsSL https://cua.ai/driver/uninstall.sh | bash.

2. Grant permissions (macOS). The driver needs Accessibility and Screen Recording. Don't do this by hand in System Settings — run:

cua-driver permissions grant

This exists because TCC grants attach to the responsible app identity, and the intuitive route grants the wrong one. Approving a dialog raised from your terminal grants your terminal, not the driver; the driver still can't read a window, and every status check lies to you about it in the cheeriest way. The grant subcommand launches CuaDriver via LaunchServices so the dialog attributes to com.trycua.driver, then confirms.

Verify with cua-driver permissions status --json — it answers through the running daemon, so a true carries the driver's own identity ("attribution": "driver-daemon"). With no daemon it reports unknown rather than guessing.

Don't trust cua-driver check_permissions. It reports the caller's TCC identity, so from your shell it happily says accessibility: true on a machine where the driver has no grant at all. permissions status is the honest one. (This plugin's Test button uses permissions status; an earlier cut used the tool and reported a missing grant on a fully-granted machine.)

3. Enable the plugin. Settings ▸ Computer use ▸ Enable computer use, then hit Test connection. That's not decorative: it's the only thing that checks all three failure modes at once (binary found, TCC granted for the driver, and configured tool names real for this driver build). A healthy result reads like:

/Users/you/.local/bin/cua-driver · 17 tool(s) bound of 38 published

There is no step 4 — you don't manage a daemon. When protoAgent spawns cua-driver mcp, the driver notices it lacks grants under whatever launched it and auto-launches the CuaDriver daemon, then proxies through it:

cua-driver-rs: mcp launched without CuaDriver.app's TCC grants; auto-launching
the daemon via `open -n -g -a CuaDriver --args serve` and proxying MCP requests
through it. Pass --no-daemon-relaunch to stay in-process.

That's why one permissions grant is enough: the grant lives on com.trycua.driver and holds no matter who spawns the server — your shell, scripts/dev.sh, or the packaged desktop app. Don't pass --no-daemon-relaunch; it stays in-process and puts the grant burden back on protoAgent's own identity.

Settings

Why core is the default

The driver publishes 38 tools (verified on 0.8.3). Binding all of them spends context on every turn for surface the agent almost never needs (protoAgent ADR 0005). core is the snapshot→act loop; everything else is one config edit away.

Left out of core on purpose:

• bring_to_front — steals focus, which is the one thing this driver exists to avoid.

• kill_app — destructive.

• get_accessibility_tree — despite the name, a desktop-level snapshot (apps + visible windows + bounds/z-order) that overlaps list_apps/list_windows. get_window_state is the per-window AX tree you actually act on.

• page — the browser capability, with its own session protocol (WEB_APPS.md).

• start_session / end_session, recording, cursor styling — MCP mints a session per connection, so you don't need these.

• check_for_update — driver self-update. Deliberately operator-driven (ADR 0084): plugins.update_policy can't see a binary updating itself.

extra_tools names that the driver doesn't publish are dropped silently by the MCP filter — a wrong name is a missing tool, not an error. Test connection lists the driver's real tools and flags any that don't match.

Gotchas

Don't turn off persistent MCP sessions. mcp.persistent_sessions: false breaks element-indexed workflows. The driver caches the per-(pid, window_id) accessibility element map in the process your session talks to; a stateless session spawns a fresh cua-driver mcp per call, so the cache is gone between the snapshot that minted an element_index and the click that uses it. This plugin sets persistent: true on its entry, but the host computes mcp.persistent_sessions AND entry.persistent — a global false wins and there is nothing the entry can do about it.

The desktop app has a different PATH than your shell. ~/.local/bin is routinely absent from a service's environment. If the driver works in dev and vanishes in the packaged app, set cua-driver path to the absolute path.

screenshot doesn't exist. It was removed upstream (cua PR #1692) — capture is get_window_state with capture_mode: "vision". Docs and blog posts that tell you to call screenshot are out of date.

MCP doesn't need to be on. A plugin-contributed server activates MCP by itself; you don't need mcp.enabled: true.

What it registers

• A managed MCP server (register_mcp_server) spawning cua-driver mcp over stdio. Tools arrive namespaced cua-driver__*. The factory returns None — the documented "don't start" path — when disabled or when the binary is absent. Verified end-to-end against driver 0.8.3: 17 tools bind, the allowlist holds exactly (no leakage from the other 21), and it activates MCP on its own — mcp.enabled: true is not required.

• A skill (register_skill_dir) teaching the snapshot-before-and-after invariant. This is not documentation garnish: the driver is accessibility-tree-first, element_index values are minted per snapshot and go stale across turns, and actions that silently no-op look identical to actions that worked. Without the protocol the tools look broken.

• POST /api/config/test-cua backing the Settings Test button.

No Python tools, no runtime dependencies.

Development

uv venv --python 3.11 && uv pip install -r requirements-dev.txt ruff
.venv/bin/ruff check . && .venv/bin/ruff format --check . && .venv/bin/pytest -q

The suite runs with no protoAgent host — tests/conftest.py registers the plugin as a synthetic package the way the real loader does, and fakes the registry seam.

License

MIT. Wraps trycua/cua (MIT), installed separately and not vendored here.