<picture>
  <source media="(prefers-color-scheme: dark)" srcset="assets/icon-dark.svg">
  <img src="assets/icon-light.svg" width="80" height="80" alt="Codewhale whale and pointer icon">
</picture>

# Computer Use

**By Codewhale · macOS beta (source) · Windows and Linux experimental, source only**

Let Codewhale see and operate your apps. Read accessible controls, enter
text, click, scroll and capture the selected app through the same MCP tools.
The macOS helper keeps permissions and human controls in one menu-bar app.

- **Set up once.** See Accessibility and Screen Recording status, open the
  right Settings pane, then run a check in a disposable practice window.
- **Keep working.** macOS selects apps in background mode by default. Actions
  that need the shared pointer are refused until foreground control is chosen
  with the user's authorization. Background support varies by application.
- **Stay in control.** See selected apps and their input modes. Pause cancels
  queued work and releases held input; Stop ends existing sessions. Only the
  person using the menu-bar controls can allow input again.
- **Update deliberately.** The app checks for stable releases on request.
  Updates verify the digest, Codewhale signature and Apple notarization before
  replacing the app, and retain the previous install for rollback.

[Setup page](https://codewhale.net/computer-use) ·
[Setup and troubleshooting](docs/TROUBLESHOOTING.md) ·
[Release notes](CHANGELOG.md) · [Distribution](docs/DISTRIBUTION.md) ·
[Background demo](docs/DEMO.md) · [Contributing](CONTRIBUTING.md) ·
[Security](SECURITY.md)

**Status.** This repository starts at the 0.3.1 source snapshot, a macOS beta
candidate. No signed download has been published yet: the macOS app is built
from source today, and the first public release will appear under this
repository's GitHub Releases only after the remaining qualification gates in
[the release checklist](docs/RELEASE_CHECKLIST.md) pass. Earlier versions were
developed privately; their notes and receipts are kept for context.

The native setup panel, background check and updater require **macOS 13.5+**
for the self-contained bundle. The source MCP server includes experimental
Windows and Linux backends, with HarmonyOS devices over hdc. Windows and
Linux are source-only: they do not yet have the native human controls,
exact-window targeting or qualified installers, and they are excluded from the
plugin's public host eligibility until those gates pass. Their raw input uses
the shared desktop and must not be treated as background control. Windows
semantic mutations currently refuse scoped element targets. See the
[publication review](docs/PUBLICATION_REVIEW.md) and
[porting plan](docs/PORTING.md).
The SSH route remains experimental. See the platform-specific
[limitations](docs/LIMITATIONS.md).

The server works with MCP hosts including Codewhale, Kimi Code, Claude Code,
Codex CLI and Cursor. It has zero runtime npm dependencies. Developer
checkouts use Node 20+; the macOS distribution includes a pinned Node 24 LTS
runtime. Codewhale still reviews, trusts and enables a plugin through its
existing Engine authority before the model can use it.

## Verification status

**Source (this snapshot).** `npm test` on macOS: 240 passed, 0 failed,
15 platform skips. The GitHub Actions workflow runs the same suite plus the
receipt hygiene check on macOS and Ubuntu runners. Source tests exercise the
protocol, routing, session and injected-runner paths; they perform no native
input and do not qualify a distributed app.

**Native macOS 0.3.1 candidate (one maintainer Mac, arm64).** A signed 0.3.1
build passed the menu-bar owner crash and reopen check with isolated state.
The updater's apply step replaced an installed notarized 0.3.0 app with the
notarized 0.3.1 build: the previous bundle was retained for rollback, the
helper restarted with controls stopped, and all 33 runtime files plus the 3
native executables in the installed bundle matched the build. Still unproven:
a clean-machine install with fresh OS permission grants, and a model-driven
task through an installed Codewhale Engine. The 0.3.1 app is not published.

**Live-verified during development: macOS (arm64, single Retina display,
macOS 26.1).** Each of the 27 fixture workflows has a five-trial passing run.
The broad 26-task run passed 129/130 trials; its dynamic-page failure was a
fixture clock race, corrected and repeated 5/5. The repaired file-picker flow
also passed 5/5 separately. The matrix retains the broad failure and both
focused runs; these are 0.2.1 development receipts, not full Codex parity or
final release qualification. Shared-desktop pointer displacement is measured
and sometimes nonzero. OS permissions survived the signed 0.2.1 update. Commit
identifiers inside the matrix and result files refer to the private
pre-publication history, not to commits in this repository.

**Text and vision use the same actions.** App observations default to a text
summary containing controls, values, actions and layout. `detail:"full"`
exposes nested menus and tree structure. On macOS, `include_ocr:true` adds
on-device recognition of visible text with confidence and coordinate targets,
without a vision model or remote service. The default does not capture an
image. OCR was verified on a generated image and the actual Codewhale app;
it does not interpret unlabeled icons, charts or other graphical meaning.
Screenshots and zoom remain available to models that support images.

**Live-verified: Linux X11** — shared desktop and the isolated Xvfb route,
26/27 demonstrated. Those rows predate the parity runner's split into a
platform-neutral engine plus per-platform drivers and have not been re-run
since; see [docs/PARITY_MATRIX.md](docs/PARITY_MATRIX.md).

**Background input has a native AppKit verification harness** with an independent
foreground/cursor observer: `node scripts/verify-background-macos.mjs`.
Older per-family receipts do not establish background isolation — details in
[docs/LIMITATIONS.md](docs/LIMITATIONS.md).

**Not live-verified:** macOS non-Retina and mixed-DPI, Windows, Wayland,
HarmonyOS, SSH remote — implemented, no device receipts. A
[same-document native editing comparison](parity/results/native-text-comparison-darwin-2026-09-07.json)
completed 5/5 trials with each of Codewhale and Codex; it covers one workflow,
not the full task suite. A separate
[public-browser navigation comparison](parity/results/public-browser-comparison-darwin-2026-09-07.json)
also completed 5/5 trials on each surface, using Codewhale 0.2.1. Known behavioral limitations and the
release gating checklist are in
[docs/LIMITATIONS.md](docs/LIMITATIONS.md) and
[docs/RELEASE_CHECKLIST.md](docs/RELEASE_CHECKLIST.md); how to run or extend
the suite is in [docs/PARITY.md](docs/PARITY.md).

## The Mac app: beta, not yet downloadable

The setup page is
[codewhale.net/computer-use](https://codewhale.net/computer-use), also linked
from Codewhale’s install page and plugin marketplace. A download becomes
available only when a stable GitHub release in this repository includes the
notarized universal app and its matching qualification receipt. No such
release exists yet, so the page reports availability without offering an
installer, and the app's **Check for updates…** reports that no stable
installer has been published. Until then, build and install from source with
the developer quick start below, which requires a Mac with Xcode Command Line
Tools. See [docs/DISTRIBUTION.md](docs/DISTRIBUTION.md) for the packaging,
notarization and release procedure.

## Developer quick start

```bash
git clone https://github.com/Hmbown/codewhale-cu-plugin
cd codewhale-cu-plugin
npm test                # unit and protocol tests, no GUI input performed
npm run build:app       # dist/{macos,linux,windows}
npm run install:app     # puts the app in place, registers it, opens it once
```

Open **Computer Use…** from the whale menu-bar icon. Grant the missing
permissions using its setup buttons, run the background check, and point your
host at the server:

| Host | Configuration |
|---|---|
| Codewhale | Discovers the Agent Plugins v1 bundle (`plugin.json` + `mcp.json`); nothing to configure. |
| Kimi Code | Run `/plugins install /path/to/codewhale-cu-plugin`, then `/reload`. The native `kimi.plugin.json` registers the server and skills. |
| Claude Code | `claude mcp add computer -- node /path/to/mcp/server.mjs` |
| Codex CLI | `~/.codex/config.toml`: `[mcp_servers.computer]` `command = "node"` `args = ["/path/to/mcp/server.mjs"]` |
| Cursor / Windsurf / VS Code | `{"mcpServers":{"computer":{"command":"node","args":["/path/to/mcp/server.mjs"]}}}` |
| Gemini CLI | same JSON in `~/.gemini/settings.json` |
| opencode | `{"mcp":{"computer":{"type":"local","command":["node","/path/to/mcp/server.mjs"]}}}` |

`/path/to/mcp/server.mjs` can be this checkout or the stable copy inside the
installed app (`install:app` prints it — on macOS
`~/Applications/Codewhale Computer Use.app/Contents/Resources/plugin/mcp/server.mjs`).
`npm link` also gives you a `codewhale-cu` command for hosts that want one.

### Kimi Code

Install the desktop helper above first, then use Kimi's `/plugins install`
command with this checkout's absolute path. Review the local plugin and choose
**Trust and install**. Kimi copies it into its managed plugin directory and
enables its MCP server. Run `/reload` in an existing session, then `/mcp`:
`plugin-codewhale-computer-use:computer` should show **connected** and **39 tools**.
`/plugins info codewhale-computer-use` shows the installed version and status.
This flow was verified with Kimi Code 0.41.0. See the
[Kimi plugin documentation](https://moonshotai.github.io/kimi-code/en/customization/plugins.html)
for managed updates and removal.

The server needs Node 20 or newer on Kimi's PATH and uses the installed
Codewhale permission helper. Existing MCP servers and model settings are
preserved. Models without vision can read the default text observations and
request local macOS OCR; models with vision can also request screenshots.

## The desktop app

macOS grants Accessibility and Screen Recording to the *responsible process*
of a permission check. A bare `node mcp/server.mjs` inherits its host's
identity, so the grant lands on Terminal, Cursor, or Claude, and every host
needs its own. The app fixes that:

- **What it is** — a tiny native launcher (`app/macos/launcher.c` on macOS)
  that runs `app/daemon.mjs`, a long-lived process that executes the platform
  backend and answers one-line JSON requests over a per-user socket
  (`~/.codewhale-cu/app.sock`, a named pipe on Windows). Only the same
  allow-listed tool set the ssh agent accepts will execute; the socket is
  never a shell.
- **How the server uses it** — every call on the local computer goes through
  the app when it is running. If it is installed but not running, the server
  launches it (via LaunchServices on macOS, so it is its own responsible
  process) and waits for it. If it is not installed, calls run directly in
  the server process as before. `request_access` reports which mode is active
  (`via: "app"` or `"direct"`) and, in direct mode, how to install the app.
  A registered helper takes priority over Codewhale's embedded native helper;
  if the registered helper cannot start, input fails closed. Without a
  standalone registration, Codewhale can use its embedded permission identity.
  `CODEWHALE_CU_APP=off` is an explicit developer override for direct mode,
  never an agent workaround for the person's Pause or Stop choice.
- **First launch** — open the whale menu to review permission status. Only the
  setup buttons request grants; starting the helper does not prompt automatically.

Where `install:app` puts things:

| OS | App | Also |
|---|---|---|
| macOS | `~/Applications/Codewhale Computer Use.app` (registered with LaunchServices) | `--login`: `~/Library/LaunchAgents/net.codewhale.computer-use.plist` |
| Linux | `~/.local/share/codewhale-computer-use` | `~/.local/bin/codewhale-computer-use`, `~/.local/share/applications/net.codewhale.computer-use.desktop`, hicolor icons; `--login`: `~/.config/autostart/` |
| Windows | `%LOCALAPPDATA%\Programs\Codewhale Computer Use` | Start Menu shortcut with icon; `--login`: Startup-folder shortcut |

### macOS permission recovery

The build signs the complete app bundle, and installation signs again after
pinning Node, then verifies the installed signature. Signing only the launcher
leaves an invalid app identity: macOS can show an enabled switch while refusing
the grant and repeatedly prompting for Accessibility.

If upgrading from that broken package, remove the old Codewhale Computer Use
entry from Accessibility, add `~/Applications/Codewhale Computer Use.app`, and
enable it. Refresh its Screen & System Audio Recording grant and restart the
helper. The probe must report Accessibility `granted` and screen capture `ok`;
`via: "app"` alone is not a pass. Authentication in System Settings is handled
by the user. Never modify the TCC database.

Builds use `CODEWHALE_CU_SIGN_IDENTITY` when configured, otherwise an available
Developer ID or Apple Development identity, and fall back to ad-hoc signing.
Ad-hoc updates can require a new grant. Installation preserves the selected
identity and verifies the finished bundle. Building the native macOS helper
requires Xcode Command Line Tools on a Mac; the Codewhale distribution embeds
the compiled helper. Do not edit installed resources without re-signing.

### Background control and visible cursor on macOS

Use `open_application` with `activate: false` to select the input destination
without bringing it forward. Keyboard events go to that process and semantic
actions use Accessibility — both are quiet: no cursor movement, no activation.
Unqualified `get_app_state`, `list_windows`, and `screenshot` follow that app,
including behind other windows. Explicit display/region captures remain available.
Text insertion prefers writable accessibility selection over process key events.
For an authorized workflow that requires keyboard focus, explicitly select
`activate: true`; receipts then say `keyboard_delivery: "foreground-guarded"`.
Keystrokes and raw mouse gestures stop if another app takes focus; gestures never
reactivate it. Select `activate: false` again to
return to process-bound delivery. In either mode, verify the application
result: a successful dispatch alone does not prove the app handled it.

Coordinate `left_click` first resolves the bound application's accessibility
control and uses accessibility (`strategy: "a11y"`). Text fields focus directly,
rows can select, and menu items can use their advertised pick action. Right-click
uses an advertised context-menu action. Background scrolling uses the selected
scrollbar: native increments when advertised, otherwise normalized 5% steps,
with the actual unit and value change in the receipt. Unsupported gestures,
including raw double/triple/middle click, drag and hover, remain unavailable in
background mode. There is no automatic foreground fallback.

When the user authorizes exclusive desktop use, select `activate: true` for
shared-desktop control. Pointer gestures move the real cursor while the selected
app stays frontmost; restoring the cursor afterward is not isolation. The receipt
reports `strategy: "event"`, `pointer_moved` and `foreground_taken`. A point
covered by another application's window is still refused. Return to
`activate: false` when the shared-desktop step ends.

The binding receipt exposes `input_scope`, `shared_pointer` and
`isolated_desktop: false`. The preview title distinguishes background app
control from shared-desktop control. It is a view of the app, not a sandbox.

Only when the user asks to watch, enable `preview` with `enabled: true` to show a small, nonactivating window
containing the controlled app and a cyan cursor labeled Codewhale. The preview
updates after agent actions, and its cursor is separate from the hardware
pointer. Close the panel or use `enabled: false` to hide it.

This is background control of a local app, not an isolated desktop. Some apps,
system dialogs, and workflows may still require foreground interaction. The
opt-in `node scripts/verify-macos.mjs` test exercises a uniquely named TextEdit fixture (closed automatically afterward; add `--preview` to test the overlay),
Unicode input, selection, screenshots and foreground preservation through a
fresh MCP connection. Recording is separately opt-in with `--recording`. It creates local receipts and is
separate from the automated unit suite.

Logs: `~/Library/Logs/Codewhale Computer Use/app.log` (macOS),
`~/.local/state/codewhale-computer-use/app.log` (Linux),
`%LOCALAPPDATA%\Codewhale Computer Use\app.log` (Windows).
`npm run remove:app` reverses the install. Every bundle carries a complete
runtime copy of the plugin, so hosts can target the installed path and survive
deleting this checkout.

## Session ownership

Each MCP connection has independent computer selection, application binding,
rasters, accessibility observations and held-input ownership. Local actions
are serialized by the permission-owning app; stopping or cancelling one
session releases its own held input. Normal client shutdown closes the app
session; losing the client connection also cancels and releases its input.
On macOS, the native input helper also releases its press when its owning
process disappears, including in direct mode. On macOS, session exit stops a
recording started by that session; cancelling an ordinary request leaves an explicitly
started recording running until stopped or the session exits.

Session protocol 2 (introduced in 0.2.1) is required: upgrade the helper and
restart existing MCP connections together. An old client or helper is refused
with an upgrade error instead of sharing another client's input state.

## Frontier ability set

- **Observe & resolve** — `list_apps`, `list_windows`, `list_displays`,
  `switch_display`, `get_app_state` (accessibility/UIA/uitest tree with
  element indices + `state_id`), `screenshot` (display/region, raster-bound
  coordinates), `zoom` (close-up crop of the last raster), `cursor_position`,
  `open_application` (exact-name rule), `request_access` (fail-closed
  permission/capability probe).
- **Pointer** — left/double/triple/right/middle click, move, drag,
  down/up, scroll (4 directions).
- **Keyboard & text** — `type` (unicode), `key` (chords + repeat),
  `hold_key`, `set_value` (semantic, background-safe), `select_text`,
  `perform_action` (element's own actions: AXPress / UIA Invoke / AT-SPI / uitest).
- **Recording** — `recording_start/stop/status/list` (see below).
- **Computers** — `computer_list`, `computer_switch`, `computer_register`
  (ssh agent auto-push), `computer_remove`.
- **Safety** — `stop_computer_control` kill switch; permission probes that
  name the missing grant; receipts on every call naming the computer it
  happened on.

## Requirements

Tools and permissions are probed at call time; `request_access` reports what is
missing and every capability **fails closed naming the missing tool or
permission** — it never guesses and never half-acts.

- **macOS** — Accessibility + Screen Recording for the app (or, in direct
  mode, for the terminal that hosts the server). python3+pyobjc or cliclick
  improves cursor reads.
- **Windows** — PowerShell (built in). Recording is unavailable pending
  session-owned recorder cleanup.
- **Linux** — X11: xdotool, wmctrl, scrot or imagemagick, xclip; Wayland:
  grim, wtype, ydotool+ydotoold, wl-clipboard; python3-pyatspi
  for the accessibility tree. Recording is unavailable pending session-owned
  recorder cleanup.
- **HarmonyOS** — `hdc` on PATH with the device connected
  (`hdc list targets`); ffmpeg on the host for snapshot-series recordings.

## How the four platforms map

| Ability | macOS | Windows | Linux | HarmonyOS |
|---|---|---|---|---|
| Accessibility tree | Native Accessibility API | UIAutomation | AT-SPI (pyatspi) | `uitest dumpLayout` |
| Raw input | Native CGEvent to the selected process | user32 SendInput/mouse_event (PowerShell) | xdotool (X11) / ydotool+wtype (Wayland) | `uitest uiInput` |
| Screenshots | `screencapture` | .NET CopyFromScreen | scrot/import (X11), grim (Wayland) | `snapshot_display` |
| Recording | ScreenCaptureKit → .mov (no recorder overlay) | unavailable pending owned cleanup | unavailable pending owned cleanup | snapshot-series + ffmpeg mux |
| Clipboard | pbcopy/pbpaste | Get/Set-Clipboard | xclip/xsel, wl-clipboard | fail-closed (not exposed by hdc) |

## Remote computers (ssh)

```json
computer_register { "computer": "winbox", "transport": "ssh", "host": "winbox.lan", "user": "me" }
```

Registration pushes the self-contained agent (`agent.mjs` + `src/`) to
`~/.codewhale-cu/agent/` on the remote over scp, probes the remote platform
through it, and pins the result. Remote calls run
`node agent.mjs <base64 json>` — one JSON receipt line back. Only an
allow-listed tool set executes remotely; arguments travel as data, never as
shell. Requires publickey ssh (BatchMode) and Node ≥ 20 on the remote.

## HarmonyOS computers

```json
computer_register { "computer": "pad", "transport": "hdc" }
```

Drives the device over `hdc shell uitest ...` and `snapshot_display`. Element
targets come from `dumpLayout`; input is touch-synthesis (click / swipe /
inputText / keyEvent). Recording is honestly labeled `snapshot-series`
(frame captures muxed on stop) because HarmonyOS exposes no CLI screen
recorder.

## Layout

```
mcp/server.mjs        MCP stdio server (JSON-RPC 2.0), tool dispatch, receipts
src/tools.mjs         tool schemas — single source of truth for tools/list
src/backends/         darwin / win32 / linux / harmonyos
src/transport.mjs     app socket · local · ssh · hdc executors
src/app-socket.mjs    app naming, socket protocol client, launch-on-demand
src/app-handler.mjs   allow-listed request handler shared by app + ssh agent
app/daemon.mjs        the desktop app process
app/macos/launcher.c  native bundle executable (keeps TCC attribution on the app)
agent.mjs             one-shot ssh remote agent
assets/               icon source + generated .png/.icns/.ico/hicolor, prebuilt mac launcher
scripts/              build-icons · build-app · install-app · smoke
commands/, skills/    Agent Plugins v1 command + skills for hosts that read them
```

## Development

```bash
npm test              # unit + protocol + app tests (no GUI input performed)
npm run smoke         # live end-to-end against this machine (isolated state dirs)
npm run build:icons   # regenerate assets/ from assets/icon-source.png
npm run build:app     # bundles into dist/ (compiles the mac launcher when clang is present)
```

Proven levels are separated: local live (this Mac: darwin) > mocked transport
(ssh protocol, harmony backend logic) > code-complete (win32/linux paths,
implemented to their documented tool interfaces but only verifiable on those
platforms).

Codewhale embeds a copy of this repository's runtime tree as its built-in
Computer Use plugin; this repository is the upstream source. Changing this
checkout or the standalone helper does not update an installed Codewhale
binary.

## Support and contributing

Use [GitHub issues](https://github.com/Hmbown/codewhale-cu-plugin/issues) for
bugs and questions. Include the plugin version, macOS version, the failed
action and the error text, with private app contents and credentials removed
from any log excerpt. [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)
covers the common setup problems first. Contribution expectations are in
[CONTRIBUTING.md](CONTRIBUTING.md); vulnerability reporting and the security
model are in [SECURITY.md](SECURITY.md). This is a beta maintained on a
best-effort basis; there is no support commitment or response-time promise.

## License

MIT — see `LICENSE`.