<p align="center">
  <img src="docs/assets/silenthand-mark.svg" width="88" height="88" alt="silenthand mark">
</p>

<h1 align="center">silenthand-cua</h1>

<p align="center">
  <strong>Unofficial Computer Use host for a local Mac.</strong><br>
  Click, type, and read AX trees without stealing focus. The MCP wrapper does not start the service.<br>
  Ready on macOS. Windows and Linux are in development.
</p>

<p align="center">
  <img alt="macOS" src="https://img.shields.io/badge/macOS-ready-111?logo=apple&amp;logoColor=white">
  <img alt="Windows" src="https://img.shields.io/badge/Windows-in%20development-6b7280?logo=windows&amp;logoColor=white">
  <img alt="Linux" src="https://img.shields.io/badge/Linux-in%20development-6b7280?logo=linux&amp;logoColor=white">
  <img alt="Node.js" src="https://img.shields.io/badge/node-24.14.0-3c873a?logo=nodedotjs&amp;logoColor=white">
  <img alt="MCP" src="https://img.shields.io/badge/MCP-stdio-0d9488">
  <img alt="CI" src="https://github.com/edosulai/silenthand-cua/actions/workflows/test.yml/badge.svg">
  <img alt="License" src="https://img.shields.io/github/license/edosulai/silenthand-cua">
  <img alt="npm" src="https://img.shields.io/badge/npm-not%20published-6b7280">
</p>

<p align="center">
  <a href="#quick-start">Quick start</a> ·
  <a href="#how-it-works">How it works</a> ·
  <a href="#mcp-setup">MCP setup</a> ·
  <a href="#available-tools">Tools</a> ·
  <a href="#critical-constraints">Constraints</a> ·
  <a href="#origin">Origin</a>
</p>

<p align="center">
  <img src="docs/assets/terminal.svg" width="760" alt="node cli.mjs doctor — BINARIES_MISSING until fetch-runtime copies ChatGPT CUA binaries">
</p>

An unofficial research host for OpenAI Computer Use. The CUA binary already speaks MCP, but over **newline JSON-RPC**. IDE hosts speak **Content-Length**. `mcp-wrapper.sh` is the agent-facing entry.

This is **not** a published npm package. Do not `npx silenthand`. There is no Homebrew formula. The runtime needs proprietary ChatGPT CUA binaries that **must not ship** — clone the repo, copy them from a local ChatGPT.app, attach `mcp-wrapper.sh`.

This can break whenever ChatGPT updates CUA. Bundle IDs and binary names are load-bearing. Auth bypass is a local DYLD interpose for debugging.

---

## Why this exists

ChatGPT's Computer Use service can drive local apps without a Window Server focus switch. Agents still need a host that speaks Content-Length MCP, a readiness check that does not auto-start anything, and a skill that forbids `Raise` / `open -a`.

Most wrappers start a daemon and claim success from `ok=true`. SilentHand does neither. `silenthand_doctor` is read-only. Re-observe after every action.

---

## Quick start

**Ready on macOS 14.4+ arm64.** Windows and Linux are in development — do not treat this checkout as an install path there. Node 24.14.0, ChatGPT for Mac already installed:

```bash
git clone https://github.com/edosulai/silenthand-cua.git
cd silenthand-cua
nvm use          # v24.14.0
npm install

# 1. Copy SkyComputerUse* from ChatGPT.app (gitignored — never committed):
./scripts/fetch-runtime.sh
#    or: node cli.mjs fetch-runtime

# 2. Read-only doctor. Does not start the service:
node cli.mjs doctor

# 3. Copy the agent skill and attach Hermes MCP (no hermes mcp add, no npx):
node cli.mjs setup --platform hermes

# 4. Start SilentHand CUA.app via LaunchServices (TCC = this app, not Terminal):
./run-silenthand.sh
```

`setup --platform hermes` copies `~/.hermes/skills/silenthand/` **and**
attaches `mcp-wrapper.sh`. Restart Hermes / `/reload-mcp` so this chat
has `mcp_silenthand_*` tools. There is no `npx`.

REPL / one-shot (after the service is up):

```bash
node silenthand-repl.mjs
node silenthand-client.mjs list_apps
node silenthand-client.mjs get_app_state Finder
node silenthand-client.mjs click Finder --x 200 --y 300
```

## Prerequisites

- macOS 14.4+ (arm64)
- Node.js v24.14.0 (via nvm)
- ChatGPT for Mac (source of `SkyComputerUseService` / `SkyComputerUseClient`)
- System permissions: **Screen Recording** + **Accessibility** granted for **SilentHand CUA.app** (LaunchServices `open -n -g` on the app path — not Terminal, not node, not Hermes)

`./scripts/fetch-runtime.sh` looks at `/Applications/ChatGPT.app` by default. Override with `SILENTHAND_CHATGPT_APP=/path/to/ChatGPT.app`.

## Available Tools

| Tool | Description | Required Args | Evidence notes |
|------|-------------|---------------|----------------|
| `silenthand_doctor` | Read-only readiness. Preferred first call | — | Does not start the service or touch any app |
| `list_apps` | Official sky mapper of `MacComputerUseClient.listApps()` | — | Observed `{ id, displayName, isRunning, lastUsedDate, useCount }`. Not the binary MCP text. |
| `get_app_state` | A11y tree + screenshot | `app` | Observed; may return transient `-3811` or app-specific timeouts. Pass `disableDiff=true` for a full AX tree. |
| `click` | Click by `element_index` and/or coordinates | `app` + (`element_index` **or** `x`+`y`) | Element-index click Observed without coords; coord-only may return `noWindowsAvailable` on background windows |
| `type_text` | Type text | `app`, `text` | Observed preserve-focus on background Chrome |
| `press_key` | Press keyboard key | `app`, `key` | Use `super` for Cmd; `Command+l` Observed NOT working for Chrome address bar |
| `scroll` | Scroll element | `app`, `element_index`, `direction` | Background element-index scroll Observed `noWindowsAvailable` without focus steal |
| `drag` | Drag between coords | `app`, `from_x`, `from_y`, `to_x`, `to_y` | Limited testing |
| `set_value` | Set element value | `app`, `element_index`, `value` | Browser chrome AX IDs may be invalid. WhatsApp empty composer is not settable until a letter is present. |
| `select_text` | Select text in element | `app`, `element_index`, `text` | Untested in matrix |
| `perform_secondary_action` | Trigger AX action | `app`, `element_index`, `action` | **Never `Raise`.** WhatsApp `Cancel` on PTT trash accepted but did not dismiss. |
| `start_app` | Host-side `ComputerUseIPCAppStartRequest` | `app` | Live result `START_APP_TRANSPORT_UNAVAILABLE` — signed nativePipe is not implemented. Never `open -a`. |
| `silenthand_pip` | Path B spectator PiP | optional `live` | Default dry-run. `floatingPipClaimed` stays false unless Observed. No `windowID`. Does not start ChatGPT.app. |
| `silenthand_lock_screen_guardian` | Official CUALockScreenGuardian contract | — | Extracted. Does not spawn. |
| `silenthand_turn_ended` | Official `cua turn-ended` CLI/IPC names | — | Extracted. Does not spawn or send a payload. |

`event_stream_*` / `computer_history_*` are official extra servers when those binaries answer. Do not start a recorder unless asked.

## REPL Usage

```
cua> list_apps
cua> get_app_state "System Settings"
cua> click "System Settings" --x 150 --y 400
cua> type_text Safari --text "hello world"
cua> press_key Safari --key Return
cua> scroll Safari --element_index 5 --direction down
cua> drag Finder --from_x 100 --from_y 200 --to_x 300 --to_y 400
cua> set_value "System Settings" --element_index 48 --value "Display"
cua> raw {"name":"click","arguments":{"app":"Finder","x":100,"y":200}}
cua> quit
```

## How it works

```
MCP host (VS Code / Claude / Hermes)
  └─ Content-Length stdio → mcp-wrapper.sh → mcp-server.mjs
        ├─ silenthand_doctor / start_app / silenthand_pip /
        │  silenthand_lock_screen_guardian / silenthand_turn_ended (host-side)
        └─ NDJSON + elicitation proxy → SkyComputerUseClient
              └─ Unix socket → SkyComputerUseService
                    ↑ LaunchServices open -n -g SilentHand CUA.app
                      trampoline DYLD-inserts in-bundle interpose_auth.dylib
```

The service appears to use `SyntheticAppFocusEnforcer` so a target can accept input without a Window Server focus switch. **Inferred** from binary analysis. Some Chrome routes were **Observed** to keep the agent’s editor frontmost. This is **not** a universal guarantee. Verify with `NSWorkspace.shared.frontmostApplication`, not `Secondary Actions: Raise`.

## MCP setup

This is Computer Use for local apps, not a WhatsApp API. The wrapper does **not** start `./run-silenthand.sh`. Call `silenthand_doctor` first. If `ready=false`, follow `userMessage` / `agentNextSteps`.

VS Code / Copilot (`mcp.json`):

```json
{
  "servers": {
    "silenthand": {
      "type": "stdio",
      "command": "/ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh"
    }
  }
}
```

Claude Desktop (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "silenthand": {
      "command": "/ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh"
    }
  }
}
```

Copy [`mcp.json.example`](./mcp.json.example). Do not put a real home path in
the repo. After the service is up, first tool call should be
`silenthand_doctor`, then `get_app_state` (pass `disableDiff=true` for a full
AX tree), then actions. Host-side extras: `start_app` (Observed
`START_APP_TRANSPORT_UNAVAILABLE` — signed nativePipe is not implemented) and
`silenthand_pip` (Path B dry-run unless `live=true`), and
`silenthand_lock_screen_guardian` (official contract, does not spawn),
and `silenthand_turn_ended` (official CLI/IPC names, does not send a
payload). `list_apps` is the official sky mapper, not binary MCP text.
`event_stream_*` / `computer_history_*` / Messages MCP (`find_chats`,
`read_messages`, `search_messages`, `send_message`,
`count_message_activity`, `read_image`) are official extra servers when
those binaries answer; do not start a recorder or `send_message` unless
asked. Never `Raise`. Never `open -a`.

Agent skill (optional):

```bash
node cli.mjs install-skill                  # default hosts
node cli.mjs install-skill --platform hermes
node cli.mjs uninstall-skill --platform hermes
```

`install-skill` copies `skills/silenthand/` into agent skill dirs. It does not start the CUA service.

## Known Quirks

- **`element_index` must be string** — MCP binary treats numeric 0 as falsy. The REPL/client handle this automatically.
- **`click` with coordinates** may require the window to be accessible; pass `element_index` (string) for AX-based clicks that do not require foreground state. Do **NOT** use `open -a` — this violates the no-focus-steal rule and is not part of the CUA workflow.
- **Multi-word app names** — use quotes in REPL: `get_app_state "System Settings"`.
- **Elicitation** — service sends `elicitation/create`. The wrapper forwards it
  to the MCP host. Default without a host decision is **cancel**, not accept.
  Research harnesses (`silenthand-client.mjs`, `focus-guard.mjs`) still
  auto-accept. `SILENTHAND_AUTO_ACCEPT_ELICITATION=1` is test-only.
- **Chrome background AX** is chrome-only (page widgets unavailable).
- **WhatsApp empty composer** is not settable until it contains text. Do not
  `type_text` next to the Voice / mic button (opens a PTT draft).

## Architecture

- **Service**: `bin/macos/SilentHand CUA.app/Contents/MacOS/SkyComputerUseService` (copied by `fetch-runtime`)
- **Client**: `bin/macos/SilentHand CUA.app/Contents/SharedSupport/SkyComputerUseClient.app/Contents/MacOS/SkyComputerUseClient`
- **IPC Socket**: `~/Library/Group Containers/2DC432GLL2.com.openai.sky.CUAService/IPC/computeruse.sock`
- **Interpose dylib**: `native/interpose_auth.dylib` (built from `native/interpose_auth.c`)

## Critical Constraints

| Constraint | Reason |
|---|---|
| Info.plist / codesign id is `com.silenthand.cua`; in-process host checks still see `com.openai.sky.CUAService` | Same ChatGPT bundle id coalesced the Screen Recording row onto Codex Computer Use. `interpose_auth.c` spoofs the in-process id so the binary's host check still passes. |
| Binary names `SkyComputerUseService`/`SkyComputerUseClient` must stay | Load-bearing identifiers |
| Service needs persistent connection OR watchdog | Idle timeout causes exit, macOS respawns ChatGPT's copy |
| Launch via `./run-silenthand.sh` (`ditto` + `open -n -g` on `$HOME/Applications/SilentHand CUA.app`) | TCC is SilentHand CUA. Sequoia list row appears after the user clicks **Open System Settings** on that prompt (not Deny), then slides it On. Never Terminal / node / Hermes / Codex Computer Use |
| Do not publish this package to npm | Proprietary ChatGPT CUA binaries must not ship |
| Do not commit `bin/macos/*.app` or `native/*.node` | `./scripts/fetch-runtime.sh` copies them locally |

## Launcher Options

```bash
./run-silenthand.sh           # watchdog mode: auto-restarts on exit
./run-silenthand.sh --once    # one-shot: exits when service dies
./run-silenthand.sh &         # background with watchdog
```

## Permissions (TCC)

The trampoline (`native/launcher.c`) requests Screen Recording via `CGRequestScreenCaptureAccess()` **before** exec, while it is still `CFBundleExecutable` of **SilentHand CUA** (`com.silenthand.cua`). Do not grant Screen Recording to Terminal, node, Hermes, or ChatGPT’s **Codex Computer Use** row. Accessibility is also required for AX trees.

## Origin

Extracted from ChatGPT.app `Contents/Resources/cua_node/` (ChatGPT Computer Use). See [`docs/UPSTREAM_PROVENANCE.md`](docs/UPSTREAM_PROVENANCE.md). The host MCP, packaging CLI, agent skill, and interpose source in this repo are MIT. ChatGPT CUA binaries stay on your Mac.

## Status

Ready on **macOS** (arm64). **Windows** and **Linux** are in development — not a supported install path yet.

Research host. Not an OpenAI product. CUA binaries stay on your Mac via `fetch-runtime`. The MCP wrapper does not start the service. This can break on a ChatGPT.app update.

## License

MIT for the host in this repository. See [`LICENSE`](LICENSE) and [`NOTICE`](NOTICE). OpenAI’s CUA binaries, bundle IDs, and sockets are not ours and are not redistributed.