silenthand

An unofficial research host for OpenAI Computer Use. The CUA binary already speaks MCP, but over newline JSON-RPC. IDE hosts speak Content-Length. mcp-wrapper.sh is the agent-facing entry.

This is not a published npm package. Do not npx silenthand. There is no Homebrew formula. The runtime needs proprietary ChatGPT CUA binaries that must not ship — clone the repo, copy them from a local ChatGPT.app, attach mcp-wrapper.sh.

This can break whenever ChatGPT updates CUA. Bundle IDs and binary names are load-bearing. Auth bypass is a local DYLD interpose for debugging.

Why this exists

ChatGPT's Computer Use service can drive local apps without a Window Server focus switch. Agents still need a host that speaks Content-Length MCP, a readiness check that does not auto-start anything, and a skill that forbids Raise / open -a.

Most wrappers start a daemon and claim success from ok=true. SilentHand does neither. silenthand_doctor is read-only. Re-observe after every action.

Related MCP server: macos-desktop-control

Quick start

macOS 14.4+ arm64, Node 24.14.0, ChatGPT for Mac already installed:

git clone https://github.com/edosulai/silenthand-cua.git
cd silenthand-cua
nvm use          # v24.14.0
npm install

# 1. Copy SkyComputerUse* from ChatGPT.app (gitignored — never committed):
./scripts/fetch-runtime.sh
#    or: node cli.mjs fetch-runtime

# 2. Read-only doctor. Does not start the service:
node cli.mjs doctor

# 3. Copy the agent skill (default: copilot, claude, codex, agents, hermes):
node cli.mjs setup --platform hermes

# 4. Start the CUA service only if you asked for it (watchdog):
./run-silenthand.sh

Hermes attach (local wrapper — there is no npx):

printf 'Y\n' | hermes mcp add silenthand --command /ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh

(non-TTY: the CLI prompts “Enable all tools?” and cancels on EOF unless you pipe Y). Verify with both hermes mcp list and hermes config get mcp_servers. Restart Hermes / /reload-mcp so this chat actually has the tools.

REPL / one-shot (after the service is up):

node silenthand-repl.mjs
node silenthand-client.mjs list_apps
node silenthand-client.mjs get_app_state Finder
node silenthand-client.mjs click Finder --x 200 --y 300

Prerequisites

• macOS 14.4+ (arm64)

• Node.js v24.14.0 (via nvm)

• ChatGPT for Mac (source of SkyComputerUseService / SkyComputerUseClient)

• System permissions: Screen Recording + Accessibility granted for the terminal app (e.g. VS Code, Terminal.app)

./scripts/fetch-runtime.sh looks at /Applications/ChatGPT.app by default. Override with SILENTHAND_CHATGPT_APP=/path/to/ChatGPT.app.

Available Tools

event_stream_* / computer_history_* are official extra servers when those binaries answer. Do not start a recorder unless asked.

REPL Usage

cua> list_apps
cua> get_app_state "System Settings"
cua> click "System Settings" --x 150 --y 400
cua> type_text Safari --text "hello world"
cua> press_key Safari --key Return
cua> scroll Safari --element_index 5 --direction down
cua> drag Finder --from_x 100 --from_y 200 --to_x 300 --to_y 400
cua> set_value "System Settings" --element_index 48 --value "Display"
cua> raw {"name":"click","arguments":{"app":"Finder","x":100,"y":200}}
cua> quit

How it works

MCP host (VS Code / Claude / Hermes)
  └─ Content-Length stdio → mcp-wrapper.sh → mcp-server.mjs
        ├─ silenthand_doctor / start_app / silenthand_pip /
        │  silenthand_lock_screen_guardian / silenthand_turn_ended (host-side)
        └─ NDJSON + elicitation proxy → SkyComputerUseClient
              └─ Unix socket → SkyComputerUseService
                    ↑ DYLD_INSERT_LIBRARIES=interpose_auth.dylib

The service appears to use SyntheticAppFocusEnforcer so a target can accept input without a Window Server focus switch. Inferred from binary analysis. Some Chrome routes were Observed to keep the agent’s editor frontmost. This is not a universal guarantee. Verify with NSWorkspace.shared.frontmostApplication, not Secondary Actions: Raise.

MCP setup

This is Computer Use for local apps, not a WhatsApp API. The wrapper does not start ./run-silenthand.sh. Call silenthand_doctor first. If ready=false, follow userMessage / agentNextSteps.

VS Code / Copilot (mcp.json):

{
  "servers": {
    "silenthand": {
      "type": "stdio",
      "command": "/ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh"
    }
  }
}

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "silenthand": {
      "command": "/ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh"
    }
  }
}

Copy mcp.json.example. Do not put a real home path in the repo. After the service is up, first tool call should be silenthand_doctor, then get_app_state (pass disableDiff=true for a full AX tree), then actions. Host-side extras: start_app (Observed START_APP_TRANSPORT_UNAVAILABLE — signed nativePipe is not implemented) and silenthand_pip (Path B dry-run unless live=true), and silenthand_lock_screen_guardian (official contract, does not spawn), and silenthand_turn_ended (official CLI/IPC names, does not send a payload). list_apps is the official sky mapper, not binary MCP text. event_stream_* / computer_history_* are official extra servers when those binaries answer; do not start a recorder unless asked. Never Raise. Never open -a.

Agent skill (optional):

node cli.mjs install-skill                  # default hosts
node cli.mjs install-skill --platform hermes
node cli.mjs uninstall-skill --platform hermes

install-skill copies skills/silenthand/ into agent skill dirs. It does not start the CUA service.

Known Quirks

• element_index must be string — MCP binary treats numeric 0 as falsy. The REPL/client handle this automatically.

• click with coordinates may require the window to be accessible; pass element_index (string) for AX-based clicks that do not require foreground state. Do NOT use open -a — this violates the no-focus-steal rule and is not part of the CUA workflow.

• Multi-word app names — use quotes in REPL: get_app_state "System Settings".

• Elicitation — service sends elicitation/create. The wrapper forwards it to the MCP host. Default without a host decision is cancel, not accept. Research harnesses (silenthand-client.mjs, focus-guard.mjs) still auto-accept. SILENTHAND_AUTO_ACCEPT_ELICITATION=1 is test-only.

• Chrome background AX is chrome-only (page widgets unavailable).

• WhatsApp empty composer is not settable until it contains text. Do not type_text next to the Voice / mic button (opens a PTT draft).

Architecture

• Service: bin/macos/SilentHand CUA.app/Contents/MacOS/SkyComputerUseService (copied by fetch-runtime)

• Client: bin/macos/SilentHand CUA.app/Contents/SharedSupport/SkyComputerUseClient.app/Contents/MacOS/SkyComputerUseClient

• IPC Socket: ~/Library/Group Containers/2DC432GLL2.com.openai.sky.CUAService/IPC/computeruse.sock

• Interpose dylib: native/interpose_auth.dylib (built from native/interpose_auth.c)

Critical Constraints

Launcher Options

./run-silenthand.sh           # watchdog mode: auto-restarts on exit
./run-silenthand.sh --once    # one-shot: exits when service dies
./run-silenthand.sh &         # background with watchdog

Permissions (TCC)

The service requests Screen Recording via CGRequestScreenCaptureAccess() in the interpose constructor. Whether it inherits the terminal's TCC grant or triggers its own prompt depends on the macOS TCC model and has not been fully verified. Ensure your terminal app has Screen Recording enabled, then check the service log for Screen Recording access: granted. Accessibility is also required for AX trees.

Origin

Extracted from ChatGPT.app Contents/Resources/cua_node/ (Codex Computer Use). See docs/UPSTREAM_PROVENANCE.md. The host MCP, packaging CLI, agent skill, and interpose source in this repo are MIT. ChatGPT CUA binaries stay on your Mac.

Status

Research host. macOS arm64 only. Not an OpenAI product. CUA binaries stay on your Mac via fetch-runtime. The MCP wrapper does not start the service. This can break on a ChatGPT.app update.

License

MIT for the host in this repository. See LICENSE and NOTICE. OpenAI’s CUA binaries, bundle IDs, and sockets are not ours and are not redistributed.