silenthand

An unofficial research host for OpenAI Computer Use. The CUA binary already speaks MCP, but over newline JSON-RPC. IDE hosts speak Content-Length. mcp-wrapper.sh is the agent-facing entry.

This is not a published npm package. Do not npx silenthand. There is no Homebrew formula. The runtime needs proprietary ChatGPT CUA binaries that must not ship — clone the repo, copy them from a local ChatGPT.app, attach mcp-wrapper.sh.

This can break whenever ChatGPT updates CUA. Bundle IDs and binary names are load-bearing. Auth bypass is a local DYLD interpose for debugging.

Why this exists

ChatGPT's Computer Use service can drive local apps without a Window Server focus switch. Agents still need a host that speaks Content-Length MCP, a readiness check that does not auto-start anything, and a skill that forbids Raise / open -a.

Most wrappers start a daemon and claim success from ok=true. SilentHand does neither. silenthand_doctor is read-only. Re-observe after every action.

Related MCP server: macos-desktop-control

Quick start

Ready on macOS 14.4+ arm64. Windows and Linux are in development — do not treat this checkout as an install path there. Node 24.14.0, ChatGPT for Mac already installed:

git clone https://github.com/edosulai/silenthand-cua.git
cd silenthand-cua
nvm use          # v24.14.0
npm install

# 1. Copy SkyComputerUse* from ChatGPT.app (gitignored — never committed):
./scripts/fetch-runtime.sh
#    or: node cli.mjs fetch-runtime

# 2. Read-only doctor. Does not start the service:
node cli.mjs doctor

# 3. Copy the agent skill and attach Hermes MCP (no hermes mcp add, no npx):
node cli.mjs setup --platform hermes

# 4. Start SilentHand CUA.app via LaunchServices (TCC = this app, not Terminal):
./run-silenthand.sh

setup --platform hermes copies ~/.hermes/skills/silenthand/ and attaches mcp-wrapper.sh. Restart Hermes / /reload-mcp so this chat has mcp_silenthand_* tools. There is no npx.

REPL / one-shot (after the service is up):

node silenthand-repl.mjs
node silenthand-client.mjs list_apps
node silenthand-client.mjs get_app_state Finder
node silenthand-client.mjs click Finder --x 200 --y 300

Prerequisites

• macOS 14.4+ (arm64)

• Node.js v24.14.0 (via nvm)

• ChatGPT for Mac (source of SkyComputerUseService / SkyComputerUseClient)

• System permissions: Screen Recording + Accessibility granted for SilentHand CUA.app (LaunchServices open -n -g on the app path — not Terminal, not node, not Hermes)

./scripts/fetch-runtime.sh looks at /Applications/ChatGPT.app by default. Override with SILENTHAND_CHATGPT_APP=/path/to/ChatGPT.app.

Available Tools

event_stream_* / computer_history_* are official extra servers when those binaries answer. Do not start a recorder unless asked.

REPL Usage

cua> list_apps
cua> get_app_state "System Settings"
cua> click "System Settings" --x 150 --y 400
cua> type_text Safari --text "hello world"
cua> press_key Safari --key Return
cua> scroll Safari --element_index 5 --direction down
cua> drag Finder --from_x 100 --from_y 200 --to_x 300 --to_y 400
cua> set_value "System Settings" --element_index 48 --value "Display"
cua> raw {"name":"click","arguments":{"app":"Finder","x":100,"y":200}}
cua> quit

How it works

MCP host (VS Code / Claude / Hermes)
  └─ Content-Length stdio → mcp-wrapper.sh → mcp-server.mjs
        ├─ silenthand_doctor / start_app / silenthand_pip /
        │  silenthand_lock_screen_guardian / silenthand_turn_ended (host-side)
        └─ NDJSON + elicitation proxy → SkyComputerUseClient
              └─ Unix socket → SkyComputerUseService
                    ↑ LaunchServices open -n -g SilentHand CUA.app
                      trampoline DYLD-inserts in-bundle interpose_auth.dylib

The service appears to use SyntheticAppFocusEnforcer so a target can accept input without a Window Server focus switch. Inferred from binary analysis. Some Chrome routes were Observed to keep the agent’s editor frontmost. This is not a universal guarantee. Verify with NSWorkspace.shared.frontmostApplication, not Secondary Actions: Raise.

MCP setup

This is Computer Use for local apps, not a WhatsApp API. The wrapper does not start ./run-silenthand.sh. Call silenthand_doctor first. If ready=false, follow userMessage / agentNextSteps.

VS Code / Copilot (mcp.json):

{
  "servers": {
    "silenthand": {
      "type": "stdio",
      "command": "/ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh"
    }
  }
}

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "silenthand": {
      "command": "/ABSOLUTE/PATH/TO/silenthand-cua/mcp-wrapper.sh"
    }
  }
}

Copy mcp.json.example. Do not put a real home path in the repo. After the service is up, first tool call should be silenthand_doctor, then get_app_state (pass disableDiff=true for a full AX tree), then actions. Host-side extras: start_app (Observed START_APP_TRANSPORT_UNAVAILABLE — signed nativePipe is not implemented) and silenthand_pip (Path B dry-run unless live=true), and silenthand_lock_screen_guardian (official contract, does not spawn), and silenthand_turn_ended (official CLI/IPC names, does not send a payload). list_apps is the official sky mapper, not binary MCP text. event_stream_* / computer_history_* / Messages MCP (find_chats, read_messages, search_messages, send_message, count_message_activity, read_image) are official extra servers when those binaries answer; do not start a recorder or send_message unless asked. Never Raise. Never open -a.

Agent skill (optional):

node cli.mjs install-skill                  # default hosts
node cli.mjs install-skill --platform hermes
node cli.mjs uninstall-skill --platform hermes

install-skill copies skills/silenthand/ into agent skill dirs. It does not start the CUA service.

Known Quirks

• element_index must be string — MCP binary treats numeric 0 as falsy. The REPL/client handle this automatically.

• click with coordinates may require the window to be accessible; pass element_index (string) for AX-based clicks that do not require foreground state. Do NOT use open -a — this violates the no-focus-steal rule and is not part of the CUA workflow.

• Multi-word app names — use quotes in REPL: get_app_state "System Settings".

• Elicitation — service sends elicitation/create. The wrapper forwards it to the MCP host. Default without a host decision is cancel, not accept. Research harnesses (silenthand-client.mjs, focus-guard.mjs) still auto-accept. SILENTHAND_AUTO_ACCEPT_ELICITATION=1 is test-only.

• Chrome background AX is chrome-only (page widgets unavailable).

• WhatsApp empty composer is not settable until it contains text. Do not type_text next to the Voice / mic button (opens a PTT draft).

Architecture

• Service: bin/macos/SilentHand CUA.app/Contents/MacOS/SkyComputerUseService (copied by fetch-runtime)

• Client: bin/macos/SilentHand CUA.app/Contents/SharedSupport/SkyComputerUseClient.app/Contents/MacOS/SkyComputerUseClient

• IPC Socket: ~/Library/Group Containers/2DC432GLL2.com.openai.sky.CUAService/IPC/computeruse.sock

• Interpose dylib: native/interpose_auth.dylib (built from native/interpose_auth.c)

Critical Constraints

Launcher Options

./run-silenthand.sh           # watchdog mode: auto-restarts on exit
./run-silenthand.sh --once    # one-shot: exits when service dies
./run-silenthand.sh &         # background with watchdog

Permissions (TCC)

The trampoline (native/launcher.c) requests Screen Recording via CGRequestScreenCaptureAccess() before exec, while it is still CFBundleExecutable of SilentHand CUA (com.silenthand.cua). Do not grant Screen Recording to Terminal, node, Hermes, or ChatGPT’s Codex Computer Use row. Accessibility is also required for AX trees.

Origin

Extracted from ChatGPT.app Contents/Resources/cua_node/ (ChatGPT Computer Use). See docs/UPSTREAM_PROVENANCE.md. The host MCP, packaging CLI, agent skill, and interpose source in this repo are MIT. ChatGPT CUA binaries stay on your Mac.

Status

Ready on macOS (arm64). Windows and Linux are in development — not a supported install path yet.

Research host. Not an OpenAI product. CUA binaries stay on your Mac via fetch-runtime. The MCP wrapper does not start the service. This can break on a ChatGPT.app update.

License

MIT for the host in this repository. See LICENSE and NOTICE. OpenAI’s CUA binaries, bundle IDs, and sockets are not ours and are not redistributed.