claude-codex-computer-use

Drive macOS apps from Claude Code through the Computer Use engine that ships with OpenAI's ChatGPT/Codex desktop app — without spending Codex quota, and without a model round trip per click.

Claude makes every decision. The engine only supplies the "eyes and hands" (accessibility tree, screenshot, click, type, keys). No OpenAI model is called.

Türkçe: README.tr.md · Changelog · Tool reference · Configuration · Architecture analysis (Turkish) · Research reports · Security · Contributing

Contents

• What it is

• Quick start

• How it works

• Tools

• Working rules

• Verified facts

• Head-to-head with Codex

• Testing

• Repository

Related MCP server: Desktop Pilot MCP

What it is

A dependency-free MCP server (Node ≥ 22.14) that sits in front of songkeys/claude-codex-computer-use — the bridge that launches the signed Computer Use client — and adds what you need to work fast and safely:

• script — a persistent JavaScript environment modelled on Codex's own cua_repl: loops, regex, conditions, zero model turns per click.

• batch — action lists with text targeting (find), wait_for, conditionals, macros, dry-run.

• Diff output by default — only the lines that changed since the last call; action replies reuse the tree they already carry.

• paste, key-name normalisation, named service errors (all 21 codes), app notes, stuck-menu recovery, screenshot downscaling, doctor diagnostics.

Same name as the bridge, different project: this repo wraps the bridge and does not modify it or the engine.

Quick start

Requirements: macOS 14.4+, the ChatGPT/Codex desktop app with Computer Use installed and running (the service lives under its codex app-server), Node ≥ 22.14, Claude Code.

git clone https://github.com/mehmetresatdemir/claude-codex-computer-use.git
cd claude-codex-computer-use
./scripts/install.sh        # finds paths, registers the MCP server "codex-computer-use", installs example macros, runs doctor

Start a new Claude Code session. The tools appear as mcp__codex-computer-use__*. Try:

Use the Computer Use `script` tool: open Calculator, type 37*41=, read the result with lastTextUnder.

Check the installation any time:

node server.mjs doctor

How it works

Claude Code ──MCP (stdio)──▶ codex-cua-plus (this repo)
                                │  batch · script · find · diff · notes · macros · paste · key names · error names
                                ▼
                     claude-codex-computer-use bridge (npx)
                                │  starts the signed client through `codex sandbox`
                                ▼
                     SkyComputerUseClient mcp ──Unix socket──▶ SkyComputerUseService (Swift, inside the ChatGPT app)
                                                                  accessibility tree · ScreenCaptureKit · CGEvent/AX
                                                                  per-app approval · "Esc to cancel" · settle wait

The service verifies the launcher's process ancestry; that is why the bridge is needed and why this wrapper never talks to the socket itself. Everything the service does — approvals, the forbidden-app list, the browser URL policy, the settle wait after each action — stays in force. Details: architecture analysis and the five research reports.

Tools

All of the client's tools pass through (get_app_state, click, press_key, type_text, set_value, scroll, drag, select_text, perform_secondary_action, list_apps) with these additions on each: include_screenshot (off by default, downscaled when on), output: diff|full, observe, find/role/nth text targeting, press_key.repeat, key-name normalisation (cmd+c → super+c, esc → Escape; the service wants X11 keysyms).

Full parameter reference: docs/TOOLS.md. Options: docs/CONFIGURATION.md.

Every service error code (-10000…-10020) and the plain-text replies (not approved, user changed, clipboard timeout, invalid secondary action, no window) come back with a name and what to do. App notes (built-ins for Freeform, TextEdit, Calculator, Finder, Unity, Simulator; _match notes for Open/Save panels; _screenshot apps) are attached once per session; extend them in notes.json.

Working rules (from the measurements)

1. Put predictable sequences in one script or batch; keep trees in variables; let the model see a diff at the end.

2. Don't sleep after actions — the service already waits (≈0.42 s + 50 ms polling). Use waitFor for a condition.

3. Send text with one type_text or paste (one settle wait per call, not per key). Prefer keys that don't change the UI (≈10 ms) over clicks (≈0.5 s) where both work.

4. If you see -10012/-10016, stop: the user is at the keyboard.

5. Canvas apps (Freeform) ignore synthetic drag: draw with the pen tool (app.pen), insert pictures as files, move items with shift+arrow (app.nudge).

Verified facts (how we know)

• No Codex quota is consumed. During live calls the bridge, client and service processes opened zero TCP connections (scripts/net_check.sh); the client talks to the local service over a Unix socket; the binaries contain telemetry and auth endpoints only, no model API.

• Why the bridge is needed. Launching SkyComputerUseClient directly fails with -10000: Sender process is not authenticated; the service checks the launcher's process ancestry (team id / signing id). The bridge starts the client through the signed codex sandbox launcher, which only executes a program and opens no model session.

• Where the time goes (measured inside the service with log stream, scripts/service_trace.py): every action is two IPC requests, a ~0 ms policy check and the action itself. A UI-changing action costs ≈ 0.42 s settle wait (polled at 50 ms, Statsig ui_settle_poll_interval_milliseconds) + ≈ 25 ms capture + ≈ 10 ms tree ≈ 0.45–0.56 s; the first action after get_app_state ≈ 0.9 s; modifier-only keys and no-op scrolls 1–3 ms with no capture; a plain observation ≈ 60 ms. The wait lives inside the signed service and cannot be tuned; what can be removed is the model round trip per action.

Head-to-head: the same task on Codex and on this wrapper

Task: in Freeform create a board, insert an image via the Open panel, add a sticky note and a text box and place them by keyboard; in Calculator compute 37×41 and read the result from the accessibility tree; in TextEdit create a plain-text document, write three lines, save it to a path via the Save panel; verify the window title; screenshot the board.

Script: examples/scripts/multi_app_task.js. Other examples: the 5-cube and 6-cube drawn with the pen tool in one call (examples/scripts/freeform_penteract.js, freeform_hexeract.js).

Testing

npm run check   # syntax
npm test        # 14 unit tests (lib/pure.mjs) + 11 integration tests against test/mock-bridge.mjs — no service needed

The mock bridge is a fake Computer Use client with an in-memory app: it exercises find targeting after actions, stuck-menu recovery, stale-index retry, macros, the script timeout, relayed elicitation requests and key normalisation. CI runs the suite on macOS with Node 22 and 24.

Repository

server.mjs                  MCP server: tool handlers, batch/script runtime, protocol loop, CLI (doctor, --help)
lib/upstream.mjs            bridge process, JSON-RPC link, relay of upstream-initiated requests
lib/tools.mjs               tool schemas and descriptions
lib/notes.mjs               app notes, <app_specific_instructions>
lib/pure.mjs                pure helpers (tree parsing, diff, find, key names, error table)
lib/config.mjs              options (environment / config.json)
test/                       unit + integration tests, mock bridge
scripts/install.sh          registration for Claude Code
scripts/service_trace.py    per-request settle/capture/tree timings from a `log stream` capture
scripts/bench.py            wrapper/bridge latency; scripts/net_check.sh: network check during live calls
scripts/gen-tool-docs.mjs   regenerates docs/TOOLS.md
examples/                   scripts and macros (multi-app task, 5-cube, 6-cube, insert image, write & save)
docs/                       architecture analysis, research reports, configuration, tool reference, day log

Credits

Bridge: songkeys/claude-codex-computer-use (MIT). The Computer Use engine belongs to OpenAI's ChatGPT/Codex macOS app; this repo doesn't modify it and doesn't bypass its authentication.

License: MIT.