claude-codex-computer-use

Drive macOS apps from Claude Code through the Computer Use engine that ships with OpenAI's ChatGPT/Codex desktop app — without spending Codex quota, and with batching that removes the per-click model round trip.

Türkçe: README.tr.md · Changelog: CHANGELOG.md · Architecture notes (Turkish): docs/codex-computer-use-mimarisi.md

Claude makes every decision. Codex only supplies the "eyes and hands" (screenshot + accessibility tree, click, type, keys). No OpenAI model is called.

What this is

A thin, dependency-free MCP server (server.mjs + lib/pure.mjs, Node ≥ 22.14, npm test for the pure parts) that sits in front of songkeys/claude-codex-computer-use (the bridge that launches the signed Computer Use client) and adds what you need to work fast:

• script — a persistent JavaScript environment modelled on Codex's own cua_repl: loops, regex, conditions, zero model turns per click.

• batch with text targeting (find), wait_for, conditionals, macros, dry-run.

• Diff output by default: only the lines that changed since the last call.

• App notes, stuck-menu recovery, named service error codes, screenshot downscaling, status/diagnostics.

Same name as the bridge, different project: this repo does not modify the bridge; it wraps it. The wrapper's tool/file name stays codex-cua-plus.

Related MCP server: Desktop Pilot MCP

Verified facts (how we know)

• No Codex quota is consumed. During live calls the bridge, client and service processes opened zero TCP connections (lsof -i, 10 s sampling, repeated across sessions); the client talks to the local service over a Unix socket; binaries contain only telemetry/feature-flag and auth/profile endpoints, no model API. Reproduce with scripts/net_check.sh.

• Why the bridge is needed. Launching SkyComputerUseClient directly fails with -10000: Sender process is not authenticated — the service checks the launcher's process ancestry. The bridge starts the client through the signed codex sandbox launcher inside ChatGPT.app. codex sandbox is only a launcher; it opens no model session.

• The engine is fast; the loop was slow. Measured inside the service with log stream (scripts/service_trace.py): every action is two IPC requests, a ~0 ms policy check and the action itself. A UI-changing action (click, Escape, type_text) costs ≈ 0.42 s settle wait (polled at 50 ms, Statsig ui_settle_poll_interval_milliseconds) + ≈ 25 ms ScreenCaptureKit capture + ≈ 10 ms accessibility tree, ≈ 0.45–0.56 s in total; the first action after get_app_state ≈ 0.9 s (window activation). Modifier-only keys and no-op scrolls return in 1–3 ms without a capture; a plain get_app_state ≈ 60 ms. The wait lives inside the signed service and cannot be tuned; what can be removed is the model round trip per action.

Head-to-head: the same task on Codex and on this wrapper

Task: in Freeform create a board, insert an image via the Open panel, add a sticky note and a text box and place them by keyboard; in Calculator compute 37×41 and read the result from the accessibility tree; in TextEdit create a plain-text document, write three lines, save it to a path via the Save panel; verify the window title; screenshot the board.

Script: examples/scripts/multi_app_task.js. Board: docs/example-multi-app-task.jpg.

Five research reports (JS library, 59 Codex sessions with 3 627 calls, service/client binaries, host layer, our own code review) are in docs/research/. Codex's own run was decoded from its session log (~/.codex/sessions/*.jsonl) and watched with process/network monitors; the architecture notes describe the layers (Swift service, Unix-socket IPC CodexComputerUseIPC-5, the @oai/sky REPL library, the model's contract: default AX diff, emit:false, implicit settle wait, per-app policy and approval, 21 named error codes).

Tools

All of Codex's tools pass through (get_app_state, click, press_key, type_text, set_value, scroll, drag, select_text, perform_secondary_action, list_apps) with these additions on each: include_screenshot (default off; downscaled to 1280 px when on), output: diff|full, observe (re-read the tree after an action when the reply has none), find/role/nth text targeting, press_key.repeat, key-name normalisation (cmd+c → super+c, esc → Escape; the service wants X11 keysyms).

All 21 service error codes (-10000…-10020) and the plain-text replies (not approved, user changed, clipboard timeout, invalid secondary action, no window) are annotated with a name and a hint (-10012 userStoppedSession: the user pressed Esc → loops stop; -10016 userIntervened → re-read; -10018 ambiguousApp → use the bundle id; -10005 is split into app-server exited → ChatGPT.app is launched automatically, and timeoutReached → it is not).

App notes (~/.codex-cua-plus/notes.json + built-ins for Freeform, TextEdit, Calculator, Finder, Unity, Simulator) are attached once per app; _match notes trigger on tree content (Open/Save panels), _screenshot lists apps that get a screenshot by default (Simulator, Unity). Example built-in: Freeform ignores synthetic drag; move a selected item with shift+arrow; clicking an image via accessibility opens Quick Look, select by coordinate instead.

Install

Requirements: macOS 14.4+, ChatGPT.app with Computer Use installed and running (the service hangs off its codex app-server), Node ≥ 22.14, Claude Code.

git clone https://github.com/mehmetresatdemir/claude-codex-computer-use.git
cd claude-codex-computer-use
./scripts/install.sh

The script finds Node 22, the signed codex launcher and SkyComputerUseClient inside ChatGPT.app, registers the wrapper as the user-scoped MCP server codex-computer-use, and installs the example macros. Start a new Claude Code session; tools appear as mcp__codex-computer-use__*.

Environment: CUA_PLUS_DEFAULT_OUTPUT (diff), CUA_PLUS_DEFAULT_SCREENSHOT (false), CUA_PLUS_SCREENSHOT_MAX_PX (1280), CUA_PLUS_JPEG_QUALITY (70), CUA_PLUS_APP_NAME (ChatGPT), CUA_PLUS_MACRO_DIR, CUA_PLUS_DEBUG=1; the bridge's own variables (COMPUTER_USE_BRIDGE_IDLE_TIMEOUT_MS etc.) pass through.

Working rules (from the measurements)

1. Put predictable sequences in one script or batch; keep trees in variables; let the model see a diff at the end.

2. Don't add fixed sleeps after actions — the service already waits. Use waitFor for a condition.

3. Prefer keys that don't change the UI (≈10 ms) over clicks (≈1 s) where both work (Return to confirm, shift+arrow to move).

4. If you see -10012/-10016, stop: the user is at the keyboard.

5. Send text with one type_text or paste (one settle wait per call, not per key). Key names are X11 keysyms; aliases are translated.

6. Canvas apps (Freeform) ignore synthetic drag. Draw with the pen tool by clicking points; insert pictures as files; move items with shift+arrow.

Repository

server.mjs                          the wrapper (no dependencies)
lib/pure.mjs                        pure helpers (tree parsing, diff, find, key names, error table); test/pure.test.mjs
scripts/install.sh                  finds paths, registers the MCP server, installs example macros
scripts/bench.py                    latency measurement against the bridge or the wrapper
scripts/net_check.sh                network check during live calls
scripts/service_trace.py            per-request settle/capture/tree timings from a `log stream` capture
examples/scripts/multi_app_task.js  Freeform + Calculator + TextEdit in one script call
examples/scripts/freeform_penteract.js  5-cube (80 edges) drawn with the pen tool in one call
examples/macros/*.json              freeform_insert, textedit_write_save
examples/freeform_insert_image.py   end-to-end driver without Claude (Python → wrapper)
docs/codex-computer-use-mimarisi.md architecture analysis of Codex Computer Use, incl. service internals, Statsig config, app-instruction catalogue (Turkish)
docs/gunluk-2026-10-01.md           day-one log, dead ends included (Turkish)
docs/research/                      five deep-dive reports (Turkish) with an English index

Credits

Bridge: songkeys/claude-codex-computer-use (MIT). The Computer Use engine belongs to OpenAI's ChatGPT/Codex macOS app; this repo doesn't modify it and doesn't bypass its authentication.

License: MIT.