OpenComputer

OpenComputer gives AI agents a local interface for working with Mac apps in the background. Discover apps, read their accessibility trees, change controls, and capture a specific window. Connect it to an MCP client or use the JavaScript API directly. No OpenComputer account, API key, cloud service, or npm dependencies.

Watch it work

A 27-second showcase with narration, sound effects, and music. Click the preview to watch with sound.

Watch with sound · 27 seconds · Vertical social cut · Editable Hyperframes projects

Actual native UI with choreographed demo actions. Recording details.

Related MCP server: macos-desktop-control

Background control, visible activity

Inspired by the app-scoped workflow of ChatGPT Computer Use. This is an independent implementation using public macOS APIs. It is not a drop-in replacement for OpenAI's private runtime, and it does not provide ChatGPT's approval system or locked-Mac access.

Get started

macOS 14+ · Node.js 22+ · Apple Silicon or Intel

1. Download the runtime ZIP and extract it somewhere permanent. It includes the universal native helper; Node.js is installed separately.

2. In that folder, run node bin/opencomputer.mjs doctor. Approve Accessibility and Screen Recording for the launching host in System Settings → Privacy & Security, then restart that host if needed. Permission help.

3. Add the MCP server to your agent using its documented configuration. Start with one app you intend to automate:

{
  "mcpServers": {
    "opencomputer": {
      "command": "/absolute/path/to/node",
      "args": ["/absolute/path/to/opencomputer/bin/opencomputer.mjs", "serve"],
      "env": { "OPENCOMPUTER_APPS": "com.apple.TextEdit" }
    }
  }
}

Ask your agent to inspect a disposable TextEdit document. Have it read the current state before each action and check the result afterward. Empty allowlist means no app access. Use node bin/opencomputer.mjs apps to find bundle IDs. Set OPENCOMPUTER_APPS to * for explicit access to all running apps and their existing sessions; no VM or browser-profile copying.

Developer runtime: the native helper is ad-hoc signed, not Apple-notarized. macOS may block downloaded binaries. Follow the normal macOS approval flow or build from source; OpenComputer never disables Gatekeeper. This release is a developer runtime, not a graphical installer.

Checksums · Claude and Grok setup · Release notes

See what your agent is doing

A small three-dot animation sits at the top right while an agent has an app in use. Hover to expand it into AI icon → app icon, the agent/app names, current window or browser-tab title, and latest action. Move away to collapse it. Multiple agents get separate, stacked indicators. Icons come from installed apps.

The colored ghost cursor eases between targets and gently floats while idle. It appears at the agent’s target while that part of its window is visible. It follows window movement and disappears when covered, hidden, or minimized. Both overlays stay above ordinary windows, pass clicks through, and cannot become the active window. They never move your real pointer. The agent calls release when finished to remove them. Behavior and settings.

Work alongside your agent

Before each action, OpenComputer checks the target window against your current editing, held mouse gestures and keys, recent input, and pointer proximity. It waits when an action would conflict, with an amber indicator explaining why. It can continue in unrelated background windows, even another window of the same app. Nothing is queued for surprise replay. Signals, thresholds, and limits.

A familiar interface

import { createComputer } from './src/computer.mjs';

const computer = createComputer({ apps: ['com.apple.TextEdit'], agent: 'Claude' });
try {
  const app = computer.getApp('com.apple.TextEdit');
  const state = await app.getAXState();
  const field = state.nodes.find(node => node.settable && node.role === 'AXTextArea');
  if (!field) throw new Error('No writable text area in this window');
  await app.setValue(field.id, 'Hello from OpenComputer');
  console.log(await app.getAXState()); // verify the result
} finally {
  computer.close();
}

Snapshots expire after 60 seconds and are consumed by actions. Stale IDs fail with a clear error. Screenshots can return a window-selection list; select the intended window and request it again. Full API.

What works today

• Background semantic control: reading accessibility trees, AXPress, writable values, and supported secondary actions.

• Window capture: ScreenCaptureKit captures a selected window without capturing the whole desktop.

• Agent connections: local MCP over stdio, plus a JavaScript client with serialized requests and bounded timeouts.

• Text and scrolling: semantic value/selection replacement and accessibility page scrolling, where supported. Raw keyboard injection is disabled.

• Activity and focus protection: hover-to-expand dots, visible-target ghost cursor, window-aware interaction checks, and a session pause after detected focus changes.

Not implemented: arbitrary coordinate clicks/dragging, browser DOM automation, Windows/Linux, launching apps, and locked-screen operation. Canvas-only controls, minimized windows, secure fields, and apps with incomplete accessibility support may not work. OpenComputer never activates an app, sets keyboard focus, or restores the foreground window. An app can still activate itself in response to a semantic action; public macOS APIs cannot prevent every such transition. A detected change returns FOCUS_CHANGED and blocks further writes until reconnect. Compatibility and acceptance checks.

Quietly local

The server opens no network listener and sends no telemetry. It does not read browser profiles, store screenshots, or use the clipboard for text insertion. Your AI client may send tool output and screenshots to its model provider; that client's data policies apply. Accessible password values are redacted, but screenshots and ordinary text fields may still contain secrets.

The app allowlist is a process-level control, not a sandbox against an agent that can execute arbitrary shell commands. Your MCP client remains responsible for user consent, prompt-injection handling, and approval before consequential actions. Privacy · Security model

Tested with Claude and Grok

Both installed clients completed the native fixture task through OpenComputer 0.3.0: fill a field, enable an option, submit once, verify, capture a window, and release the indicators. Claude used 11 tool calls and Grok 14, with zero errors in their completed benchmark runs. Both preserved focus. Earlier Grok activity pauses and a turn-limit interruption are recorded alongside the passes. Results, methodology, and limitations.

Build, test, contribute

npm run build              # Swift helper, macOS only
npm test                   # protocol and client behavior
npm run check              # syntax, documentation links, release metadata
npm run fixture:build      # disposable native benchmark app
npm run test:integration   # real accessibility actions + window screenshot
npm run test:activity      # two agents, visibility, cleanup
npm run test:ui            # animated dots, hover, actual panel screenshots
npm run test:focus         # focus containment + two-window interaction protection
npm run test:interaction   # deterministic activity-conflict scenarios

Developer guide · Test coverage · Reproduce the AI benchmark · Report an issue