Lets agents observe and operate native macOS apps in the background via macOS Accessibility, reading windows, controls, text and selection as YAML (with optional window screenshots) and performing clicks, typing, scrolling, drags, hotkeys and Accessibility actions without taking over the pointer, keyboard or foreground app.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@opencomputeruse In Notes, add a to-do item called Buy Milk"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Background computer use for AI agents on macOS. Open Computer Use is a small menu-bar app and local MCP server that lets agents such as Claude Code observe and operate native Mac apps while you keep working: it does not take your pointer, your keyboard or the foreground app.
Licensed under the Apache License 2.0.
Early, pre-release software. There are no downloadable builds yet; build it from source. It requires macOS 15 or later and is tested on Apple silicon with a current macOS release; other versions and architectures are unvalidated. The name and app identifiers may still change.
Related MCP server: mcp-macos-cua
Observes an app through macOS Accessibility: windows, controls, text, selection and style, as compact JSON, plus an optional screenshot of the app's own windows.
Acts in the background on exact observed controls: click, double/triple and modifier clicks, right click, type, select text, scroll, drag, long press, hotkeys, open, and any Accessibility action a control advertises.
Keeps your input yours: keystrokes and clicks go only to the target app's process and window. The physical cursor never moves, and the foreground app only changes if you change it.
Runs several agents at once, one per app. An agent asking for an app
another agent is using gets busy.
Shows where agents act: each agent gets its own colored pointer, name and window highlight, drawn only above the window it works in, so it never covers other apps. Turn it off with Show agent cursors in the menu bar.
Shows when an agent is in control: while any agent holds an app, the menu-bar icon's ripples turn that agent's color and pulse.
Estimates token use: the menu bar shows roughly how many tokens each
agent's observations, screenshots and actions have cost it, and
ocu status reports the counts. These cover only Open Computer Use's part
of the conversation, not the agent's whole bill.
Stop in the menu bar stops every agent; nothing resumes until you choose Resume.
Requirements: macOS 15 or later on Apple silicon, Xcode command-line tools, Node.js and pnpm.
git clone https://github.com/andrewslabmd/opencomputeruse.git
cd opencomputeruse
pnpm install --frozen-lockfile
pnpm check
pnpm build
open -g "dist/Open Computer Use.app"pnpm build produces dist/Open Computer Use.app, signed with your Apple
Development certificate when the Mac has one, so permissions survive rebuilds.
Set OCU_SIGN_IDENTITY to choose another identity. Without either it is
signed ad hoc, and macOS asks for permissions again after each rebuild.
Then, in the menu-bar app, use Allow Accessibility… and Allow Screen
Recording… and turn on Open Computer Use in both lists (if it is already on
but still not allowed, choose Fix in Settings → Permissions), and
Install CLI to link ocu into ~/.local/bin. ocu permissions shows the
current state.
In the menu-bar app, open Settings… → Connect an Agent and choose Connect next to each coding agent found on your Mac. From a terminal:
ocu setup # each agent's status; asks which to connect
ocu setup codex # connect one: claude, codex, cursor or opencode
ocu setup cursor --remove
ocu setup --print # settings to add by hand to any other MCP clientEvery agent runs the same stdio server, ocu mcp, from wherever Open Computer
Use is installed. Claude Code and Codex are configured through their own mcp add commands (Claude Code at user scope); Cursor's ~/.cursor/mcp.json and
OpenCode's global opencode.json are edited in place, keeping other servers and
comments, with the previous file saved as *.ocu-backup. An agent connected to
another copy of the app shows Update. Restart an agent, or start a new
session, after connecting it. Give the agent the operating guidance in
integrations/claude-code/operator.md.
Tools: computer_use_list_apps, computer_use_observe,
computer_use_action_batch, computer_use_permissions,
computer_use_status, computer_use_release and report_issue.
report_issue saves Open Computer Use bug reports locally in
~/Library/Application Support/Open Computer Use/issue-reports.jsonl. Whether
the agent or you notice the bug, the agent asks once whether to share it with
the developers. Only after you say yes does it call the same tool with
{report_id, share: true} to send to the maintainers' private inbox for triage.
The preview contains the report, versions, Mac model/chip, agent client and
recent tool call names (including action kinds and outcomes), never call
arguments, results or content from apps or files. Text redaction is best effort;
leave out personal data. A failed send keeps the local copy. No link or form is
needed; GitHub issue templates remain available for people who want to file publicly.
OCU_REPORT_ENDPOINT can override the shared intake endpoint for testing.
The same operations are available from the command line:
ocu apps
ocu observe 'Application Name' --screenshot
ocu batch 'Application Name' '[{"kind":"click","at":{"x":100,"y":100}}]'
ocu stop
ocu resumeOpen Computer Use lets software operate your apps. Use it deliberately.
What it can do: read the content of any app an agent observes, and click, type and run commands in it, with your Accessibility and Screen Recording permissions. Treat anything an agent can do through it as something you did.
Where your data goes: observations and screenshots go to the agent that asked for them, and from there usually to that agent's model provider. Open Computer Use itself has no analytics. The only thing it sends is a bug report you agreed to share.
Trust boundary: the local host accepts clients that can read its token in your user account's private runtime directory. It does not protect against other software already running as your user.
Untrusted content: text inside apps (web pages, email, documents) is not an instruction. Agents must act on your actual request, and confirm actions such as sending, deleting, purchasing or changing accounts with you.
Stopping: Stop in the menu bar stops every agent. Quitting the app stops all of it. Already-dispatched actions may still complete; Stop is not undo.
Details: docs/POLICIES.md. Report security problems privately as described in SECURITY.md.
Apps whose interface exposes no Accessibility controls can only be operated through pointer and keystroke input, and results there are confirmed from screenshots rather than the Accessibility tree.
Context menus do not reliably close with AXCancel; Escape does.
Agents' native requests are processed one at a time, so a long observation by one agent briefly delays others.
pnpm check # headless contract and Swift tests; safe to run anywhere
pnpm build # app bundle in dist/
pnpm build:icon # regenerate the compiled icon from assets/icon (needs Xcode 26 or later)The native acceptance scripts are opt-in: they launch owned fixture apps in the background and send real input to them, so run them only on a Mac where that is acceptable, with the current build installed and running:
node scripts/test-native-clicks.mjs --interactive # clicks, actions, list_apps
node scripts/test-native-agents.mjs --interactive # two concurrent agents
node scripts/test-native-overlay.mjs --interactive # agent cursor overlay
node scripts/demo-agents.mjs --interactive # watch two agents work for a minute
node scripts/test-native-selection.mjs --interactive # selection in Finder and AppKit
node scripts/test-native-observation.mjs --interactive # large trees, lifecycle, Stop
node scripts/test-native-interaction.mjs --interactive # scroll, reveal, save panels
node scripts/test-native.mjs --interactive # basic native workflow
node scripts/test-claude.mjs --interactive # Claude Code end to endEach script keeps its evidence under the ignored work/ folder.
apps/macos: SwiftUI/AppKit menu-bar app.
packages/native: the signed helper: Accessibility, capture and input.
packages/engine: one engine per agent: observations, actions, verification.
packages/host: shared local host, workflows and app claims.
packages/protocol: framing and the canonical tool schemas.
packages/cli, packages/mcp: command-line and MCP adapters.
tests: contract tests, Swift tests and native fixtures.
upstream: provenance of the code extracted from OpenCode Superapp.
Product, architecture, policies, design and Claude Code integration.
See CONTRIBUTING.md and the code of conduct.
Apache License 2.0. See NOTICE and THIRD_PARTY_NOTICES.md.
This server cannot be deployed
Build and run agents and automations across hundreds of apps using natural language.
Securely control computers you explicitly pair through files, terminals, processes, screenshots, desktop UI/input, clipboard, browser automation, diagnostics, and document tools.
Build agents to automate any background task. Works with your ChatGPT/Claude subscription.
Let cloud agents talk to you: progress updates spoken on your Mac, and replies you send back.
