opencode-computer-use

An MCP server that gives opencode hands and eyes on macOS.

Screenshots · Mouse · Keyboard · Accessibility tree · App & window management

Let your agent launch the app, look at the screen, click the right button, type into the field, and verify the result.

Why

Testing an application by using it is the most honest test there is. This MCP server lets opencode drive your Mac the way a human tester would:

1. See — take a screenshot and reason about what's on screen

2. Target — read the accessibility tree to get exact coordinates and element values

3. Act — click, drag, scroll, type, press key combos, trigger menu items

4. Verify — screenshot again, or re-read the UI tree to confirm the result

Everything runs locally via a small Swift helper that talks to CoreGraphics (events, screen capture) and the macOS Accessibility API.

Related MCP server: macos-desktop-control

Requirements

• macOS (Apple Silicon or Intel)

• Xcode Command Line Tools (xcode-select --install) — for the Swift compiler

• Node.js 18+

• Two macOS permissions, granted to whatever app hosts the server (Terminal, iTerm, VS Code, ...):

Run the computer_permissions tool (or bin/computeruse perms --prompt) at any time to check status — it can open the right System Settings panes for you.

Install

git clone https://github.com/virajshoor/opencode-computer-use.git
cd opencode-computer-use
npm install && npm run build   # compiles Swift helper → bin/ and TypeScript → dist/

Configure opencode

Add to ~/.config/opencode/opencode.json:

{
  "mcp": {
    "computer-use": {
      "type": "local",
      "command": ["node", "/absolute/path/to/opencode-computer-use/dist/index.js"]
    }
  }
}

Restart opencode — you should see the computer_* tools appear.

Works with any MCP client (Claude Desktop, Cursor, etc.) — the server speaks standard MCP over stdio.

Tools (17)

Eyes

Hands

Eyes-on-the-inside

Control

The workflow

┌─────────────┐    ┌──────────────┐    ┌─────────────┐    ┌──────────────┐
│  screenshot  │───▶│ read_screen  │───▶│ click / type │───▶│  screenshot  │
│   (see it)   │    │ (exact x,y)  │    │   (do it)    │    │  (verify it) │
└─────────────┘    └──────────────┘    └─────────────┘    └──────────────┘

Example agent loop:

Agent: "Let me test that the settings save button works." computer_screenshot → sees the app · computer_read_screen → finds "Save" button at (812, 404) · computer_click → clicks it · computer_read_screen → confirms the "Saved ✓" toast appeared

How it works

opencode ──MCP/stdio──▶ dist/index.js ──exec──▶ bin/computeruse (Swift)
        TypeScript MCP server          ├── CGEvent: mouse & keyboard
        17 tools, zod-validated        ├── screencapture + sips: screenshots
                                       └── AXUIElement: accessibility tree

The Swift helper is a single-file CLI (swift/ComputerUse.swift, ~600 lines) — no external dependencies, no daemon. Each tool call is one short-lived process.

Testing it yourself

A minimal raw-protocol client is included:

node scripts/test-mcp.mjs        # drives TextEdit: launch → click → type → verify → cleanup

Security notes

• This tool gives an agent full control of your Mac. It can click, type, and read any app's UI. Only enable it for sessions where you're comfortable with that.

• All actions happen on your real user session with your real permissions. There is no sandbox.

• The computer_applescript tool is an arbitrary-code escape hatch — treat it accordingly.

• Nothing is sent anywhere except between opencode and the local server process.

Known limitations

• Menu traversal uses System Events (AppleScript) rather than raw AX — more reliable across macOS versions, but requires the host app to have Automation permission on first use.

• macOS 26's SwiftUI apps (e.g. Calculator) expose empty button titles in AX — use computer_screenshot + visual coordinates for those.

• Screenshots are downscaled to 1800px max (pass full-res intent through full-res in the CLI if you need every pixel).

License

MIT