<div align="center">

# opencode-computer-use

**An MCP server that gives [opencode](https://opencode.ai) hands and eyes on macOS.**

Screenshots · Mouse · Keyboard · Accessibility tree · App & window management

*Let your agent launch the app, look at the screen, click the right button, type into the field, and verify the result.*

</div>

---

## Why

Testing an application by *using* it is the most honest test there is. This MCP server lets opencode drive your Mac the way a human tester would:

1. **See** — take a screenshot and reason about what's on screen
2. **Target** — read the accessibility tree to get exact coordinates and element values
3. **Act** — click, drag, scroll, type, press key combos, trigger menu items
4. **Verify** — screenshot again, or re-read the UI tree to confirm the result

Everything runs locally via a small Swift helper that talks to CoreGraphics (events, screen capture) and the macOS Accessibility API.

## Requirements

- macOS (Apple Silicon or Intel)
- [Xcode Command Line Tools](https://developer.apple.com/xcode/) (`xcode-select --install`) — for the Swift compiler
- Node.js 18+
- Two macOS permissions, granted to whatever app hosts the server (Terminal, iTerm, VS Code, ...):

| Permission | Used for | Where to grant |
|---|---|---|
| **Accessibility** | Mouse clicks, keyboard events, accessibility tree | System Settings → Privacy & Security → Accessibility |
| **Screen Recording** | Screenshots, window titles | System Settings → Privacy & Security → Screen Recording |

> Run the `computer_permissions` tool (or `bin/computeruse perms --prompt`) at any time to check status — it can open the right System Settings panes for you.

## Install

```bash
git clone https://github.com/virajshoor/opencode-computer-use.git
cd opencode-computer-use
npm install && npm run build   # compiles Swift helper → bin/ and TypeScript → dist/
```

## Configure opencode

Add to `~/.config/opencode/opencode.json`:

```json
{
  "mcp": {
    "computer-use": {
      "type": "local",
      "command": ["node", "/absolute/path/to/opencode-computer-use/dist/index.js"]
    }
  }
}
```

Restart opencode — you should see the `computer_*` tools appear.

Works with any MCP client (Claude Desktop, Cursor, etc.) — the server speaks standard MCP over stdio.

## Tools (17)

### Eyes

| Tool | What it does |
|---|---|
| `computer_screenshot` | Capture screen / region / display, returned as an image the model can see (auto-downscaled, JPEG) |
| `computer_screeninfo` | Connected displays with bounds |
| `computer_permissions` | Check Accessibility & Screen Recording status, optionally open System Settings |

### Hands

| Tool | What it does |
|---|---|
| `computer_click` | Move + click (left/right/middle, single/double/triple) |
| `computer_move` | Mouse positioning / hover |
| `computer_drag` | Smooth press-drag-release (sliders, DnD, text selection, window moves) |
| `computer_scroll` | Wheel events at a point (positive dy = down, dx = right) |
| `computer_type` | Unicode text typing into the focused element |
| `computer_key` | Key combos: `cmd+c`, `return`, `escape`, arrows, F-keys, repeat |

### Eyes-on-the-inside

| Tool | What it does |
|---|---|
| `computer_read_screen` | Full accessibility tree of any app — roles, titles, values, frames. The precise-targeting superpower |
| `computer_element_at` | What UI element lives at (x, y)? Check before you click |
| `computer_list_windows` | On-screen windows with app, title and bounds |
| `computer_list_apps` | Running GUI apps with pids and frontmost state |

### Control

| Tool | What it does |
|---|---|
| `computer_app` | Launch / activate / quit an app by name or bundle id |
| `computer_menu` | Click any menu path (`['File', 'Export As PDF']`) via System Events |
| `computer_applescript` | Escape hatch — full AppleScript for anything else (dialogs, UI scripting) |
| `computer_wait` | Sleep between steps |

## The workflow

```
┌─────────────┐    ┌──────────────┐    ┌─────────────┐    ┌──────────────┐
│  screenshot  │───▶│ read_screen  │───▶│ click / type │───▶│  screenshot  │
│   (see it)   │    │ (exact x,y)  │    │   (do it)    │    │  (verify it) │
└─────────────┘    └──────────────┘    └─────────────┘    └──────────────┘
```

Example agent loop:

> **Agent:** *"Let me test that the settings save button works."*
> `computer_screenshot` → sees the app ·
> `computer_read_screen` → finds "Save" button at (812, 404) ·
> `computer_click` → clicks it ·
> `computer_read_screen` → confirms the "Saved ✓" toast appeared

## How it works

```
opencode ──MCP/stdio──▶ dist/index.js ──exec──▶ bin/computeruse (Swift)
        TypeScript MCP server          ├── CGEvent: mouse & keyboard
        17 tools, zod-validated        ├── screencapture + sips: screenshots
                                       └── AXUIElement: accessibility tree
```

The Swift helper is a single-file CLI (`swift/ComputerUse.swift`, ~600 lines) — no external dependencies, no daemon. Each tool call is one short-lived process.

## Testing it yourself

A minimal raw-protocol client is included:

```bash
node scripts/test-mcp.mjs        # drives TextEdit: launch → click → type → verify → cleanup
```

## Security notes

- **This tool gives an agent full control of your Mac.** It can click, type, and read any app's UI. Only enable it for sessions where you're comfortable with that.
- All actions happen on your real user session with your real permissions. There is no sandbox.
- The `computer_applescript` tool is an arbitrary-code escape hatch — treat it accordingly.
- Nothing is sent anywhere except between opencode and the local server process.

## Known limitations

- Menu traversal uses System Events (AppleScript) rather than raw AX — more reliable across macOS versions, but requires the host app to have Automation permission on first use.
- macOS 26's SwiftUI apps (e.g. Calculator) expose empty button titles in AX — use `computer_screenshot` + visual coordinates for those.
- Screenshots are downscaled to 1800px max (pass full-res intent through `full-res` in the CLI if you need every pixel).

## License

MIT

---

<div align="center">
Built for testing apps the way users actually use them — by clicking around.
</div>