🍏 macOS Computer Use (MCP Server)

An ultra-fast, native Model Context Protocol (MCP) server that gives LLMs and autonomous agents full visual inspection and desktop automation capabilities on macOS without hijacking your physical hardware mouse or keyboard focus.

Built with a compiled Swift core leveraging Apple's native CoreGraphics, Vision OCR (VNRecognizeTextRequest), and direct process-level event routing (CGEvent.postToPid).

⚡ Key Highlights

• 🎯 Sub-30ms Apple Vision OCR (find_text / click_text): No blind $(x, y)$ coordinate guessing. The agent finds buttons, labels, and text placeholders by name with pixel accuracy using Apple Neural Engine acceleration.

• 🛡️ Non-Intrusive Background Execution: Decoupled from global macOS hardware event taps. Clicks, scrolls, and keystrokes can be targeted directly to specific application PIDs while you continue working in your IDE or browser.

• 👁️ Virtual Cursor Overlay (mark_cursor): The agent's target coordinates are visually drawn onto screenshot frames (cyan target ring + magenta pulse dot) without moving your physical mouse on screen.

• ⚡ Reactive UI Synchronization (wait_for_text): Replaces blind sleep() timeouts with a high-speed OCR polling loop that resolves the instant dynamic autocomplete dropdowns, modals, or state changes appear.

• 🖼️ Window-Isolated Screen Capture: Captures target application buffers cleanly in the background (screencapture -l <wid> -o -x) without bringing the window to the front or capturing overlapping windows.

🛠️ Tools Exposed

🚀 Installation & Setup

1. Prerequisites

• macOS 13.0+ (Apple Silicon or Intel)

• Node.js 18+

• Xcode Command Line Tools (for compiling the native Swift binary):

  xcode-select --install

2. Clone & Build

git clone https://github.com/johnexzy/macos-computer-use.git
cd macos-computer-use
npm install

(The postinstall script automatically compiles native_helper.swift into an optimized native binary).

🔒 Required macOS Permissions

To allow desktop capture and input simulation, grant your host application (Terminal, iTerm2, Antigravity, Cursor, or VS Code) permissions in System Settings > Privacy & Security:

1. Screen & System Audio Recording: Required for screenshot and visual OCR.

2. Accessibility: Required for mouse_click, type_text, and press_key.

🔌 MCP Configuration

Add this server to your MCP client configuration:

Antigravity / Claude Desktop / Cursor / Windsurf

Add to your mcp_config.json / claude_desktop_config.json:

{
  "mcpServers": {
    "macos-computer-use": {
      "command": "node",
      "args": ["/absolute/path/to/macos-computer-use/server.mjs"]
    }
  }
}

🧪 Testing the Native Swift Helper Directly

You can test the native binary directly from your terminal:

# Display resolution and scaling
./native_helper size

# List active Chrome windows in Z-order
./native_helper list_windows "Google Chrome"

# Find a button or text on screen with Apple Vision OCR in ~30ms
./native_helper find_text "Search" "Google Chrome"

# Reactively wait for dynamic text to appear
./native_helper wait_for_text "quantum" "Google Chrome" nil 5.0

📄 License

MIT © John Oba