# Gemini macOS Computer Use MCP Server

[🇨🇳 简体中文文档](README_zh.md) | **[🇺🇸 English Documentation](README.md)**

A high-performance Model Context Protocol (MCP) server that empowers any AI agent with native macOS Computer Use capabilities, built with FastMCP and adhering to Google's Gemini Computer Use protocol.

---

## 🌟 What is this Project?

This server serves as the **"eyes and hands"** for AI models on macOS. Instead of having each AI model or agent framework reinvent screen capture, coordinate math, Retina scaling, and input synthesis, this project exposes a standard, battle-tested MCP interface. Any third-party agent client or SDK can connect to this server over local `stdio` (JSON-RPC) to inspect the screen and execute atomic user interactions (mouse clicks, movement, drags, text typing, shortcuts, and scrolling).

---

## 🏗️ Architecture

```text
+----------------------------------------------------------------------------------------------------+
|                                    Third-Party Agent Ecosystem                                     |
|  +--------------------------------+  +----------------------------------+  +--------------------+  |
|  | Desktop AI Clients             |  | Autonomous Agent Frameworks      |  | Custom / Local     |  |
|  | (Claude Desktop, Cursor, etc.) |  | (LangGraph, CrewAI, AutoGen)     |  | (Gemini SDK, Qwen) |  |
|  +--------------------------------+  +----------------------------------+  +--------------------+  |
+-------------------------------------------------+--------------------------------------------------+
                                                  |
                                                  | MCP Protocol (JSON-RPC over stdio)
                                                  v
+----------------------------------------------------------------------------------------------------+
|                                  gemini-mac-computer-use-mcp Server                                |
|                                                                                                    |
|  +----------------------------------------------------------------------------------------------+  |
|  | FastMCP Interface Layer (`server.py`)                                                         |  |
|  | - Atomic Tools: screenshot, mouse_click, move_cursor, drag, type_text, press_hotkey, scroll  |  |
|  | - Unified Adapter: computer_action (Gemini 0-1000 protocol parser & dispatcher)               |  |
|  +----------------------------------------------+-----------------------------------------------+  |
|                                                 |                                                  |
|                   +-----------------------------+-----------------------------+                    |
|                   v                                                           v                    |
|  +----------------------------------+               +-------------------------------------------+  |
|  | Display Engine (`screen.py`)     |               | Input Synthesizer (`input.py`)            |  |
|  | - Quartz display metrics         |               | - [0, 1000] -> Logical Points mapping     |  |
|  | - Retina scaling (backingScale)  |               | - PyAutoGUI mouse & drag simulation       |  |
|  | - JPEG/PNG compression & base64  |               | - Atomic Clipboard (pbcopy + Cmd+V) typing|  |
|  +----------------------------------+               +-------------------------------------------+  |
+-------------------------------------------------+--------------------------------------------------+
                                                  |
                                                  | Native macOS System Frameworks
                                                  v
+----------------------------------------------------------------------------------------------------+
| macOS System Services (CoreGraphics / Quartz, AppKit, NSPasteboard, Accessibility & ScreenCapture) |
+----------------------------------------------------------------------------------------------------+
```

---

## 🔄 Concise Interaction Flow

The interaction between third-party agents and this MCP server follows the classic **Observe ➔ Reason ➔ Act** loop:

```text
Third-Party Agent                     gemini-cu-mcp Server                   macOS System
       │                                       │                                  │
       │ 1. screenshot()                       │                                  │
       │──────────────────────────────────────>│ Capture physical pixels & Retina │
       │                                       │─────────────────────────────────>│
       │ 2. Image (JPEG) + Screen Metrics      │<─────────────────────────────────│
       │<──────────────────────────────────────│                                  │
       │                                       │                                  │
[Multimodal Vision Reasoning: locate button at (500, 320)]                                │
       │                                       │                                  │
       │ 3. mouse_click / type_text / action   │                                  │
       │──────────────────────────────────────>│ Normalize coords & clipboard pipe│
       │                                       │─────────────────────────────────>│
       │ 4. Execution confirmation (Success)   │    Inject native OS input events │
       │<──────────────────────────────────────│<─────────────────────────────────│
```

---

## 🔒 Security & Zero-API-Key Architecture

- **No Model API Keys Stored or Required**: This server is an **offline, local executor** running over local standard I/O (`stdio`). It contains **zero** AI SDK dependencies, never initiates outbound network connections, and does **not** store or require any LLM API keys.
- **Strict Decoupling**: API keys (e.g., Anthropic API keys, Google Gemini keys, OpenAI keys) belong entirely to the **third-party agent client** (such as Claude Desktop or your own agent script).
- **Zero Privacy Leakage Risk**: Pushing this repository to GitHub or running it in production poses no threat of credential or private key exposure.

---

## 🎯 Core Use Cases: Empowering Third-Party Agents

### 1. Plug-and-Play for Desktop AI Clients
- **Applicable Tools**: Claude Desktop, Cursor, Windsurf, Claude Code CLI.
- **Scenario**: Turn your everyday conversational AI into an autonomous macOS desktop operator with a single line of MCP configuration. You can ask Claude or Cursor in plain English to:
  - *"Open Safari, search for the latest quarterly earnings report of Company X, and save the chart to my desktop."*
  - *"Look at the Figma file open on my screen and copy the color codes into my project's CSS file."*

### 2. Autonomous Multi-Agent Frameworks & GUI Operators
- **Applicable Frameworks**: LangGraph, CrewAI, Microsoft AutoGen, LlamaIndex.
- **Scenario**: When building multi-agent systems, designate one specialized sub-agent as the **GUI Operator Agent**. That agent connects to `gemini-mac-computer-use-mcp` via standard MCP client libraries:
  - The framework does not need to handle macOS screen resolution nuances, 2x Retina backing store differences, or coordinate normalization.
  - The agent invokes `screenshot` to observe state, reasons about target coordinates, and calls atomic tools (`mouse_click`, `type_text`, `drag`) to complete long-horizon tasks across native desktop applications (e.g., Slack, Keynote, Excel, enterprise CRMs).

### 3. Custom Agent Loops & Google Gemini Computer Use Protocol
- **Applicable Tools**: Google GenAI SDK (Gemini 2.5 / 3.x with Computer Use capability), custom Python / Node.js agent loops.
- **Scenario**: Google Gemini's Computer Use protocol outputs structured JSON actions normalized to a `[0, 1000]` coordinate grid.
  - This server exposes a dedicated `computer_action` tool that directly parses Gemini action payloads (`click_at`, `type`, `drag_and_drop`, `scroll_at`, etc.).
  - Developers can build a complete, production-ready Gemini Computer Use agent loop on macOS in under 50 lines of code.

### 4. Local & 100% Offline Vision Agents (Maximum Privacy)
- **Applicable Tools**: Ollama, vLLM, local multimodal models (Qwen2-VL, Llama 3.2 Vision) paired with MCP-compatible clients.
- **Scenario**: For enterprise environments handling confidential intellectual property, code, or personal data:
  - Screenshots and UI input events remain strictly on the local machine.
  - The local multimodal model inspects desktop frames and sends action tool calls to `gemini-mac-computer-use-mcp` without a single byte of visual data leaving the private network.

---

## ⚡ Feature Highlights

- **Gemini 0–1000 Coordinate Normalization**:
  The coordinate space is fully normalized to `[0, 1000]` across both axes (`(0, 0)` is top-left, `(1000, 1000)` is bottom-right), independent of the monitor's physical resolution or aspect ratio.
- **macOS Retina 2x Scale Mapping**:
  Uses PyObjC (`AppKit.NSScreen` / `backingScaleFactor`) to query true display backing scale factors. Seamlessly translates between model normalized coordinates, logical points used by macOS windowing systems, and high-DPI physical pixels.
- **Clipboard Unicode & Chinese Typing**:
  Standard keystroke synthesizers fail on multi-byte Unicode, CJK characters, and emojis. This server automatically inspects typed text: pure ASCII is typed via keyboard events, while Unicode and Chinese text automatically uses an atomic clipboard pipeline (`NSPasteboard` / `pbcopy` + `Cmd+V`) for 100% fidelity without IME interference.
- **macOS Modifier Keys Translation**:
  Seamlessly maps cross-platform modifier names (`cmd`, `command`, `super`, `meta`, `win` -> `command`; `opt`, `option`, `alt` -> `option`; `ctrl`, `control` -> `ctrl`).
- **Fail-Safe & Boundary Protection**:
  Coordinates clamp safely within macOS screen boundaries, preventing edge coordinate exceptions while keeping PyAutoGUI fail-safe corner protection active.
- **Dual API Architecture**:
  Provides both fine-grained atomic FastMCP tools (`screenshot`, `mouse_click`, `type_text`, etc.) and a unified `computer_action` tool capable of parsing Gemini Computer Use JSON payloads.

---

## 🛠️ Available Tools Reference

| Tool Name | Parameters | Description |
|---|---|---|
| `screenshot` | `format: str = "JPEG"`, `quality: int = 80` | Captures the active macOS display. Returns image bytes (JPEG/PNG) and screen metrics metadata (logical size, physical size, Retina scale factor). |
| `mouse_click` | `x: int`, `y: int`, `button: str = "left"`, `click_type: str = "single"` | Moves to normalized coordinates `[0, 1000]` and clicks. Supports `button`: `left`, `right`, `middle`; `click_type`: `single`, `double`, `triple`. |
| `move_cursor` | `x: int`, `y: int`, `duration: float = 0.2` | Moves cursor smoothly to normalized coordinates `[0, 1000]` over the given duration (in seconds). |
| `mouse_down` | `x: int = None`, `y: int = None`, `button: str = "left"` | Presses and holds mouse button at normalized coordinates or current cursor position. |
| `mouse_up` | `x: int = None`, `y: int = None`, `button: str = "left"` | Releases held mouse button at normalized coordinates or current cursor position. |
| `drag` | `start_x: int`, `start_y: int`, `end_x: int`, `end_y: int`, `duration: float = 0.5`, `button: str = "left"` | Drags mouse from start coordinates to end coordinates. |
| `type_text` | `text: str`, `press_enter: bool = False` | Types text on keyboard. Supports English, Chinese, and Unicode characters via automatic clipboard fallback. |
| `press_hotkey` | `keys: List[str]` | Presses key combination simultaneously (e.g. `["command", "space"]`, `["cmd", "c"]`, `["ctrl", "alt", "t"]`). |
| `scroll` | `direction: str = "down"`, `amount: int = 5`, `x: int = None`, `y: int = None` | Scrolls wheel `up`, `down`, `left`, or `right` by step count. Optionally positions cursor first. |
| `wait` | `seconds: float = 1.0` | Pauses execution asynchronously for specified duration in seconds. |
| `computer_action` | `action: Dict[str, Any]` | Unified Gemini Computer Use protocol adapter. Accepts structured Gemini action payloads and dispatches to appropriate input methods. |

---

## 🔐 macOS Permissions Setup Guide

macOS security requires explicit user authorization for screen capture and synthetic input injection. Permissions must be granted to the host application running the server (e.g., **Terminal**, **iTerm2**, **Claude Desktop**, **Cursor**, or **VS Code**).

### 1. Accessibility (辅助功能)
*Required for mouse movements, clicking, drag-and-drop, and keystroke injection.*

1. Open **System Settings** (系统设置).
2. Navigate to **Privacy & Security** (隐私与安全性) ➔ **Accessibility** (辅助功能).
3. Toggle ON or click **+** to add:
   - Your terminal app (**Terminal**, **iTerm2**, **Ghostty**)
   - Your IDE (**Cursor**, **Visual Studio Code**, **Windsurf**)
   - **Claude Desktop** (if used as the MCP client)

### 2. Screen Recording (屏幕录制)
*Required for taking screen captures via the `screenshot` tool.*

1. Open **System Settings** (系统设置).
2. Navigate to **Privacy & Security** (隐私与安全性) ➔ **Screen Recording** (屏幕录制).
3. Ensure the switch is toggled ON for your terminal app, IDE, or **Claude Desktop**.
4. If newly added, restart the respective application for permissions to take effect.

> **Note:** The server performs automatic preflight permission checks on startup (`check_macos_permissions()`) and logs diagnostic warnings to `stderr` if permissions are missing.

---

## ⚙️ Client Configuration Guide

### Claude Desktop

Edit `~/Library/Application Support/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "gemini-cu-mcp": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/path/to/gemini-mac-computer-use-mcp",
        "gemini-cu-mcp"
      ]
    }
  }
}
```

*Replace `/path/to/gemini-mac-computer-use-mcp` with your actual repository path.*

### Claude Code CLI

Add the server to Claude Code using the CLI:

```bash
claude mcp add gemini-cu-mcp -- uv run --directory /path/to/gemini-mac-computer-use-mcp gemini-cu-mcp
```

### Cursor

1. Open Cursor **Settings** ➔ **Features** ➔ **MCP Servers**.
2. Click **Add New MCP Server**.
3. Configure the fields:
   - **Name**: `gemini-cu-mcp`
   - **Type**: `command`
   - **Command**: `uv run --directory /path/to/gemini-mac-computer-use-mcp gemini-cu-mcp`

### Windsurf

Open or create `~/.codeium/windsurf/mcp_config.json`:

```json
{
  "mcpServers": {
    "gemini-cu-mcp": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/path/to/gemini-mac-computer-use-mcp",
        "gemini-cu-mcp"
      ]
    }
  }
}
```

---

## 🚀 One-Click Deployment Script

You can install and deploy the server with a single `curl` command. The script automatically checks macOS compatibility, provisions the environment, verifies the test suite, performs permission preflight checks, and configures Claude Desktop:

### Quick Install via `curl`

```bash
# Interactive installation (prompts before modifying Claude Desktop config)
curl -fsSL https://raw.githubusercontent.com/Be-Young/gemini-mac-computer-use-mcp/main/deploy.sh | bash

# Unattended installation (automatically configures Claude Desktop without prompting)
curl -fsSL https://raw.githubusercontent.com/Be-Young/gemini-mac-computer-use-mcp/main/deploy.sh | bash -s -- --yes

# Skip Claude Desktop configuration
curl -fsSL https://raw.githubusercontent.com/Be-Young/gemini-mac-computer-use-mcp/main/deploy.sh | bash -s -- --skip-config
```

### Or Run from Cloned Repository

```bash
# Interactive run
./deploy.sh

# Automatically authorize and configure Claude Desktop
./deploy.sh --yes

# Skip Claude Desktop configuration
./deploy.sh --skip-config
```

---

## 🧪 Manual Run and Developer Guide

### Prerequisites
- macOS 12.0+ (Apple Silicon or Intel)
- Python 3.11+
- [uv](https://docs.astral.sh/uv/) package manager

### 1. Install Dependencies
```bash
uv sync
```

### 2. Run Test Suite
Verify that all unit tests and integration mocks pass:
```bash
uv run pytest -v
```

### 3. Run the Server Manually
Start the MCP server over standard input/output (stdio):
```bash
uv run gemini-cu-mcp
```

### 4. Interactive Testing with MCP Inspector
Test each tool interactively in your browser using the MCP Inspector:
```bash
npx @modelcontextprotocol/inspector uv run gemini-cu-mcp
```

---

## ❓ Troubleshooting

- **Permission Denied / Events Not Injected**:
  Verify Accessibility permissions in macOS System Settings. If permissions were granted previously but stopped working after a terminal update, toggle the permission off and on again, or reset via:
  ```bash
  tccutil reset Accessibility
  ```
- **Blank or Black Screenshot**:
  Grant Screen Recording permissions to your terminal or client application:
  ```bash
  tccutil reset ScreenCapture
  ```
- **uv command not found**:
  Install uv via Homebrew or the official installer:
  ```bash
  curl -LsSf https://astral.sh/uv/install.sh | sh
  ```

---

## 📄 License

MIT License. See [LICENSE](LICENSE) for details.