Gemini macOS Computer Use MCP Server

🇨🇳 简体中文文档 | 🇺🇸 English Documentation

A high-performance Model Context Protocol (MCP) server that empowers any AI agent with native macOS Computer Use capabilities, built with FastMCP and adhering to Google's Gemini Computer Use protocol.

🌟 What is this Project?

This server serves as the "eyes and hands" for AI models on macOS. Instead of having each AI model or agent framework reinvent screen capture, coordinate math, Retina scaling, and input synthesis, this project exposes a standard, battle-tested MCP interface. Any third-party agent client or SDK can connect to this server over local stdio (JSON-RPC) to inspect the screen and execute atomic user interactions (mouse clicks, movement, drags, text typing, shortcuts, and scrolling).

🏗️ Architecture

+----------------------------------------------------------------------------------------------------+
|                                    Third-Party Agent Ecosystem                                     |
|  +--------------------------------+  +----------------------------------+  +--------------------+  |
|  | Desktop AI Clients             |  | Autonomous Agent Frameworks      |  | Custom / Local     |  |
|  | (Claude Desktop, Cursor, etc.) |  | (LangGraph, CrewAI, AutoGen)     |  | (Gemini SDK, Qwen) |  |
|  +--------------------------------+  +----------------------------------+  +--------------------+  |
+-------------------------------------------------+--------------------------------------------------+
                                                  |
                                                  | MCP Protocol (JSON-RPC over stdio)
                                                  v
+----------------------------------------------------------------------------------------------------+
|                                  gemini-mac-computer-use-mcp Server                                |
|                                                                                                    |
|  +----------------------------------------------------------------------------------------------+  |
|  | FastMCP Interface Layer (`server.py`)                                                         |  |
|  | - Atomic Tools: screenshot, mouse_click, move_cursor, drag, type_text, press_hotkey, scroll  |  |
|  | - Unified Adapter: computer_action (Gemini 0-1000 protocol parser & dispatcher)               |  |
|  +----------------------------------------------+-----------------------------------------------+  |
|                                                 |                                                  |
|                   +-----------------------------+-----------------------------+                    |
|                   v                                                           v                    |
|  +----------------------------------+               +-------------------------------------------+  |
|  | Display Engine (`screen.py`)     |               | Input Synthesizer (`input.py`)            |  |
|  | - Quartz display metrics         |               | - [0, 1000] -> Logical Points mapping     |  |
|  | - Retina scaling (backingScale)  |               | - PyAutoGUI mouse & drag simulation       |  |
|  | - JPEG/PNG compression & base64  |               | - Atomic Clipboard (pbcopy + Cmd+V) typing|  |
|  +----------------------------------+               +-------------------------------------------+  |
+-------------------------------------------------+--------------------------------------------------+
                                                  |
                                                  | Native macOS System Frameworks
                                                  v
+----------------------------------------------------------------------------------------------------+
| macOS System Services (CoreGraphics / Quartz, AppKit, NSPasteboard, Accessibility & ScreenCapture) |
+----------------------------------------------------------------------------------------------------+

🔄 Concise Interaction Flow

The interaction between third-party agents and this MCP server follows the classic Observe ➔ Reason ➔ Act loop:

Third-Party Agent                     gemini-cu-mcp Server                   macOS System
       │                                       │                                  │
       │ 1. screenshot()                       │                                  │
       │──────────────────────────────────────>│ Capture physical pixels & Retina │
       │                                       │─────────────────────────────────>│
       │ 2. Image (JPEG) + Screen Metrics      │<─────────────────────────────────│
       │<──────────────────────────────────────│                                  │
       │                                       │                                  │
[Multimodal Vision Reasoning: locate button at (500, 320)]                                │
       │                                       │                                  │
       │ 3. mouse_click / type_text / action   │                                  │
       │──────────────────────────────────────>│ Normalize coords & clipboard pipe│
       │                                       │─────────────────────────────────>│
       │ 4. Execution confirmation (Success)   │    Inject native OS input events │
       │<──────────────────────────────────────│<─────────────────────────────────│

🔒 Security & Zero-API-Key Architecture

• No Model API Keys Stored or Required: This server is an offline, local executor running over local standard I/O (stdio). It contains zero AI SDK dependencies, never initiates outbound network connections, and does not store or require any LLM API keys.

• Strict Decoupling: API keys (e.g., Anthropic API keys, Google Gemini keys, OpenAI keys) belong entirely to the third-party agent client (such as Claude Desktop or your own agent script).

• Zero Privacy Leakage Risk: Pushing this repository to GitHub or running it in production poses no threat of credential or private key exposure.

🎯 Core Use Cases: Empowering Third-Party Agents

1. Plug-and-Play for Desktop AI Clients

• Applicable Tools: Claude Desktop, Cursor, Windsurf, Claude Code CLI.

• Scenario: Turn your everyday conversational AI into an autonomous macOS desktop operator with a single line of MCP configuration. You can ask Claude or Cursor in plain English to:

  • "Open Safari, search for the latest quarterly earnings report of Company X, and save the chart to my desktop."

  • "Look at the Figma file open on my screen and copy the color codes into my project's CSS file."

2. Autonomous Multi-Agent Frameworks & GUI Operators

• Applicable Frameworks: LangGraph, CrewAI, Microsoft AutoGen, LlamaIndex.

• Scenario: When building multi-agent systems, designate one specialized sub-agent as the GUI Operator Agent. That agent connects to gemini-mac-computer-use-mcp via standard MCP client libraries:

  • The framework does not need to handle macOS screen resolution nuances, 2x Retina backing store differences, or coordinate normalization.

  • The agent invokes screenshot to observe state, reasons about target coordinates, and calls atomic tools (mouse_click, type_text, drag) to complete long-horizon tasks across native desktop applications (e.g., Slack, Keynote, Excel, enterprise CRMs).

3. Custom Agent Loops & Google Gemini Computer Use Protocol

• Applicable Tools: Google GenAI SDK (Gemini 2.5 / 3.x with Computer Use capability), custom Python / Node.js agent loops.

• Scenario: Google Gemini's Computer Use protocol outputs structured JSON actions normalized to a [0, 1000] coordinate grid.

  • This server exposes a dedicated computer_action tool that directly parses Gemini action payloads (click_at, type, drag_and_drop, scroll_at, etc.).

  • Developers can build a complete, production-ready Gemini Computer Use agent loop on macOS in under 50 lines of code.

4. Local & 100% Offline Vision Agents (Maximum Privacy)

• Applicable Tools: Ollama, vLLM, local multimodal models (Qwen2-VL, Llama 3.2 Vision) paired with MCP-compatible clients.

• Scenario: For enterprise environments handling confidential intellectual property, code, or personal data:

  • Screenshots and UI input events remain strictly on the local machine.

  • The local multimodal model inspects desktop frames and sends action tool calls to gemini-mac-computer-use-mcp without a single byte of visual data leaving the private network.

⚡ Feature Highlights

• Gemini 0–1000 Coordinate Normalization: The coordinate space is fully normalized to [0, 1000] across both axes ((0, 0) is top-left, (1000, 1000) is bottom-right), independent of the monitor's physical resolution or aspect ratio.

• macOS Retina 2x Scale Mapping: Uses PyObjC (AppKit.NSScreen / backingScaleFactor) to query true display backing scale factors. Seamlessly translates between model normalized coordinates, logical points used by macOS windowing systems, and high-DPI physical pixels.

• Clipboard Unicode & Chinese Typing: Standard keystroke synthesizers fail on multi-byte Unicode, CJK characters, and emojis. This server automatically inspects typed text: pure ASCII is typed via keyboard events, while Unicode and Chinese text automatically uses an atomic clipboard pipeline (NSPasteboard / pbcopy + Cmd+V) for 100% fidelity without IME interference.

• macOS Modifier Keys Translation: Seamlessly maps cross-platform modifier names (cmd, command, super, meta, win -> command; opt, option, alt -> option; ctrl, control -> ctrl).

• Fail-Safe & Boundary Protection: Coordinates clamp safely within macOS screen boundaries, preventing edge coordinate exceptions while keeping PyAutoGUI fail-safe corner protection active.

• Dual API Architecture: Provides both fine-grained atomic FastMCP tools (screenshot, mouse_click, type_text, etc.) and a unified computer_action tool capable of parsing Gemini Computer Use JSON payloads.

🛠️ Available Tools Reference

🔐 macOS Permissions Setup Guide

macOS security requires explicit user authorization for screen capture and synthetic input injection. Permissions must be granted to the host application running the server (e.g., Terminal, iTerm2, Claude Desktop, Cursor, or VS Code).

1. Accessibility (辅助功能)

Required for mouse movements, clicking, drag-and-drop, and keystroke injection.

1. Open System Settings (系统设置).

2. Navigate to Privacy & Security (隐私与安全性) ➔ Accessibility (辅助功能).

3. Toggle ON or click + to add:

   • Your terminal app (Terminal, iTerm2, Ghostty)

   • Your IDE (Cursor, Visual Studio Code, Windsurf)

   • Claude Desktop (if used as the MCP client)

2. Screen Recording (屏幕录制)

Required for taking screen captures via the screenshot tool.

1. Open System Settings (系统设置).

2. Navigate to Privacy & Security (隐私与安全性) ➔ Screen Recording (屏幕录制).

3. Ensure the switch is toggled ON for your terminal app, IDE, or Claude Desktop.

4. If newly added, restart the respective application for permissions to take effect.

Note: The server performs automatic preflight permission checks on startup (check_macos_permissions()) and logs diagnostic warnings to stderr if permissions are missing.

⚙️ Client Configuration Guide

Claude Desktop

Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "gemini-cu-mcp": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/path/to/gemini-mac-computer-use-mcp",
        "gemini-cu-mcp"
      ]
    }
  }
}

Replace /path/to/gemini-mac-computer-use-mcp with your actual repository path.

Claude Code CLI

Add the server to Claude Code using the CLI:

claude mcp add gemini-cu-mcp -- uv run --directory /path/to/gemini-mac-computer-use-mcp gemini-cu-mcp

Cursor

1. Open Cursor Settings ➔ Features ➔ MCP Servers.

2. Click Add New MCP Server.

3. Configure the fields:

   • Name: gemini-cu-mcp

   • Type: command

   • Command: uv run --directory /path/to/gemini-mac-computer-use-mcp gemini-cu-mcp

Windsurf

Open or create ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "gemini-cu-mcp": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/path/to/gemini-mac-computer-use-mcp",
        "gemini-cu-mcp"
      ]
    }
  }
}

🚀 One-Click Deployment Script

You can install and deploy the server with a single curl command. The script automatically checks macOS compatibility, provisions the environment, verifies the test suite, performs permission preflight checks, and configures Claude Desktop:

Quick Install via curl

# Interactive installation (prompts before modifying Claude Desktop config)
curl -fsSL https://raw.githubusercontent.com/Be-Young/gemini-mac-computer-use-mcp/main/deploy.sh | bash

# Unattended installation (automatically configures Claude Desktop without prompting)
curl -fsSL https://raw.githubusercontent.com/Be-Young/gemini-mac-computer-use-mcp/main/deploy.sh | bash -s -- --yes

# Skip Claude Desktop configuration
curl -fsSL https://raw.githubusercontent.com/Be-Young/gemini-mac-computer-use-mcp/main/deploy.sh | bash -s -- --skip-config

Or Run from Cloned Repository

# Interactive run
./deploy.sh

# Automatically authorize and configure Claude Desktop
./deploy.sh --yes

# Skip Claude Desktop configuration
./deploy.sh --skip-config

🧪 Manual Run and Developer Guide

Prerequisites

• macOS 12.0+ (Apple Silicon or Intel)

• Python 3.11+

• uv package manager

1. Install Dependencies

uv sync

2. Run Test Suite

Verify that all unit tests and integration mocks pass:

uv run pytest -v

3. Run the Server Manually

Start the MCP server over standard input/output (stdio):

uv run gemini-cu-mcp

4. Interactive Testing with MCP Inspector

Test each tool interactively in your browser using the MCP Inspector:

npx @modelcontextprotocol/inspector uv run gemini-cu-mcp

❓ Troubleshooting

• Permission Denied / Events Not Injected: Verify Accessibility permissions in macOS System Settings. If permissions were granted previously but stopped working after a terminal update, toggle the permission off and on again, or reset via:

  tccutil reset Accessibility

• Blank or Black Screenshot: Grant Screen Recording permissions to your terminal or client application:

  tccutil reset ScreenCapture

• uv command not found: Install uv via Homebrew or the official installer:

  curl -LsSf https://astral.sh/uv/install.sh | sh

📄 License

MIT License. See LICENSE for details.