Skip to main content
Glama
macos-control-mcp

macos-control-mcp

by vito1317

macos-control-mcp

MCP Server,讓 AI 完全操控你的 macOS 電腦 — 滑鼠、鍵盤、終端機、截圖、視窗管理、UI 元件偵測,並提供 AI 優化的資訊回報。

功能模組

🖱️ 滑鼠控制

  • mouse_move — 移動滑鼠到指定座標

  • mouse_click — 點擊(左/右/中鍵,單擊/雙擊/三擊)

  • mouse_drag — 拖曳操作

  • mouse_scroll — 滾輪捲動

  • mouse_position — 取得目前滑鼠位置

⌨️ 鍵盤輸入

  • keyboard_type — 打字輸入(支援中文、日文、emoji)

  • keyboard_press — 按鍵 + 修飾鍵組合

  • keyboard_hotkey — 快捷鍵組合(如 cmd+c)

📸 截圖與螢幕分析

  • screenshot — 全螢幕/區域截圖,支援座標格線疊加

  • screenshot_annotated — 標註特定點位的截圖

  • screen_info — 螢幕解析度、位置、縮放比例

💻 終端機操作

  • terminal_execute — 執行 shell 指令

  • terminal_execute_background — 背景執行長時間指令

  • terminal_applescript — 執行 AppleScript

🪟 視窗管理

  • window_list — 列出所有可見視窗

  • window_focus — 切換/聚焦應用程式

  • window_resize — 移動和調整視窗大小

  • window_minimize — 最小化視窗

  • window_close — 關閉視窗

  • apps_list — 列出所有執行中的應用程式

♿ Accessibility API 元件偵測

  • accessibility_check — 檢查無障礙權限

  • accessibility_tree — 取得 UI 元件樹(按鈕、輸入框、標籤等位置和屬性)

  • accessibility_element_at — 取得指定座標的 UI 元件

  • accessibility_click — 透過 Accessibility API 點擊元件(比座標更可靠)

🤖 AI 資訊優化工具

  • ai_screen_context — 一鍵取得完整螢幕上下文(截圖+座標格線+Accessibility 樹+滑鼠位置)

  • ai_find_element — 用自然語言搜尋 UI 元件

  • ai_ocr_region — 區域 OCR 文字辨識(使用 macOS Vision framework)

  • clipboard_read / clipboard_write — 剪貼簿讀寫

Related MCP server: AutoMac MCP

安裝

前置需求

  • macOS 13+

  • Node.js 18+

  • Xcode Command Line Tools(xcode-select --install)

一鍵安裝

curl -fsSL https://raw.githubusercontent.com/vito1317/macos-control-mcp/main/install.sh | bash

手動安裝

# Clone
git clone https://github.com/vito1317/macos-control-mcp.git ~/.local/share/macos-control-mcp
cd ~/.local/share/macos-control-mcp

# Build
npm install && npm run setup

# 註冊 MCP
claude mcp add macos-control -s user -- node ~/.local/share/macos-control-mcp/dist/index.js

使用方式

# 註冊 MCP
claude mcp add macos-control -s user -- node ~/.local/share/macos-control-mcp/dist/index.js

# 移除 MCP
claude mcp remove macos-control -s user

# 更新
cd ~/.local/share/macos-control-mcp && git pull && npm run setup

# 完整移除
claude mcp remove macos-control -s user && rm -rf ~/.local/share/macos-control-mcp

權限設定

安裝後需要在系統設定中授予權限:

  • System Settings > Privacy & Security > Accessibility — 加入你的終端機應用

  • System Settings > Privacy & Security > Screen Recording — 加入你的終端機應用

設定(進階)

手動 JSON 設定

也可直接編輯 MCP 設定檔:

{
  "mcpServers": {
    "macos-control": {
      "command": "node",
      "args": ["~/.local/share/macos-control-mcp/dist/index.js"]
    }
  }
}

權限設定

必須授予 Accessibility 權限:

System Settings → Privacy & Security → Accessibility → 加入你的終端機應用程式

架構

macos-control-mcp/
├── src/
│   ├── index.ts              # MCP Server 進入點
│   ├── server.ts             # 伺服器設定與工具註冊
│   ├── tools/
│   │   ├── mouse.ts          # 滑鼠控制
│   │   ├── keyboard.ts       # 鍵盤輸入
│   │   ├── screenshot.ts     # 截圖與螢幕分析
│   │   ├── terminal.ts       # 終端機操作
│   │   ├── window.ts         # 視窗管理
│   │   ├── accessibility.ts  # UI 元件偵測
│   │   └── ai-optimize.ts    # AI 資訊優化
│   ├── utils/
│   │   ├── swift-bridge.ts   # Swift 原生助手橋接
│   │   └── image.ts          # 影像處理(格線、壓縮、標註)
│   └── types/
│       └── index.ts          # TypeScript 型別定義
├── swift-helpers/
│   ├── MacControl.swift      # Swift 原生助手(CoreGraphics, Accessibility API)
│   └── build.sh              # Swift 編譯腳本
└── bin/                      # 編譯後的 Swift binary

AI 使用建議

AI Agent 操控電腦的推薦流程:

  1. 觀察 — 呼叫 ai_screen_context 取得完整螢幕狀態

  2. 定位 — 使用 ai_find_element 搜尋目標元件,或從截圖座標格線判斷位置

  3. 操作 — 使用 mouse_click、keyboard_type、keyboard_hotkey 等工具互動

  4. 驗證 — 再次呼叫 ai_screen_context 確認操作結果

License

MIT

Available Tools

36 tools
accessibility_checkA

Check if Accessibility permissions are granted. Required for UI element detection and programmatic interaction. If not granted, provides instructions for enabling it.

Inputs: no named parameters.

accessibility_clickA

Click a UI element by its accessibility role and optional title. This performs an accessibility "press" action, which is more reliable than coordinate-based clicking for buttons, menu items, etc.

Inputs: role (required), pid, title.

accessibility_element_atA

Get the UI element at a specific screen coordinate. Returns the element's role, title, value, position, and size. Useful for identifying what is under the mouse cursor or at a specific screen location.

Inputs: x (required), y (required).

accessibility_treeA

Get the accessibility UI element tree of an application. Returns a hierarchical tree of all UI elements (buttons, text fields, labels, etc.) with their role, title, value, position, size, and state. This is the primary way to understand what's on screen without relying on screenshots. Use maxDepth to control detail level.

Inputs: pid, maxDepth.

ai_find_elementA

[AI-Optimized] Find a UI element by description in natural language. Searches the accessibility tree of the frontmost app and returns matching elements with their exact coordinates. Use this to locate buttons, text fields, menu items, etc.

Inputs: description (required), pid.

ai_ocr_regionA

[AI-Optimized] Extract text from a screen region using macOS Vision framework OCR. Returns recognized text with positions. Useful for reading text that isn't accessible via the accessibility API.

Inputs: x (required), y (required), width (required), height (required).

ai_screen_contextA

[AI-Optimized] Capture a comprehensive snapshot of the current screen state for AI analysis. Returns: 1) Screenshot with coordinate grid overlay, 2) Accessibility tree of the frontmost app (interactive elements with positions), 3) Current mouse position, 4) Frontmost app info. Good for understanding screen context. For CLICKING elements, prefer ai_screen_elements which gives precise coordinates.

Inputs: maxDepth, maxWidth, gridSpacing, includeScreenshot, +1 more.

ai_screen_elementsA

🎯 PREFERRED — Use this FIRST when you need to click or interact with UI elements. Auto-detects ALL interactive elements (buttons, fields, links, etc.) with precise center coordinates. Uses accessibility tree for native apps, AND automatically scans web page elements via Chrome DevTools Protocol when a browser is frontmost. Returns: 1) Annotated screenshot with numbered markers on every element, 2) Element list with [number] role "title" center=(x,y). Web elements are prefixed with "web:" (e.g. web:input, web:button, web:link). These coordinates are MORE ACCURATE than manually reading positions from a plain screenshot. Always prefer this over screenshot+manual coordinate guessing.

Inputs: pid, maxWidth.

ai_web_elementsA

[AI-Optimized] Scan ALL interactive elements on a web page via Chrome DevTools Protocol. Auto-launches a CDP-enabled Chrome if needed. Provide a URL to navigate to, or omit to scan the current page. Detects every button, link, input, select, textarea, and clickable element with screen coordinates for mouse_click.

Inputs: url, cdpPort, tabIndex, includeText, +1 more.

animation_clickA

Show a visual click ripple animation at the specified coordinates. Does NOT actually click — only shows the animation effect. Use for visual feedback.

Inputs: x (required), y (required), color, button.

animation_highlightA

Highlight a rectangular region on screen with a pulsing border and optional label. Use to draw attention to UI elements.

Inputs: x (required), y (required), width (required), height (required), +3 more.

animation_scrollA

Show a visual scroll direction indicator animation. Does NOT actually scroll.

Inputs: x (required), y (required), direction (required), color.

animation_trailA

Show a visual mouse movement trail animation along a path of coordinates. Does NOT actually move the mouse.

Inputs: points (required), color, duration.

animation_typeA

Show a visual typing indicator animation with text at specified coordinates. Does NOT actually type — only shows the animation.

Inputs: x (required), y (required), text (required), color.

apps_listA

List all running applications with their name, PID, bundle ID, and active/hidden status.

Inputs: no named parameters.

clipboard_readA

Read the current text content from the macOS clipboard/pasteboard.

Inputs: no named parameters.

clipboard_writeA

Write text content to the macOS clipboard/pasteboard.

Inputs: text (required).

keyboard_hotkeyB

Press a keyboard shortcut / hotkey combination. Use "+" to separate keys. Examples: "cmd+c" (copy), "cmd+v" (paste), "cmd+shift+s" (save as), "cmd+tab" (switch app), "ctrl+cmd+f" (fullscreen).

Inputs: keys (required).

keyboard_pressA

Press a key with optional modifier keys. Available keys: letters (a-z), numbers (0-9), function keys (f1-f12), arrow keys (up/down/left/right), return/enter, tab, space, delete/backspace, escape/esc, home, end, pageup, pagedown, and symbols. Modifiers: command/cmd, shift, option/alt, control/ctrl, fn.

Inputs: key (required), modifiers.

keyboard_typeA

Type a string of text character by character, simulating real keyboard input. Supports Unicode (Chinese, Japanese, emoji, etc.).

Inputs: text (required).

mouse_clickC

Click the mouse at specified coordinates. Supports left/right/middle button and single/double/triple click. Defaults: button=left, clicks=1.

Inputs: x (required), y (required), button, clicks.

mouse_dragA

Drag the mouse from one position to another. Useful for drag-and-drop, selecting text, resizing windows.

Inputs: toX (required), toY (required), fromX (required), fromY (required), +1 more.

mouse_moveA

Move the mouse cursor to specified screen coordinates (x, y). Coordinates use top-left origin.

Inputs: x (required), y (required).

mouse_positionA

Get the current mouse cursor position on screen. Returns both CoreGraphics (top-left origin) and AppKit (bottom-left origin) coordinates.

Inputs: no named parameters.

mouse_scrollB

Scroll at specified coordinates. Positive deltaY scrolls down, negative scrolls up. Positive deltaX scrolls right.

Inputs: x (required), y (required), deltaY (required), deltaX.

screen_infoA

Get information about all connected displays/screens — resolution, position, scale factor, visible area.

Inputs: no named parameters.

screenshotA

Take a screenshot of the entire screen or a specific region. Returns the image optimized for AI analysis. Options include coordinate grid overlay, compression, and region cropping. ⚠️ PREFER ai_screen_elements over this tool when you need to find clickable elements — it auto-detects ALL interactive elements with precise coordinates. Use screenshot only when you need a raw visual overview or to verify results. If you must estimate positions from the screenshot, aim for the vertical center or slightly below center of targets (visual perception biases Y upward).

Inputs: format, region, quality, maxWidth, +3 more.

screenshot_annotatedA

Take a screenshot and annotate specific points with labels. Useful for marking UI elements, buttons, or areas of interest for AI reference.

Inputs: points (required), maxWidth.

terminal_applescriptA

Execute AppleScript code. Useful for macOS-specific automation like controlling apps via their AppleScript dictionaries, showing dialogs, etc.

Inputs: script (required), timeout.

terminal_executeB

Execute a shell command in the macOS terminal and return the output. Supports any shell command including pipes, redirects, and chaining. Default shell is zsh. Timeout defaults to 30 seconds.

Inputs: command (required), cwd, env, shell, +1 more.

terminal_execute_backgroundA

Start a long-running process in the background. Returns the PID immediately. Use terminal_execute with "kill " to stop it.

Inputs: command (required), cwd.

window_closeB

Close the front window of an application.

Inputs: appName (required).

window_focusA

Bring an application to the foreground and focus it. Matches app name partially (case-insensitive).

Inputs: appName (required).

window_listA

List all visible windows on screen with their app name, window title, position, size, and process ID. Useful for understanding what is currently open and finding window IDs for screenshots.

Inputs: no named parameters.

window_minimizeB

Minimize the front window of an application.

Inputs: appName (required).

window_resizeB

Move and resize an application window to specific coordinates and dimensions.

Inputs: x (required), y (required), width (required), height (required), +1 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.3/5.0

    Scored across 36 tools

    Disambiguation3/5

    Mouse, keyboard, window, and clipboard tools are clearly distinct, but the screen/UI inspection space is crowded with overlapping tools like screenshot, screenshot_annotated, accessibility_tree, ai_screen_context, ai_screen_elements, ai_web_elements, and ai_ocr_region. Descriptions try to differentiate them, but an agent could easily pick the wrong one.

    Naming Consistency3/5

    All names use snake_case and domain prefixes, giving a readable structure. However, the verb/noun ordering is inconsistent: some are resource_action (mouse_move, window_resize), some are action_resource (terminal_execute, clipboard_read), and some are plain nouns (screenshot, screen_info, apps_list).

    Tool Count2/5

    36 tools is well past the 25+ threshold for being considered heavy, and several are non-essential or overlapping, especially the five animation tools and the many AI/accessibility screen-inspection tools. The core capability could be covered with a tighter set.

    Completeness4/5

    The surface covers most macOS control needs: mouse input, keyboard input, screenshots, terminal execution, window management, app listing, clipboard access, and accessibility-tree inspection. Minor gaps like explicit app launch/quit or raw key-up/key-down events are workaroundable via terminal_execute or existing tools.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      F
      maintenance
      An experimental MCP server providing full control over the macOS user interface through mouse, keyboard, and window management tools. It enables AI assistants to automate desktop tasks by utilizing native accessibility APIs and OCR for real-time screen comprehension.
      7
      Creative Commons Zero v1.0 Universal
    • A
      license
      Not graded
      quality
      A
      maintenance
      A lightweight MCP server that bridges AI agents and macOS, enabling automation of file navigation, application control, UI interaction, browser automation, and system operations.
      244
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      Standalone MCP server that gives AI agents full GUI control over macOS — screenshots, mouse, keyboard, apps, clipboard, and multi-display — with zero private dependencies.
      20
      MIT

    Appeared in Searches