macos-control-mcp
Allows AI to fully control a macOS computer, including mouse, keyboard, terminal, screenshots, window management, UI element detection, and AI-optimized context reporting.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@macos-control-mcp move the mouse to coordinates (100, 200) and click"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
macos-control-mcp
MCP Server,讓 AI 完全操控你的 macOS 電腦 — 滑鼠、鍵盤、終端機、截圖、視窗管理、UI 元件偵測,並提供 AI 優化的資訊回報。
功能模組
🖱️ 滑鼠控制
mouse_move— 移動滑鼠到指定座標mouse_click— 點擊(左/右/中鍵,單擊/雙擊/三擊)mouse_drag— 拖曳操作mouse_scroll— 滾輪捲動mouse_position— 取得目前滑鼠位置
⌨️ 鍵盤輸入
keyboard_type— 打字輸入(支援中文、日文、emoji)keyboard_press— 按鍵 + 修飾鍵組合keyboard_hotkey— 快捷鍵組合(如cmd+c)
📸 截圖與螢幕分析
screenshot— 全螢幕/區域截圖,支援座標格線疊加screenshot_annotated— 標註特定點位的截圖screen_info— 螢幕解析度、位置、縮放比例
💻 終端機操作
terminal_execute— 執行 shell 指令terminal_execute_background— 背景執行長時間指令terminal_applescript— 執行 AppleScript
🪟 視窗管理
window_list— 列出所有可見視窗window_focus— 切換/聚焦應用程式window_resize— 移動和調整視窗大小window_minimize— 最小化視窗window_close— 關閉視窗apps_list— 列出所有執行中的應用程式
♿ Accessibility API 元件偵測
accessibility_check— 檢查無障礙權限accessibility_tree— 取得 UI 元件樹(按鈕、輸入框、標籤等位置和屬性)accessibility_element_at— 取得指定座標的 UI 元件accessibility_click— 透過 Accessibility API 點擊元件(比座標更可靠)
🤖 AI 資訊優化工具
ai_screen_context— 一鍵取得完整螢幕上下文(截圖+座標格線+Accessibility 樹+滑鼠位置)ai_find_element— 用自然語言搜尋 UI 元件ai_ocr_region— 區域 OCR 文字辨識(使用 macOS Vision framework)clipboard_read/clipboard_write— 剪貼簿讀寫
Related MCP server: AutoMac MCP
安裝
前置需求
macOS 13+
Node.js 18+
Xcode Command Line Tools(
xcode-select --install)
一鍵安裝
curl -fsSL https://raw.githubusercontent.com/vito1317/macos-control-mcp/main/install.sh | bash手動安裝
# Clone
git clone https://github.com/vito1317/macos-control-mcp.git ~/.local/share/macos-control-mcp
cd ~/.local/share/macos-control-mcp
# Build
npm install && npm run setup
# 註冊 MCP
claude mcp add macos-control -s user -- node ~/.local/share/macos-control-mcp/dist/index.js使用方式
# 註冊 MCP
claude mcp add macos-control -s user -- node ~/.local/share/macos-control-mcp/dist/index.js
# 移除 MCP
claude mcp remove macos-control -s user
# 更新
cd ~/.local/share/macos-control-mcp && git pull && npm run setup
# 完整移除
claude mcp remove macos-control -s user && rm -rf ~/.local/share/macos-control-mcp權限設定
安裝後需要在系統設定中授予權限:
System Settings > Privacy & Security > Accessibility — 加入你的終端機應用
System Settings > Privacy & Security > Screen Recording — 加入你的終端機應用
設定(進階)
手動 JSON 設定
也可直接編輯 MCP 設定檔:
{
"mcpServers": {
"macos-control": {
"command": "node",
"args": ["~/.local/share/macos-control-mcp/dist/index.js"]
}
}
}權限設定
必須授予 Accessibility 權限:
System Settings → Privacy & Security → Accessibility → 加入你的終端機應用程式
架構
macos-control-mcp/
├── src/
│ ├── index.ts # MCP Server 進入點
│ ├── server.ts # 伺服器設定與工具註冊
│ ├── tools/
│ │ ├── mouse.ts # 滑鼠控制
│ │ ├── keyboard.ts # 鍵盤輸入
│ │ ├── screenshot.ts # 截圖與螢幕分析
│ │ ├── terminal.ts # 終端機操作
│ │ ├── window.ts # 視窗管理
│ │ ├── accessibility.ts # UI 元件偵測
│ │ └── ai-optimize.ts # AI 資訊優化
│ ├── utils/
│ │ ├── swift-bridge.ts # Swift 原生助手橋接
│ │ └── image.ts # 影像處理(格線、壓縮、標註)
│ └── types/
│ └── index.ts # TypeScript 型別定義
├── swift-helpers/
│ ├── MacControl.swift # Swift 原生助手(CoreGraphics, Accessibility API)
│ └── build.sh # Swift 編譯腳本
└── bin/ # 編譯後的 Swift binaryAI 使用建議
AI Agent 操控電腦的推薦流程:
觀察 — 呼叫
ai_screen_context取得完整螢幕狀態定位 — 使用
ai_find_element搜尋目標元件,或從截圖座標格線判斷位置操作 — 使用
mouse_click、keyboard_type、keyboard_hotkey等工具互動驗證 — 再次呼叫
ai_screen_context確認操作結果
License
MIT
Available Tools
36 toolsaccessibility_checkA
Check if Accessibility permissions are granted. Required for UI element detection and programmatic interaction. If not granted, provides instructions for enabling it.
Inputs: no named parameters.
accessibility_clickA
Click a UI element by its accessibility role and optional title. This performs an accessibility "press" action, which is more reliable than coordinate-based clicking for buttons, menu items, etc.
Inputs: role (required), pid, title.
accessibility_element_atA
Get the UI element at a specific screen coordinate. Returns the element's role, title, value, position, and size. Useful for identifying what is under the mouse cursor or at a specific screen location.
Inputs: x (required), y (required).
accessibility_treeA
Get the accessibility UI element tree of an application. Returns a hierarchical tree of all UI elements (buttons, text fields, labels, etc.) with their role, title, value, position, size, and state. This is the primary way to understand what's on screen without relying on screenshots. Use maxDepth to control detail level.
Inputs: pid, maxDepth.
ai_find_elementA
[AI-Optimized] Find a UI element by description in natural language. Searches the accessibility tree of the frontmost app and returns matching elements with their exact coordinates. Use this to locate buttons, text fields, menu items, etc.
Inputs: description (required), pid.
ai_ocr_regionA
[AI-Optimized] Extract text from a screen region using macOS Vision framework OCR. Returns recognized text with positions. Useful for reading text that isn't accessible via the accessibility API.
Inputs: x (required), y (required), width (required), height (required).
ai_screen_contextA
[AI-Optimized] Capture a comprehensive snapshot of the current screen state for AI analysis. Returns: 1) Screenshot with coordinate grid overlay, 2) Accessibility tree of the frontmost app (interactive elements with positions), 3) Current mouse position, 4) Frontmost app info. Good for understanding screen context. For CLICKING elements, prefer ai_screen_elements which gives precise coordinates.
Inputs: maxDepth, maxWidth, gridSpacing, includeScreenshot, +1 more.
ai_screen_elementsA
🎯 PREFERRED — Use this FIRST when you need to click or interact with UI elements. Auto-detects ALL interactive elements (buttons, fields, links, etc.) with precise center coordinates. Uses accessibility tree for native apps, AND automatically scans web page elements via Chrome DevTools Protocol when a browser is frontmost. Returns: 1) Annotated screenshot with numbered markers on every element, 2) Element list with [number] role "title" center=(x,y). Web elements are prefixed with "web:" (e.g. web:input, web:button, web:link). These coordinates are MORE ACCURATE than manually reading positions from a plain screenshot. Always prefer this over screenshot+manual coordinate guessing.
Inputs: pid, maxWidth.
ai_web_elementsA
[AI-Optimized] Scan ALL interactive elements on a web page via Chrome DevTools Protocol. Auto-launches a CDP-enabled Chrome if needed. Provide a URL to navigate to, or omit to scan the current page. Detects every button, link, input, select, textarea, and clickable element with screen coordinates for mouse_click.
Inputs: url, cdpPort, tabIndex, includeText, +1 more.
animation_clickA
Show a visual click ripple animation at the specified coordinates. Does NOT actually click — only shows the animation effect. Use for visual feedback.
Inputs: x (required), y (required), color, button.
animation_highlightA
Highlight a rectangular region on screen with a pulsing border and optional label. Use to draw attention to UI elements.
Inputs: x (required), y (required), width (required), height (required), +3 more.
animation_scrollA
Show a visual scroll direction indicator animation. Does NOT actually scroll.
Inputs: x (required), y (required), direction (required), color.
animation_trailA
Show a visual mouse movement trail animation along a path of coordinates. Does NOT actually move the mouse.
Inputs: points (required), color, duration.
animation_typeA
Show a visual typing indicator animation with text at specified coordinates. Does NOT actually type — only shows the animation.
Inputs: x (required), y (required), text (required), color.
apps_listA
List all running applications with their name, PID, bundle ID, and active/hidden status.
Inputs: no named parameters.
clipboard_readA
Read the current text content from the macOS clipboard/pasteboard.
Inputs: no named parameters.
clipboard_writeA
Write text content to the macOS clipboard/pasteboard.
Inputs: text (required).
keyboard_hotkeyB
Press a keyboard shortcut / hotkey combination. Use "+" to separate keys. Examples: "cmd+c" (copy), "cmd+v" (paste), "cmd+shift+s" (save as), "cmd+tab" (switch app), "ctrl+cmd+f" (fullscreen).
Inputs: keys (required).
keyboard_pressA
Press a key with optional modifier keys. Available keys: letters (a-z), numbers (0-9), function keys (f1-f12), arrow keys (up/down/left/right), return/enter, tab, space, delete/backspace, escape/esc, home, end, pageup, pagedown, and symbols. Modifiers: command/cmd, shift, option/alt, control/ctrl, fn.
Inputs: key (required), modifiers.
keyboard_typeA
Type a string of text character by character, simulating real keyboard input. Supports Unicode (Chinese, Japanese, emoji, etc.).
Inputs: text (required).
mouse_clickC
Click the mouse at specified coordinates. Supports left/right/middle button and single/double/triple click. Defaults: button=left, clicks=1.
Inputs: x (required), y (required), button, clicks.
mouse_dragA
Drag the mouse from one position to another. Useful for drag-and-drop, selecting text, resizing windows.
Inputs: toX (required), toY (required), fromX (required), fromY (required), +1 more.
mouse_moveA
Move the mouse cursor to specified screen coordinates (x, y). Coordinates use top-left origin.
Inputs: x (required), y (required).
mouse_positionA
Get the current mouse cursor position on screen. Returns both CoreGraphics (top-left origin) and AppKit (bottom-left origin) coordinates.
Inputs: no named parameters.
mouse_scrollB
Scroll at specified coordinates. Positive deltaY scrolls down, negative scrolls up. Positive deltaX scrolls right.
Inputs: x (required), y (required), deltaY (required), deltaX.
screen_infoA
Get information about all connected displays/screens — resolution, position, scale factor, visible area.
Inputs: no named parameters.
screenshotA
Take a screenshot of the entire screen or a specific region. Returns the image optimized for AI analysis. Options include coordinate grid overlay, compression, and region cropping. ⚠️ PREFER ai_screen_elements over this tool when you need to find clickable elements — it auto-detects ALL interactive elements with precise coordinates. Use screenshot only when you need a raw visual overview or to verify results. If you must estimate positions from the screenshot, aim for the vertical center or slightly below center of targets (visual perception biases Y upward).
Inputs: format, region, quality, maxWidth, +3 more.
screenshot_annotatedA
Take a screenshot and annotate specific points with labels. Useful for marking UI elements, buttons, or areas of interest for AI reference.
Inputs: points (required), maxWidth.
terminal_applescriptA
Execute AppleScript code. Useful for macOS-specific automation like controlling apps via their AppleScript dictionaries, showing dialogs, etc.
Inputs: script (required), timeout.
terminal_executeB
Execute a shell command in the macOS terminal and return the output. Supports any shell command including pipes, redirects, and chaining. Default shell is zsh. Timeout defaults to 30 seconds.
Inputs: command (required), cwd, env, shell, +1 more.
terminal_execute_backgroundA
Start a long-running process in the background. Returns the PID immediately. Use terminal_execute with "kill " to stop it.
Inputs: command (required), cwd.
window_closeB
Close the front window of an application.
Inputs: appName (required).
window_focusA
Bring an application to the foreground and focus it. Matches app name partially (case-insensitive).
Inputs: appName (required).
window_listA
List all visible windows on screen with their app name, window title, position, size, and process ID. Useful for understanding what is currently open and finding window IDs for screenshots.
Inputs: no named parameters.
window_minimizeB
Minimize the front window of an application.
Inputs: appName (required).
window_resizeB
Move and resize an application window to specific coordinates and dimensions.
Inputs: x (required), y (required), width (required), height (required), +1 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 36 tools
Mouse, keyboard, window, and clipboard tools are clearly distinct, but the screen/UI inspection space is crowded with overlapping tools like screenshot, screenshot_annotated, accessibility_tree, ai_screen_context, ai_screen_elements, ai_web_elements, and ai_ocr_region. Descriptions try to differentiate them, but an agent could easily pick the wrong one.
All names use snake_case and domain prefixes, giving a readable structure. However, the verb/noun ordering is inconsistent: some are resource_action (mouse_move, window_resize), some are action_resource (terminal_execute, clipboard_read), and some are plain nouns (screenshot, screen_info, apps_list).
36 tools is well past the 25+ threshold for being considered heavy, and several are non-essential or overlapping, especially the five animation tools and the many AI/accessibility screen-inspection tools. The core capability could be covered with a tighter set.
The surface covers most macOS control needs: mouse input, keyboard input, screenshots, terminal execution, window management, app listing, clipboard access, and accessibility-tree inspection. Minor gaps like explicit app launch/quit or raw key-up/key-down events are workaroundable via terminal_execute or existing tools.
Maintenance
Related MCP Connectors
Remote MCP server exposing your Node Command fleet (files, shell, processes, browser) to AI hosts.
Remote MCP server for supportsheep: run AI interviews and manage support content for your blog.
Remote MCP server for supportsheep: run AI interviews and manage support content for your blog.
Remote MCP server for AI.TV creators — delegate account operations to your AI agent over MCP.
Related MCP Servers
- AlicenseAqualityDmaintenanceThe first open-source MCP server that enables AI to fully control remote macOS systems.8492MIT
- AlicenseNot gradedqualityFmaintenanceAn experimental MCP server providing full control over the macOS user interface through mouse, keyboard, and window management tools. It enables AI assistants to automate desktop tasks by utilizing native accessibility APIs and OCR for real-time screen comprehension.7Creative Commons Zero v1.0 Universal
- AlicenseNot gradedqualityAmaintenanceA lightweight MCP server that bridges AI agents and macOS, enabling automation of file navigation, application control, UI interaction, browser automation, and system operations.244MIT
- AlicenseNot gradedqualityDmaintenanceStandalone MCP server that gives AI agents full GUI control over macOS — screenshots, mouse, keyboard, apps, clipboard, and multi-display — with zero private dependencies.20MIT