macbot-mcp

An MCP server that gives AI agents hands on macOS — via the Accessibility API, AppleScript, and Quartz CGEvents.

The macOS counterpart to ahk-mcp. Same thesis: the accessibility tree already contains a machine-readable description of everything on screen. Screenshots throw that away and make the model re-derive it from pixels. Why?

Token cost per action: ~200-700 tokens (vs ~2000-3500 for screenshot-based).

Because the output is structured text, not images, any language model can drive it — including small open-source models with no vision capability. A 7B model can parse AXButton title="Save" @450,320 88x32 and call mac_click(x=490, y=336). It cannot interpret a screenshot. The accessibility approach makes computer use available to models that were previously locked out of it entirely.

How it works

macbot-mcp exposes 17 tools over MCP's stdio transport:

• Observation tools read the macOS accessibility tree, window properties, and browser URLs — returning structured text with element roles, names, values, and screen coordinates

• Action tools click, drag, scroll, type, and send keystrokes via Quartz CGEvents and AppleScript

• mac_batch runs a known sequence of existing tools in one round trip

• mac_wait polls for an expected UI state and returns evidence or a timeout

• mac_run_applescript is the escape hatch — execute arbitrary AppleScript for anything the built-in tools don't cover

Every observation tool returns coordinates. Find a button with mac_ui_find, get its x,y position, and click it with mac_click — no screenshot needed.

Related MCP server: mcp-macos-cua

Token cost comparison

A 20-step workflow: ~50k tokens with screenshots, ~8k with macbot. The structured output is also more reliable — the model doesn't guess where the "Save" button is when the accessibility tree says AXButton title="Save" @1043,672 88x32.

Installation

Prerequisites

• macOS 12+ (Monterey or later)

• Python 3.10+

Setup

git clone https://github.com/anomalous3/macbot-mcp.git
cd macbot-mcp

# Create a virtual environment
python3 -m venv .venv
# or: uv venv .venv

# Activate
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

This version requires MCP Python SDK 2.3.0 or newer, below 3.0. The server uses the v2 handler and result APIs; use its updated environment when reconnecting. The separate voice prototype has its own dependencies and is unaffected. The official SDK migration guide describes the handler/result and field-name changes. requirements.lock.txt records the exact versions tested on this Mac; uv pip sync --python .venv/bin/python requirements.lock.txt recreates that environment.

Run the isolated behavior tests with .venv/bin/python -m unittest -v test_batch test_wait. They mock desktop input. Live checks also verified v1 client/v2 server compatibility, timeout error flags, and a seven-step browser batch with a delayed element, enabled state, field value, disappearance, URL, and title conditions.

macOS Permissions

macbot needs two permissions, granted to your terminal app (Terminal, kitty, iTerm2, etc.):

1. Accessibility — System Settings > Privacy & Security > Accessibility

2. Screen Recording — System Settings > Privacy & Security > Screen Recording

Without Accessibility, observation and input tools won't work. Without Screen Recording, mac_screenshot will fail.

Three gotchas:

• Granting is not enough — fully quit and reopen the terminal (Cmd-Q, not just a new tab/window). Running processes keep their old permission state.

• macOS 15+ periodically re-asks for Screen Recording. If screenshots worked yesterday and now fail with could not create image from display (and mac_get_windows returns []), respond to the re-consent prompt in System Settings, then restart the terminal.

• If you run Claude Code inside an IDE's integrated terminal (VS Code etc.), grant the permissions to the IDE, not Terminal.app.

Claude Code MCP configuration

Add to your ~/.claude.json:

{
  "mcpServers": {
    "macbot": {
      "command": "/path/to/macbot-mcp/.venv/bin/python3",
      "args": ["/path/to/macbot-mcp/server.py"],
      "env": {
        "MACBOT_SCREENSHOT_DIR": "/tmp/macbot"
      }
    }
  }
}

After adding the config, restart Claude Code. The tools appear with the mcp__macbot__ prefix.

Tool reference

Observation

Action

Scroll amounts are wheel lines, not pixels; visual distance depends on the app. The scroll event is explicitly positioned at the requested coordinates so an asynchronous pointer move cannot send it to the previous pointer location. A successful response confirms event posting, not that the page moved (it may already be at a boundary or the target may not accept scrolling).

The focus race (why app= exists)

An agent's tool calls are separated by seconds of thinking time. If you call mac_focus_app in one tool call and mac_type_text in the next, the terminal running the agent frequently reclaims focus in the gap — and the text lands in the agent's own input box instead of the target app. Focus and action must happen inside a single call. That's what the app= parameter on mac_type_text and mac_key_press does: activate, wait until the app is verified frontmost, then act.

Batch a known sequence

mac_batch takes 1–50 actions and an optional default app. Each action has tool (an existing tool name) and args (that tool's argument object). The default app applies only to tools with an app argument; a per-step app overrides it. mac_focus_app still takes its own explicit app_name.

{
  "app": "Firefox",
  "actions": [
    {"tool": "mac_key_press", "args": {"key": "l", "modifiers": ["command"]}},
    {"tool": "mac_type_text", "args": {"text": "https://example.com"}},
    {"tool": "mac_key_press", "args": {"key": "return"}},
    {"tool": "mac_wait", "args": {"condition": "exists", "name": "Example Domain", "role": "AXHeading", "timeout": 5}}
  ]
}

This replaces four model/tool round trips with one. It reduces orchestration overhead; the individual actions still run sequentially and produce results. All steps are schema-validated before any run, including required arguments, types, and unknown keys. The batch stops on the first returned error or raised exception. Earlier actions remain applied; there is no rollback or automatic retry. Results identify each attempted step by zero-based index and tool. completed counts successful steps; failed_index identifies a failed step. Validation errors return status: "invalid" with completed: 0.

Batch only sequences whose next action is already known. A successful search with zero matches is an observation, not an error: batching does not interpret UI results, substitute values, or branch. Split at those decision points. mac_ui_find and mac_ui_press retain their normal polling behavior.

Use mac_wait when the sequence needs a specific postcondition. Unlike an empty mac_ui_find result, a wait timeout is an error and stops the remaining batch.

Wait for a state change

mac_wait takes condition, optional app, and the following selectors:

For *_changed, capture the previous value before the action. Missing URL or value data cannot satisfy a change. Matching uses case-sensitive comparisons; element names use the existing token-based search. Element predicates use the first match, so make selectors specific. App/Accessibility read failures are errors, not evidence of absence. Waits inspect the selected app without focusing it; they do not execute, retry, or roll back actions.

timeout defaults to 10 seconds (0–30); interval defaults to 0.2 seconds (0.05–1). A zero timeout performs one immediate check. Timeout limits polling; an individual macOS accessibility call can itself take time. A match confirms only the requested condition, not that the whole page has finished loading. An absence condition can match immediately if the element has not appeared yet; wait for its appearance first when the workflow requires an appear/disappear cycle.

For example, a batch can press the video's Play (k) button, then wait for {"condition": "exists", "name": "Pause", "role": "AXButton"}. The model gets one batch response containing the action result and playback-state evidence. Successful waits return status: "matched", elapsed seconds, and an observation. Timeouts return status: "timeout", an error, and the last observation. Tool errors also set the MCP result's isError flag.

Batches cannot contain other batches. Calls through the same server process cannot interleave with a batch, although human input and other MCP server processes can still affect the desktop. Keep passing app for input tools.

Approving mac_batch permits its contained tools, including AppleScript; clients do not ask separately for every nested step. Review and authorize the whole sequence just as you would the individual calls.

After updating an existing installation, install requirements and reconnect the MacBot MCP server (or start a new client session) to discover mac_batch.

Escape hatch

Browser automation via the Accessibility API

Modern browsers expose their full UI through the macOS Accessibility API. macbot reads this directly — no browser extension, no WebDriver, no Playwright needed.

Read the tab bar with element coordinates:

> mac_ui_tree app="Firefox" max_depth=5 max_nodes=50

AXApplication title="Firefox"
  AXWindow title="GitHub - anomalous3/macbot-mcp" @36,30 1752x957
    AXGroup desc="GitHub - anomalous3/macbot-mcp" @36,30 1752x957
      AXToolbar desc="Browser tabs" @36,30 1752x44
        AXTabGroup (tab group) @190,30 1518x44
          AXRadioButton title="GitHub - anomalous3/macbot-mcp" value="True" @193,30 210x44
          AXRadioButton title="New Tab" @403,30 210x44

Find a specific element and get its click coordinates:

> mac_ui_find app="Firefox" name="Submit" role="AXButton"

{
  "match_count": 1,
  "window_titles": ["GitHub - anomalous3/macbot-mcp"],
  "matches": [
    {"role": "AXButton", "title": "Submit", "x": 450, "y": 320, "width": 80, "height": 32}
  ]
}

The window titles come back with every search, so an empty result tells you why it was empty — wrong page, still loading, or the window is gone.

Press it without coordinates — no mouse, no focus change:

> mac_ui_press app="Firefox" name="Submit" role="AXButton"

{"status": "ok", "pressed": {...}, "match_count": 1}

Get the URL without screenshots or clipboard tricks:

> mac_ui_url app="Firefox"

{"url": "https://github.com/anomalous3/macbot-mcp", "field": "Search with Google or enter address"}

Use Firefox. It exposes the richest accessibility tree of the major browsers — more element detail, better labeling, and more consistent structure than Chrome or Safari.

The coordinate system

All coordinates are absolute screen pixels in macOS logical coordinates (not Retina physical pixels). Origin (0,0) is the top-left of the primary display.

The @x,y WxH format in mac_ui_tree output gives position and size directly. To click the center of @450,320 80x32, click at (490, 336).

Works with everything

macbot works with any macOS application that implements the Accessibility API (which is most of them):

• Browsers — Firefox, Chrome, Safari (Firefox recommended for richest tree)

• Terminals — kitty, Terminal.app, iTerm2 (can read content, type commands)

• Editors — VS Code, Sublime Text, TextEdit

• System apps — Finder, System Settings, Activity Monitor

• Any app — if it has windows and controls, macbot can probably read and drive it

Configuration

Platform

macOS only. For Windows, see ahk-mcp. The approach is the same — read the accessibility tree, act via synthetic input — just different platform APIs.

License

MIT