# macbot-mcp

An MCP server that gives AI agents hands on macOS — via the Accessibility API, AppleScript, and Quartz CGEvents.

The macOS counterpart to [ahk-mcp](https://github.com/anomalous3/ahk-mcp). Same thesis: the accessibility tree already contains a machine-readable description of everything on screen. Screenshots throw that away and make the model re-derive it from pixels. Why?

Token cost per action: **~200-700 tokens** (vs ~2000-3500 for screenshot-based).

Because the output is structured text, not images, **any language model can drive it** — including small open-source models with no vision capability. A 7B model can parse `AXButton title="Save" @450,320 88x32` and call `mac_click(x=490, y=336)`. It cannot interpret a screenshot. The accessibility approach makes computer use available to models that were previously locked out of it entirely.

## How it works

macbot-mcp exposes 17 tools over MCP's stdio transport:

- **Observation tools** read the macOS accessibility tree, window properties, and browser URLs — returning structured text with element roles, names, values, and **screen coordinates**
- **Action tools** click, drag, scroll, type, and send keystrokes via Quartz CGEvents and AppleScript
- **`mac_batch`** runs a known sequence of existing tools in one round trip
- **`mac_wait`** polls for an expected UI state and returns evidence or a timeout
- **`mac_run_applescript`** is the escape hatch — execute arbitrary AppleScript for anything the built-in tools don't cover

Every observation tool returns coordinates. Find a button with `mac_ui_find`, get its `x,y` position, and click it with `mac_click` — no screenshot needed.

## Token cost comparison

| Approach | Tokens per action | What you get |
|---|---|---|
| Screenshot-based (full screen PNG) | ~2000-3500 | Pixels. Model must OCR, locate elements, interpret layout. |
| macbot-mcp (structured text) | ~200-700 | Element roles, titles, values, and pixel coordinates. |

A 20-step workflow: ~50k tokens with screenshots, ~8k with macbot. The structured output is also more reliable — the model doesn't guess where the "Save" button is when the accessibility tree says `AXButton title="Save" @1043,672 88x32`.

## Installation

### Prerequisites

- **macOS 12+** (Monterey or later)
- **Python 3.10+**

### Setup

```bash
git clone https://github.com/anomalous3/macbot-mcp.git
cd macbot-mcp

# Create a virtual environment
python3 -m venv .venv
# or: uv venv .venv

# Activate
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt
```

This version requires MCP Python SDK **2.3.0 or newer, below 3.0**. The server
uses the v2 handler and result APIs; use its updated environment when reconnecting.
The separate voice prototype has its own dependencies and is unaffected.
The [official SDK migration guide](https://github.com/modelcontextprotocol/python-sdk/blob/main/docs/migration.md)
describes the handler/result and field-name changes. `requirements.lock.txt`
records the exact versions tested on this Mac; `uv pip sync --python
.venv/bin/python requirements.lock.txt` recreates that environment.

Run the isolated behavior tests with `.venv/bin/python -m unittest -v
test_batch test_wait`. They mock desktop input. Live checks also verified v1
client/v2 server compatibility, timeout error flags, and a seven-step browser
batch with a delayed element, enabled state, field value, disappearance, URL,
and title conditions.

### macOS Permissions

macbot needs two permissions, granted to your terminal app (Terminal, kitty, iTerm2, etc.):

1. **Accessibility** — System Settings > Privacy & Security > Accessibility
2. **Screen Recording** — System Settings > Privacy & Security > Screen Recording

Without Accessibility, observation and input tools won't work. Without Screen Recording, `mac_screenshot` will fail.

**Three gotchas:**

- **Granting is not enough — fully quit and reopen the terminal** (Cmd-Q, not
  just a new tab/window). Running processes keep their old permission state.
- **macOS 15+ periodically re-asks for Screen Recording.** If screenshots worked
  yesterday and now fail with `could not create image from display` (and
  `mac_get_windows` returns `[]`), respond to the re-consent prompt in System
  Settings, then restart the terminal.
- If you run Claude Code inside an IDE's integrated terminal (VS Code etc.),
  grant the permissions to the **IDE**, not Terminal.app.

## Claude Code MCP configuration

Add to your `~/.claude.json`:

```json
{
  "mcpServers": {
    "macbot": {
      "command": "/path/to/macbot-mcp/.venv/bin/python3",
      "args": ["/path/to/macbot-mcp/server.py"],
      "env": {
        "MACBOT_SCREENSHOT_DIR": "/tmp/macbot"
      }
    }
  }
}
```

After adding the config, restart Claude Code. The tools appear with the `mcp__macbot__` prefix.

## Tool reference

### Observation

| Tool | Description |
|---|---|
| `mac_ui_tree` | Dump the accessibility tree of any app. Returns roles, titles, values, descriptions, and screen coordinates. Configurable depth and node limit. **This is the primary observation tool — use it before reaching for screenshots.** |
| `mac_ui_find` | Search for UI elements by name and/or role. Returns matches with coordinates. Find that "Submit" button without scanning the whole tree. |
| `mac_ui_url` | Get the current URL from the active browser's address bar (Firefox, Chrome, Safari). |
| `mac_get_windows` | List all visible windows with app name, title, position, and size. |
| `mac_wait` | Wait for a named element, enabled state, field value, URL, or front-window title condition. Returns the matched observation or a timeout with the last observation. |
| `mac_screenshot` | Capture full screen, a specific window, or a region. Returns a PNG file path. **The fallback when you genuinely need pixels.** |

### Action

| Tool | Description |
|---|---|
| `mac_ui_press` | **Press an element by name/role directly through the Accessibility API — no coordinates, no mouse, no focus change.** Works on occluded windows and elements scrolled out of view. Prefer this over find+click for buttons, links, and tabs. |
| `mac_click` | Click at screen coordinates. Uses Quartz CGEvents (falls back to cliclick if installed). Accepts `app=` to bring the target window frontmost first — a click lands on whatever window is at those coordinates. |
| `mac_drag` | Click and drag between two points. Configurable duration for smooth drags. Works for rotating 3D plots, selecting text, moving windows. |
| `mac_scroll` | Scroll at a screen position. Vertical and horizontal. Also accepts `app=`. |
| `mac_type_text` | Type text into an app. Pass `app=` to focus the target atomically — recommended, see below. Uses clipboard paste by default (fast, Unicode-safe). Optional keystroke mode for modifier-sensitive fields. |
| `mac_key_press` | Send a key press with modifiers. Supports named keys (return, tab, escape, arrows, F-keys) and characters with command/option/control/shift. Also accepts `app=`. |
| `mac_focus_app` | Bring an application to the front and verify it's frontmost. For typing, prefer `app=` on the typing tools instead. |
| `mac_get_clipboard` | Read the system clipboard. |
| `mac_set_clipboard` | Set the system clipboard. |

Scroll amounts are wheel lines, not pixels; visual distance depends on the app.
The scroll event is explicitly positioned at the requested coordinates so an
asynchronous pointer move cannot send it to the previous pointer location.
A successful response confirms event posting, not that the page moved (it may
already be at a boundary or the target may not accept scrolling).

### The focus race (why `app=` exists)

An agent's tool calls are separated by seconds of thinking time. If you call
`mac_focus_app` in one tool call and `mac_type_text` in the next, the terminal
running the agent frequently reclaims focus in the gap — and the text lands in
the agent's own input box instead of the target app. Focus and action must
happen inside a single call. That's what the `app=` parameter on
`mac_type_text` and `mac_key_press` does: activate, wait until the app is
verified frontmost, then act.

### Batch a known sequence

`mac_batch` takes 1–50 actions and an optional default `app`. Each action has
`tool` (an existing tool name) and `args` (that tool's argument object).
The default app applies only to tools with an `app` argument; a per-step app
overrides it. `mac_focus_app` still takes its own explicit `app_name`.

```json
{
  "app": "Firefox",
  "actions": [
    {"tool": "mac_key_press", "args": {"key": "l", "modifiers": ["command"]}},
    {"tool": "mac_type_text", "args": {"text": "https://example.com"}},
    {"tool": "mac_key_press", "args": {"key": "return"}},
    {"tool": "mac_wait", "args": {"condition": "exists", "name": "Example Domain", "role": "AXHeading", "timeout": 5}}
  ]
}
```

This replaces four model/tool round trips with one. It reduces orchestration
overhead; the individual actions still run sequentially and produce results.
All steps are schema-validated before any run, including required arguments,
types, and unknown keys. The batch stops on the first returned error or raised
exception. Earlier actions remain applied; there is no rollback or automatic
retry. Results identify each attempted step by zero-based `index` and `tool`.
`completed` counts successful steps; `failed_index` identifies a failed step.
Validation errors return `status: "invalid"` with `completed: 0`.

Batch only sequences whose next action is already known. A successful search
with zero matches is an observation, not an error: batching does not interpret
UI results, substitute values, or branch. Split at those decision points.
`mac_ui_find` and `mac_ui_press` retain their normal polling behavior.

Use `mac_wait` when the sequence needs a specific postcondition. Unlike an empty
`mac_ui_find` result, a wait timeout is an error and stops the remaining batch.

### Wait for a state change

`mac_wait` takes `condition`, optional `app`, and the following selectors:

| Condition | Additional arguments | Matches when |
| --- | --- | --- |
| `exists` / `absent` | `name` and/or `role` | An element matches / no element matches. |
| `enabled` | `name` and/or `role` | The first matching element reports `AXEnabled`. |
| `value_equals` | `name` and/or `role`, `value` | Its full accessibility value equals the expected text. |
| `value_changed` | `name` and/or `role`, `value` | Its full accessibility value differs from the supplied previous value. |
| `url_equals` / `url_contains` | `value` | The browser address matches the expected text. |
| `url_changed` | `value` | The browser address differs from the supplied previous URL. |
| `title_contains` | `value` | The front window's title contains the text. |

For `*_changed`, capture the previous value **before** the action. Missing URL
or value data cannot satisfy a change. Matching uses case-sensitive comparisons;
element names use the existing token-based search. Element predicates use the
first match, so make selectors specific. App/Accessibility read failures are
errors, not evidence of absence. Waits inspect the selected app without focusing
it; they do not execute, retry, or roll back actions.

`timeout` defaults to 10 seconds (0–30); `interval` defaults to 0.2 seconds
(0.05–1). A zero timeout performs one immediate check. Timeout limits polling;
an individual macOS accessibility call can itself take time. A match confirms
only the requested condition, not that the whole page has finished loading.
An absence condition can match immediately if the element has not appeared yet;
wait for its appearance first when the workflow requires an appear/disappear cycle.

For example, a batch can press the video's `Play (k)` button, then wait for
`{"condition": "exists", "name": "Pause", "role": "AXButton"}`. The model gets
one batch response containing the action result and playback-state evidence.
Successful waits return `status: "matched"`, elapsed seconds, and an observation.
Timeouts return `status: "timeout"`, an error, and the last observation. Tool
errors also set the MCP result's `isError` flag.

Batches cannot contain other batches. Calls through the same server process
cannot interleave with a batch, although human input and other MCP server
processes can still affect the desktop. Keep passing `app` for input tools.

Approving `mac_batch` permits its contained tools, including AppleScript;
clients do not ask separately for every nested step. Review and authorize the
whole sequence just as you would the individual calls.

After updating an existing installation, install requirements and reconnect
the MacBot MCP server (or start a new client session) to discover `mac_batch`.

### Escape hatch

| Tool | Description |
|---|---|
| `mac_run_applescript` | Execute arbitrary AppleScript. Full access to System Events, app scripting dictionaries, and everything else AppleScript can do. |

## Browser automation via the Accessibility API

Modern browsers expose their full UI through the macOS Accessibility API. macbot reads this directly — no browser extension, no WebDriver, no Playwright needed.

**Read the tab bar** with element coordinates:
```
> mac_ui_tree app="Firefox" max_depth=5 max_nodes=50

AXApplication title="Firefox"
  AXWindow title="GitHub - anomalous3/macbot-mcp" @36,30 1752x957
    AXGroup desc="GitHub - anomalous3/macbot-mcp" @36,30 1752x957
      AXToolbar desc="Browser tabs" @36,30 1752x44
        AXTabGroup (tab group) @190,30 1518x44
          AXRadioButton title="GitHub - anomalous3/macbot-mcp" value="True" @193,30 210x44
          AXRadioButton title="New Tab" @403,30 210x44
```

**Find a specific element** and get its click coordinates:
```
> mac_ui_find app="Firefox" name="Submit" role="AXButton"

{
  "match_count": 1,
  "window_titles": ["GitHub - anomalous3/macbot-mcp"],
  "matches": [
    {"role": "AXButton", "title": "Submit", "x": 450, "y": 320, "width": 80, "height": 32}
  ]
}
```

The window titles come back with every search, so an empty result tells you
*why* it was empty — wrong page, still loading, or the window is gone.

**Press it without coordinates** — no mouse, no focus change:
```
> mac_ui_press app="Firefox" name="Submit" role="AXButton"

{"status": "ok", "pressed": {...}, "match_count": 1}
```

**Get the URL** without screenshots or clipboard tricks:
```
> mac_ui_url app="Firefox"

{"url": "https://github.com/anomalous3/macbot-mcp", "field": "Search with Google or enter address"}
```

**Use Firefox.** It exposes the richest accessibility tree of the major browsers — more element detail, better labeling, and more consistent structure than Chrome or Safari.

## The coordinate system

All coordinates are absolute screen pixels in macOS logical coordinates (not Retina physical pixels). Origin (0,0) is the top-left of the primary display.

The `@x,y WxH` format in `mac_ui_tree` output gives position and size directly. To click the center of `@450,320 80x32`, click at `(490, 336)`.

## Works with everything

macbot works with any macOS application that implements the Accessibility API (which is most of them):

- **Browsers** — Firefox, Chrome, Safari (Firefox recommended for richest tree)
- **Terminals** — kitty, Terminal.app, iTerm2 (can read content, type commands)
- **Editors** — VS Code, Sublime Text, TextEdit
- **System apps** — Finder, System Settings, Activity Monitor
- **Any app** — if it has windows and controls, macbot can probably read and drive it

## Configuration

| Variable | Default | Description |
|---|---|---|
| `MACBOT_SCREENSHOT_DIR` | `/tmp/macbot` | Directory for screenshot PNGs |

## Platform

macOS only. For Windows, see [ahk-mcp](https://github.com/anomalous3/ahk-mcp). The approach is the same — read the accessibility tree, act via synthetic input — just different platform APIs.

## License

MIT