# macbot-mcp

An MCP server that gives AI agents hands on macOS — via the Accessibility API, AppleScript, and Quartz CGEvents.

The macOS counterpart to [ahk-mcp](https://github.com/anomalous3/ahk-mcp). Same thesis: the accessibility tree already contains a machine-readable description of everything on screen. Screenshots throw that away and make the model re-derive it from pixels. Why?

Token cost per action: **~200-700 tokens** (vs ~2000-3500 for screenshot-based).

Because the output is structured text, not images, **any language model can drive it** — including small open-source models with no vision capability. A 7B model can parse `AXButton title="Save" @450,320 88x32` and call `mac_click(x=490, y=336)`. It cannot interpret a screenshot. The accessibility approach makes computer use available to models that were previously locked out of it entirely.

## How it works

macbot-mcp exposes 15 tools over MCP's stdio transport:

- **Observation tools** read the macOS accessibility tree, window properties, and browser URLs — returning structured text with element roles, names, values, and **screen coordinates**
- **Action tools** click, drag, scroll, type, and send keystrokes via Quartz CGEvents and AppleScript
- **`mac_run_applescript`** is the escape hatch — execute arbitrary AppleScript for anything the built-in tools don't cover

Every observation tool returns coordinates. Find a button with `mac_ui_find`, get its `x,y` position, and click it with `mac_click` — no screenshot needed.

## Token cost comparison

| Approach | Tokens per action | What you get |
|---|---|---|
| Screenshot-based (full screen PNG) | ~2000-3500 | Pixels. Model must OCR, locate elements, interpret layout. |
| macbot-mcp (structured text) | ~200-700 | Element roles, titles, values, and pixel coordinates. |

A 20-step workflow: ~50k tokens with screenshots, ~8k with macbot. The structured output is also more reliable — the model doesn't guess where the "Save" button is when the accessibility tree says `AXButton title="Save" @1043,672 88x32`.

## Installation

### Prerequisites

- **macOS 12+** (Monterey or later)
- **Python 3.10+**

### Setup

```bash
git clone https://github.com/anomalous3/macbot-mcp.git
cd macbot-mcp

# Create a virtual environment
python3 -m venv .venv
# or: uv venv .venv

# Activate
source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt
```

### macOS Permissions

macbot needs two permissions, granted to your terminal app (Terminal, kitty, iTerm2, etc.):

1. **Accessibility** — System Settings > Privacy & Security > Accessibility
2. **Screen Recording** — System Settings > Privacy & Security > Screen Recording

Without Accessibility, observation and input tools won't work. Without Screen Recording, `mac_screenshot` will fail.

**Three gotchas:**

- **Granting is not enough — fully quit and reopen the terminal** (Cmd-Q, not
  just a new tab/window). Running processes keep their old permission state.
- **macOS 15+ periodically re-asks for Screen Recording.** If screenshots worked
  yesterday and now fail with `could not create image from display` (and
  `mac_get_windows` returns `[]`), respond to the re-consent prompt in System
  Settings, then restart the terminal.
- If you run Claude Code inside an IDE's integrated terminal (VS Code etc.),
  grant the permissions to the **IDE**, not Terminal.app.

## Claude Code MCP configuration

Add to your `~/.claude.json`:

```json
{
  "mcpServers": {
    "macbot": {
      "command": "/path/to/macbot-mcp/.venv/bin/python3",
      "args": ["/path/to/macbot-mcp/server.py"],
      "env": {
        "MACBOT_SCREENSHOT_DIR": "/tmp/macbot"
      }
    }
  }
}
```

After adding the config, restart Claude Code. The tools appear with the `mcp__macbot__` prefix.

## Tool reference

### Observation

| Tool | Description |
|---|---|
| `mac_ui_tree` | Dump the accessibility tree of any app. Returns roles, titles, values, descriptions, and screen coordinates. Configurable depth and node limit. **This is the primary observation tool — use it before reaching for screenshots.** |
| `mac_ui_find` | Search for UI elements by name and/or role. Returns matches with coordinates. Find that "Submit" button without scanning the whole tree. |
| `mac_ui_url` | Get the current URL from the active browser's address bar (Firefox, Chrome, Safari). |
| `mac_get_windows` | List all visible windows with app name, title, position, and size. |
| `mac_screenshot` | Capture full screen, a specific window, or a region. Returns a PNG file path. **The fallback when you genuinely need pixels.** |

### Action

| Tool | Description |
|---|---|
| `mac_ui_press` | **Press an element by name/role directly through the Accessibility API — no coordinates, no mouse, no focus change.** Works on occluded windows and elements scrolled out of view. Prefer this over find+click for buttons, links, and tabs. |
| `mac_click` | Click at screen coordinates. Uses Quartz CGEvents (falls back to cliclick if installed). Accepts `app=` to bring the target window frontmost first — a click lands on whatever window is at those coordinates. |
| `mac_drag` | Click and drag between two points. Configurable duration for smooth drags. Works for rotating 3D plots, selecting text, moving windows. |
| `mac_scroll` | Scroll at a screen position. Vertical and horizontal. Also accepts `app=`. |
| `mac_type_text` | Type text into an app. Pass `app=` to focus the target atomically — recommended, see below. Uses clipboard paste by default (fast, Unicode-safe). Optional keystroke mode for modifier-sensitive fields. |
| `mac_key_press` | Send a key press with modifiers. Supports named keys (return, tab, escape, arrows, F-keys) and characters with command/option/control/shift. Also accepts `app=`. |
| `mac_focus_app` | Bring an application to the front and verify it's frontmost. For typing, prefer `app=` on the typing tools instead. |
| `mac_get_clipboard` | Read the system clipboard. |
| `mac_set_clipboard` | Set the system clipboard. |

### The focus race (why `app=` exists)

An agent's tool calls are separated by seconds of thinking time. If you call
`mac_focus_app` in one tool call and `mac_type_text` in the next, the terminal
running the agent frequently reclaims focus in the gap — and the text lands in
the agent's own input box instead of the target app. Focus and action must
happen inside a single call. That's what the `app=` parameter on
`mac_type_text` and `mac_key_press` does: activate, wait until the app is
verified frontmost, then act.

### Escape hatch

| Tool | Description |
|---|---|
| `mac_run_applescript` | Execute arbitrary AppleScript. Full access to System Events, app scripting dictionaries, and everything else AppleScript can do. |

## Browser automation via the Accessibility API

Modern browsers expose their full UI through the macOS Accessibility API. macbot reads this directly — no browser extension, no WebDriver, no Playwright needed.

**Read the tab bar** with element coordinates:
```
> mac_ui_tree app="Firefox" max_depth=5 max_nodes=50

AXApplication title="Firefox"
  AXWindow title="GitHub - anomalous3/macbot-mcp" @36,30 1752x957
    AXGroup desc="GitHub - anomalous3/macbot-mcp" @36,30 1752x957
      AXToolbar desc="Browser tabs" @36,30 1752x44
        AXTabGroup (tab group) @190,30 1518x44
          AXRadioButton title="GitHub - anomalous3/macbot-mcp" value="True" @193,30 210x44
          AXRadioButton title="New Tab" @403,30 210x44
```

**Find a specific element** and get its click coordinates:
```
> mac_ui_find app="Firefox" name="Submit" role="AXButton"

{
  "match_count": 1,
  "window_titles": ["GitHub - anomalous3/macbot-mcp"],
  "matches": [
    {"role": "AXButton", "title": "Submit", "x": 450, "y": 320, "width": 80, "height": 32}
  ]
}
```

The window titles come back with every search, so an empty result tells you
*why* it was empty — wrong page, still loading, or the window is gone.

**Press it without coordinates** — no mouse, no focus change:
```
> mac_ui_press app="Firefox" name="Submit" role="AXButton"

{"status": "ok", "pressed": {...}, "match_count": 1}
```

**Get the URL** without screenshots or clipboard tricks:
```
> mac_ui_url app="Firefox"

{"url": "https://github.com/anomalous3/macbot-mcp", "field": "Search with Google or enter address"}
```

**Use Firefox.** It exposes the richest accessibility tree of the major browsers — more element detail, better labeling, and more consistent structure than Chrome or Safari.

## The coordinate system

All coordinates are absolute screen pixels in macOS logical coordinates (not Retina physical pixels). Origin (0,0) is the top-left of the primary display.

The `@x,y WxH` format in `mac_ui_tree` output gives position and size directly. To click the center of `@450,320 80x32`, click at `(490, 336)`.

## Works with everything

macbot works with any macOS application that implements the Accessibility API (which is most of them):

- **Browsers** — Firefox, Chrome, Safari (Firefox recommended for richest tree)
- **Terminals** — kitty, Terminal.app, iTerm2 (can read content, type commands)
- **Editors** — VS Code, Sublime Text, TextEdit
- **System apps** — Finder, System Settings, Activity Monitor
- **Any app** — if it has windows and controls, macbot can probably read and drive it

## Configuration

| Variable | Default | Description |
|---|---|---|
| `MACBOT_SCREENSHOT_DIR` | `/tmp/macbot` | Directory for screenshot PNGs |

## Platform

macOS only. For Windows, see [ahk-mcp](https://github.com/anomalous3/ahk-mcp). The approach is the same — read the accessibility tree, act via synthetic input — just different platform APIs.

## License

MIT