# mac-reader

An MCP server that reads and acts on a Mac's screen **from another computer, over SSH**, through
the macOS Accessibility API. You get the window's structured content (roles, names, values,
enabled state) as text instead of a screenshot for a vision model to guess at. **Nothing is
installed on the Mac.**

## Why

Driving a Mac from a Windows PC usually means screen sharing and screenshots. But macOS already
maintains a structured tree of every window's controls for accessibility, and it can be read over
a plain SSH login. It needs no remote desktop session and no app on the Mac, just Python's
standard library.

It has been used for real work: driving Xcode through an archive build and a TestFlight upload
from a Windows PC, reading the archive list, the run-destination picker and the distribution
flow as text.

## How it works

Each tool call:

1. **finds the Mac by its SSH host key**, not by its address (`find_mac.py`, below);
2. **sends `mac_agent.py` over SSH** to the Mac's built-in `/usr/bin/python3 -`;
3. **reads the JSON it prints back.**

`mac_agent.py` calls the Accessibility API through `ctypes`, so there is no pyobjc and no pip on
the Mac side.

## Setup

### On the Mac

1. **Turn on Remote Login:** System Settings → General → Sharing → Remote Login.
2. **Allow key login** from the computer you'll drive it from: add that computer's public key to
   `~/.ssh/authorized_keys` on the Mac. Recommended: turn password login off, since this only
   ever uses a key. Put this in `/etc/ssh/sshd_config.d/000-key-only.conf` (needs `sudo`):
   ```
   PasswordAuthentication no
   KbdInteractiveAuthentication no
   ```
3. **Grant Accessibility to `/usr/libexec/sshd-keygen-wrapper`:** System Settings → Privacy &
   Security → Accessibility → **+**, then press ⌘⇧G and type the path. macOS runs that program
   for every SSH login, so this is where the permission has to go for SSH commands. Without it,
   every read comes back empty and the tool says so.
4. **Read the Mac's host key at the Mac itself** (not over the network, so you know it's the
   real one):
   ```bash
   cat /etc/ssh/ssh_host_ed25519_key.pub
   ```

### On your computer

```bash
pip install -r requirements.txt
cp mac.example.json mac.json
```

Fill in `mac.json` (it's gitignored):

| field | what |
|---|---|
| `user` | your login name on the Mac |
| `host_key` | the whole line from step 4: `ssh-ed25519 AAAA...` |
| `host` | optional. A fixed address or hostname, if you have one. Then no network scan is needed. |
| `subnet` | optional. The network to scan, e.g. `192.168.1.0/24`. Default: the /24 around this computer. |

Check it can find the Mac:

```bash
python find_mac.py
```

Register it with Claude Code at user scope, so every project can use it:

```bash
claude mcp add mac-reader -s user -- python "/path/to/mac-reader/server.py"
```

Replace `/path/to/mac-reader` with wherever you cloned this repo.

## Tools

| Tool | What it does |
|---|---|
| `list_windows` | Running apps and their windows. The front app is marked. Start here. |
| `read_window(app, window?)` | The window's element tree as text. |
| `find_in_window(app, query)` | Elements whose role, name or value contains the query. Shows the exact names to click. |
| `click_element(app, name, role?, index?, allow_mouse?)` | Presses a control **once**. Cursor-free unless `allow_mouse=True`. |
| `type_into_element(app, name, text)` | Sets a field's value (no keystrokes), then reads it back to verify. |
| `simulator_screenshot(device)` | A PNG from a booted iOS Simulator device, saved on your computer. |

## Finding the Mac safely

**A device is only "the Mac" if it proves it with the Mac's SSH host key.** Something answering
on port 22 is not identity: after the router reshuffles addresses, another device can hold the
old one. So:

- every candidate is checked **by `ssh` itself**: a connection pinned to the key in `mac.json`
  runs `exit 0`, and any other key stops ssh before it logs in;
- every real command uses the same pinned connection (`HostKeyAlias`, a dedicated known_hosts
  file, `StrictHostKeyChecking=yes`), so connecting to an impostor **fails** instead of working;
- `BatchMode=yes`: it logs in with a key or not at all, and **never asks for a password**;
- if two devices ever present the Mac's key, it refuses to pick one;
- "presented a different key" (a security warning) and "couldn't connect" (a network problem) are
  reported separately, because they need different responses.

It tries the last verified address first, then scans the local network (about 3 seconds for a
/24). It won't scan anything larger than 1024 addresses.

## Safety rules

Each of these came from a real incident, on this tool or its Windows counterpart.

- **Press at most once.** `AXPress` is sent once. It falls back to `AXPick` or `AXConfirm` only
  when the app reports the action **was never delivered** (unsupported, invalid element).
  - `kAXErrorCannotComplete` does *not* fall through: it also comes back when the app got the
    press and answered slowly. Treating that as "didn't happen" is how one click becomes two.
  - An action is never retried over SSH either: the connection can drop after the command ran.
    Reads retry once.
- **Confirm by what changed.** After a press it watches the window for a second. If nothing
  changed, it says **unconfirmed** and warns not to press again.
- **The real pointer is opt-in.** Accessibility actions don't move it. `allow_mouse=True` is for
  controls with no action at all (Xcode's run-destination picker is one). It moves the pointer
  for one click and puts it back, even if the click fails.
- **Never type a password.** Password fields (`AXSecureTextField`) are refused, with no override.
- **Read-only apps.** System Settings, Passwords and Keychain Access can be read but never acted
  in, with no override. The Claude desktop app needs `allow_other_session=True`, because acting
  there uses another Claude session's permissions.
- **Ambiguous matches are listed, never guessed.** Pass `index` only after reading the list.

## What we learned building it

Most of this isn't written down anywhere we could find.

- **The permission goes to `/usr/libexec/sshd-keygen-wrapper`.** Not to Terminal, not to Python,
  not to `sshd`.
- **SSH runs in a background session.** From there, asking the system-wide Accessibility element
  "which app is focused?" fails with `kAXErrorCannotComplete`. Asking each app directly
  (`AXUIElementCreateApplication(pid)`) works, with pids from `lsappinfo`, which needs no
  permission.
- **A locked screen doesn't fail, it lies.** Reads succeed, but every window title comes back as
  its app's name. Every reply here ends with a LOCKED note when `CGSSessionScreenIsLocked` is set.
- **Decode `AXUIElement` references.** They have their own CoreFoundation type. A decoder that
  only knows strings, numbers, booleans and point/size values silently returns nothing for
  `AXMenuBar`, `AXParent` and `AXFocusedWindow`.
- **Invisible characters break matching.** Calculator's display reads U+200E followed by "0". It
  looks like "0" and never matches it. Names are normalised (format characters dropped,
  whitespace collapsed) before matching.
- **Xcode 26's SwiftUI panes are readable this way.** AppleScript's System Events saw no text or
  buttons in the Organizer's archive list. This tool read all of it: rows, versions, statuses,
  **Distribute App**, **Validate App**. The content sits inside SwiftUI hosting views, which is
  our guess at why, not confirmed.
- **Windows' built-in `ssh-keyscan` can't talk to current macOS.** Windows OpenSSH 9.5 fails to
  negotiate a key exchange with macOS's OpenSSH 10.3 (`unsupported KEX method
  sntrup761x25519-sha512@openssh.com`), and Git for Windows' `ssh-keyscan` sometimes hung past
  its own timeout. Plain `ssh` works with both, so identity is checked with `ssh` itself.
- **WebView apps expose nothing in the Simulator.** A Capacitor app's content group had zero
  children while the screenshot showed a full page. `simulator_screenshot` exists so a read can
  be checked against something that doesn't come from Accessibility.

## Tested

With macOS 26.6 (OpenSSH 10.3) as the Mac, driven from Windows 11.

- **Identity:** the real Mac is recognised through both Windows OpenSSH's `ssh` and Git for
  Windows' `ssh`. A freshly generated decoy key pinned in its place is rejected through both,
  and `ssh -v` shows it stopping at the host-key check with no key offered for login. An
  unroutable address is reported as a connection failure, not a mismatch.
- **Presses:** on Calculator, `7` was pressed once and confirmed by a separate read. `All Clear`
  pressed at 0 came back **unconfirmed**, correctly, so the confirmation can fail. With the press
  replaced by a recorder: a timeout sends `AXPress` only, "unsupported" sends `AXPress` then
  `AXPick`, and the old "any error falls through" rule is caught sending all three.
- **Writes:** a value set in a TextEdit file read back identically. Password fields: refused,
  zero writes.
- **Refusals:** System Settings and the Claude app refused as described.

Linux and macOS as the *driving* computer should work (it only needs `ssh` and Python), but
haven't been tried.

## Not yet tested

- **Presses inside a native iOS app in the Simulator.** Its elements offer `AXPress`; none has
  been sent yet.
- **Menus and sheets that open in a separate window after a press.** The confirmation only
  watches the pressed window, so it may say "unconfirmed" for a press that worked. Read the
  screen before doing anything else.
- **Xcode Cloud prompts.** "Xcode Cloud can automate app signing and distribution" appears on
  Product → Archive in Xcode 26. Whether its buttons are readable here isn't known yet. Its
  default button is "Get Started…", so never press Enter at it blind.

## Related

[uia-reader](https://github.com/thomiasj/uia-reader) does the same for native Windows windows,
through UI Automation.

## License

[MIT](LICENSE)