# voiceover-mcp

Use `voiceover-mcp` to examine a web page through the actions and speech of a screen-reader user. The server exposes four Model Context Protocol (MCP) tools and keeps one Playwright WebKit session open while an agent moves through the page, listens to each announcement, and decides what to do next.

This guide takes you from installation through a practical accessibility check. See the [interface reference](docs/interface.md) for exact fields, limits, results, and backend support. For the development story and tradeoffs, read [Building `voiceover-mcp`: I Made an AI Use VoiceOver](docs/building-voiceover-mcp.md).

## Choose a Backend

Start with the virtual reader, then confirm important findings with real VoiceOver.

| | `virtual` | `voiceover` |
| --- | --- | --- |
| Reader | Guidepup simulation of the page's accessibility tree | Real macOS VoiceOver |
| Platform | macOS, Linux, or continuous integration | macOS only |
| Browser | Headless by default | Headed WebKit |
| Machine takeover | None | Speaks aloud and controls the keyboard |
| Speed | Fast | Slow because each action uses AppleScript |
| Best use | Routine development and broad checks | Final reader-specific confirmation |

> **Safety:** The `voiceover` backend turns on system-wide VoiceOver and takes over the Mac's speech and keyboard until the session stops. Run it only on a prepared machine, and always call `stop_session`.

## Install and Build

You need Node.js 20 or later.

```bash
cd voiceover-mcp
npm ci
npx playwright install webkit
npm run build
```

These commands install the locked dependencies, install WebKit, compile the TypeScript server, and bundle the virtual reader for browser injection. The virtual backend is now ready.

### Prepare Real VoiceOver

Real VoiceOver needs one-time macOS setup:

```bash
npx @guidepup/setup
```

Then configure the Mac:

1. Grant the terminal, editor, or MCP client that starts the server **Accessibility** and **Automation** access in **System Settings → Privacy & Security**.
2. Open VoiceOver Utility and enable **Allow VoiceOver to be controlled with AppleScript**.
3. Run the real-reader smoke check only after completing both steps.

See Guidepup's [manual VoiceOver setup guide](https://www.guidepup.dev/docs/guides/manual-voiceover-setup) for platform details.

## Connect an MCP Client

Register the built server with an absolute path:

```jsonc
{
  "mcpServers": {
    "voiceover": {
      "command": "node",
      "args": ["/absolute/path/to/voiceover-mcp/dist/index.js"],
      "env": {
        "SCREEN_READER": "virtual"
      }
    }
  }
}
```

Restart the client after changing its configuration. `SCREEN_READER=virtual` gives you the safer default; an explicit `backend` in `start_session` overrides it. Without either setting, the server defaults to `voiceover`.

The server communicates through standard input and output. Keep standard output reserved for MCP JSON-RPC messages; application diagnostics belong on standard error.

## Run Your First Session

### 1. Open a Page

Call `start_session` with the virtual backend:

```json
{
  "url": "https://www.guidepup.dev",
  "backend": "virtual"
}
```

The result confirms the backend, current cursor position, and supported jump targets. A landing warning means the server could not confirm that the cursor entered page content; navigate to the URL again or try one `next` action.

### 2. Inspect the Page

Send related actions in one `screen_reader` call:

```json
{
  "steps": [
    { "do": "findNextHeading" },
    { "do": "next", "count": 2 },
    { "do": "findNextLink" }
  ]
}
```

The response shows where the cursor started, what the reader announced after each action, and where the cursor ended:

```text
Started on: "banner"

Step 1 · findNextHeading
    heading, Guidepup, level 1
Step 2 · next ×2
    Introduction
    Automate screen readers
Step 3 · findNextLink
    Get started, link

Now focused: "Get started, link"
```

The reading cursor persists between calls. Continue from the reported `Now focused` item, or use `navigate` to return to a known page position.

### 3. Interact

Use `act` for links and buttons, `type` for text fields, and `press` for keys:

```json
{
  "steps": [
    { "do": "findNextFormControl" },
    { "do": "type", "text": "screen reader testing" },
    { "do": "press", "keys": "Enter" }
  ]
}
```

If an action opens another page, the server detects the navigation, waits for the page, and moves the reader back into its content. The transcript reports the new URL.

### 4. Stop the Session

Call `stop_session` when the check is complete. The server also attempts cleanup when the MCP connection closes or the process stops, but an explicit stop gives you a clear result.

## Perform a Screen-Reader QA Pass

Use the same order for every page so results remain comparable.

1. **Check structure.** On `virtual`, move through landmarks with `findNextLandmark`. Then inspect the heading outline with `findNextHeading`. Look for a clear page title, logical heading levels, and named regions.
2. **Check navigation.** Move through links with `findNextLink`. Flag empty names, raw URLs, vague phrases such as “click here,” and duplicate names that lead to different places.
3. **Check controls and media.** Use `findNextFormControl` and `findNextGraphic`. Confirm that each control has a useful name and each meaningful image has appropriate alternative text.
4. **Check reading order.** Use `readAll` to read up to 20 items at a time. Confirm that the spoken order matches the page's meaning and that no empty or hidden content interrupts it.
5. **Check keyboard access.** Use `press` with `Tab` and `Shift+Tab`. Confirm a sensible focus order, keyboard operation, a visible focus indicator, and a way out of every component. Use `screenshot` to inspect the visible focus indicator.
6. **Confirm on VoiceOver.** Repeat findings that affect task completion, image text, embeds, or reader-specific wording with the real `voiceover` backend.

Record the exact announcement, action, backend, and URL for each finding. A concrete transcript makes the result reproducible.

## Use the Action Vocabulary

The tool exposes the keyboard actions an ordinary screen-reader user needs:

- Move: `next`, `previous`, `interact`, and `stopInteracting`.
- Activate: `act`.
- Jump: next or previous heading, link, landmark, form control, table, list, or graphic.
- Read: `readAll`.
- Enter input: `type` with `text`, or `press` with `keys`.
- Open a page: `navigate` with `url`.

A `screen_reader` call accepts 1–64 steps. Add `count` from 1–50 to repeat an action. Repeated sweeps stop when the announcement no longer changes, and `readAll` stops after 20 items or the end of content. The [interface reference](docs/interface.md) lists every exact action name.

## Interpret Results Correctly

- **Reading cursor and keyboard focus differ.** `next` and `previous` move the reader cursor. `press` with `Tab` moves browser focus. Test both: an element may be readable but unreachable by keyboard, or focusable but absent from the reading order.
- **No speech is evidence, not a verdict.** `(no new speech)` means the reader added nothing to its speech log. Check the cursor, take a screenshot, or try a nearby action before calling it a defect.
- **Errors stop the batch.** The first failed action reports an error, later actions appear as skipped, and the MCP result sets `isError: true`.
- **Backend wording differs.** The virtual reader often announces role first and includes container boundaries. VoiceOver uses its own phrasing. Compare meaning and order, not literal strings.

## Troubleshoot Common Problems

| Symptom | What to do |
| --- | --- |
| “No screen-reader session is running” | Call `start_session` before `screen_reader` or `screenshot`. |
| VoiceOver is unavailable | Run `npx @guidepup/setup`, verify AppleScript control, and grant Accessibility and Automation permissions. |
| VoiceOver lands outside page content | Bring the WebKit window forward, call `navigate` again, or try `next`. |
| A jump finds nothing | Remember that forward jumps start at the current cursor. Try the matching previous jump or re-run `navigate`. |
| An activation appears to do nothing | Read the navigation note and take a screenshot; the action may have opened another page. |
| Virtual cannot read an embed | Switch to VoiceOver. The virtual reader announces a cross-origin iframe but cannot enter its contents. |
| VoiceOver calls are slow | Expect seconds per action and roughly 30–60 seconds to start. Batch related actions. |

The server supports one WebKit tab and one session per process. The virtual backend provides fast accessibility-tree feedback; it cannot replace confirmation with a real screen reader.

## Run Checks Without an MCP Client

Use the smoke scripts to verify the built server:

```bash
npm run build
npm run smoke:virtual
node scripts/smoke-virtual.mjs https://www.guidepup.dev
```

`smoke:virtual` runs headlessly and saves a screenshot. On a prepared Mac, run `npm run smoke` to perform the same check with real VoiceOver.

## Develop and Verify Changes

The main implementation paths are:

- `src/index.ts`: MCP tools, schemas, backend selection, and shutdown.
- `src/verbs.ts`: action names, required inputs, and execution limits.
- `src/executor.ts`: validation, ordered execution, and captured speech.
- `src/backends/voiceover.ts`: real VoiceOver and headed WebKit.
- `src/backends/virtual.ts`: injected virtual reader and headless WebKit.
- `src/format.ts`: human-readable transcripts.
- `test/voiceover.test.ts`: unit coverage with a fake reader.

Before submitting a change, run:

```bash
npm test
npm run build
npm run smoke:virtual
```

Run `npm run smoke` when the change affects real VoiceOver behavior and the Mac is ready for takeover. Keep the [interface reference](docs/interface.md) synchronized when a tool, action, limit, or result changes.