# vincent-vnc-mcp

A self-hosted, VNC-backed MCP server that gives AI agents (Claude Desktop, Codex,
etc.) real computer-use control over a dedicated macOS user session on a remote
Mac Studio, reached over Tailscale — without disturbing anyone physically using
that same Mac Studio at the console.

## Getting started (2 steps, no technical setup required)

You'll need: a Mac you want the AI to control (this guide assumes it's a "Mac
Studio", but any Mac works), [Tailscale](https://tailscale.com) installed on
both that Mac and your own computer, and a dedicated macOS user account on
that Mac that isn't the one you normally sit in front of (System Settings →
Users & Groups → Add Account) — this keeps the AI's session separate from
whoever's using the Mac in person.

**Step 1 — on the Mac Studio:** open Terminal (logged in as that dedicated
account, e.g. via Screen Sharing) and paste:

```bash
git clone https://github.com/cookiemonster0921/vincent-vnc-mcp.git ~/vncmcp && cd ~/vncmcp && ./install.sh
```

It'll ask a few plain-English questions (a password, where to save
recordings) and set everything up automatically. At the end it prints a
block of text to paste into Claude Desktop — keep that window open.

**Step 2 — on your own computer:** open Claude Desktop's settings, find its
developer/MCP configuration file, and paste in the block from Step 1.
Restart Claude Desktop. Tools like "take a screenshot", "click", "type", and
"open an app" now appear, and Claude can control that Mac.

That's it — everything below is reference material for anyone who wants to
understand how it works, adjust configuration, or troubleshoot.

```
Claude Desktop / Codex (your laptop)
        │  mcpServers config: ssh automation@mac-studio.tailnet python3 server.py
        ▼  (stdio JSON-RPC, tunneled over SSH via Tailscale)
┌───────────────────────────────────────────────────────────┐
│ Mac Studio — dedicated "automation" user, headless session │
│                                                             │
│  server.py  (one process, one asyncio event loop)          │
│    ├─ vnc_client.py — one persistent VNC connection to      │
│    │                  localhost:5900 (its own Screen        │
│    │                  Sharing session) + reconnect loop      │
│    │                  + shared in-memory frame cache         │
│    ├─ mcp_tools.py  — @mcp.tool() functions                  │
│    ├─ recorder.py   — ffmpeg + actions.jsonl                 │
│    └─ viewer.py     — FastAPI, bound to Tailscale interface  │
└───────────────────────────────────────────────────────────┘
```

## Why this works: macOS Screen Sharing and multiple sessions

macOS Screen Sharing has supported multi-user "Fast User Switching over the
network" since Mac OS X 10.7 Lion, and this remains true on current macOS
(verified against 15.6 Sequoia). When a VNC/Screen-Sharing client authenticates
with a **different** user account than whoever is logged in at the physical
console, macOS creates (or reattaches to) a **separate, headless virtual
WindowServer session** for that account — exactly like Fast User Switching,
just triggered remotely. The console user's physical session and screen are
completely undisturbed.

(Connecting with the *same* credentials as the console user instead mirrors or
shares their live screen — that's the interruptive case, and it's why this
project requires a **dedicated, non-console user account**, not your own.)

Reference: [Apple's VNC access/control guide for Remote Desktop](https://support.apple.com/guide/remote-desktop/virtual-network-computing-access-and-control-apde0dd523e/mac).

**Practical consequence:** the dedicated account's virtual session is
persistent. Disconnecting and reconnecting the VNC client re-attaches to the
*same* running session — open apps, windows, and clipboard state are
preserved — it is not recreated on every connection.

**Known caveat:** headless/virtual sessions can occasionally behave oddly with
apps that expect an attached physical display or heavy GPU acceleration. This
is a non-issue for typical automation targets (Finder, Safari, TextEdit,
Terminal, most productivity apps).

## One-time setup on the Mac Studio

1. Create a dedicated macOS user account for automation (e.g. `automation`).
   Give it a strong password that is **not** shared with any admin account or
   Apple ID.
2. System Settings → General → Sharing → **Screen Sharing** → Options/"Allow
   access for" → include the `automation` user (or "All users").
3. Log in **once** as `automation` via Screen Sharing ("log in as different
   user" when prompted, or `open vnc://localhost` while at the console as a
   different user) to establish its persistent virtual session. After this,
   leave it logged in — do not log it out at the console.
4. Install [Tailscale](https://tailscale.com) on the Mac Studio and confirm it
   is reachable at a stable tailnet hostname.
5. Install `ffmpeg` (`brew install ffmpeg`) under the `automation` account.

## Quick install (recommended)

On the Mac Studio, logged in as the `automation` user, get this repo onto disk
(e.g. `git clone` or `rsync` it to `~/vncmcp`), then run:

```bash
cd ~/vncmcp
./install.sh
```

It asks a few plain-English questions (the automation account's Screen
Sharing password, where to save recordings, etc.), generates a viewer access
token for you, installs everything into a local virtual environment, and runs
the integration test automatically. At the end it prints the viewer URL, the
access token, and a ready-to-paste Claude Desktop config snippet. Re-run it
anytime to change your answers — it remembers what you entered last time.

## Manual installation

If you'd rather not use `install.sh`, on the Mac Studio as the `automation` user:

```bash
cd ~/vncmcp
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
```

Then create a `.env` file in this directory (see the table below for the
variables) — `config.py` loads it automatically — or export the same
variables in your shell before running `server.py`.

## Configuration (environment variables)

| Variable         | Default        | Description                                             |
|------------------|----------------|-----------------------------------------------------------|
| `VNC_HOST`       | `localhost`    | VNC target (the server runs on the Mac Studio itself)     |
| `VNC_PORT`       | `5900`         | VNC port                                                   |
| `VNC_USER`       | *(required)*   | The dedicated automation account's username                |
| `VNC_PASSWORD`   | *(required)*   | Its Screen Sharing password                                |
| `VIEWER_HOST`    | auto-detected  | Interface the viewer binds to — `install.sh` auto-detects your Tailscale IP, falling back to `127.0.0.1` |
| `VIEWER_PORT`    | fixed at install| `install.sh` picks a free port once and pins it in `.env` — set to `0` to go back to picking a new free port on every restart |
| `AUTH_TOKEN`     | *(required)*   | Bearer token / Basic-auth password for the viewer          |
| `RECORDINGS_DIR` | `~/vncmcp_runs`| Base directory for per-run recordings/logs/screenshots     |
| `REPORTS_DIR`    | `~/vncmcp_reports` | Fixed, shared location for structured test-run reports (see "Structured test reports" below) — unlike `RECORDINGS_DIR`, all processes must agree on this one path |

## Running

If you used `install.sh`, your settings are already saved in `.env` and
`config.py` loads it automatically:

```bash
.venv/bin/python server.py
```

Otherwise, pass the variables via your shell:

```bash
VNC_USER=automation VNC_PASSWORD=... AUTH_TOKEN=$(openssl rand -hex 24) \
VIEWER_HOST=<tailscale-ip-of-mac-studio> \
.venv/bin/python server.py
```

`server.py` runs the MCP server on stdio (for the SSH transport below) and
starts the viewer's HTTP server as a background task in the same process. Logs
go to `<run_dir>/logs.txt` and stderr — **never** stdout, since stdout carries
the MCP JSON-RPC stream.

Each run creates a timestamped directory under `RECORDINGS_DIR` containing:

- `recording.mp4` — screen recording (once `start_recording` is called)
- `actions.jsonl` — one JSON line per tool call
- `screenshots/` — saved screenshots
- `logs.txt` — structured, human-readable logs
- `metadata.json` — run start time, screen size, redacted config snapshot

### Running the viewer persistently (LaunchAgent)

`server.py` itself is meant to be spawned fresh per MCP client connection (see
below) — its stdio transport needs a live client attached to stdin, so it's
not suitable for `launchd` directly (with no client, stdin hits EOF
immediately and the process would exit right after starting).

For a viewer/manual-tool-tester that's always reachable and survives crashes —
`viewer_daemon.py` is a standalone entrypoint that runs the same VNC
connection + viewer HTTP server (via the same `app_lifespan`) with no MCP
session attached, so it can run under `launchd` indefinitely.

Set it up once:

```bash
mkdir -p ~/Library/LaunchAgents ~/vncmcp/launchd_logs
cat > ~/Library/LaunchAgents/com.vincent.vncmcp.plist <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key><string>com.vincent.vncmcp</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/vincent/vncmcp/.venv/bin/python3</string>
        <string>/Users/vincent/vncmcp/viewer_daemon.py</string>
    </array>
    <key>WorkingDirectory</key><string>/Users/vincent/vncmcp</string>
    <key>RunAtLoad</key><true/>
    <key>KeepAlive</key><true/>
    <key>ProcessType</key><string>Background</string>
    <key>StandardOutPath</key><string>/Users/vincent/vncmcp/launchd_logs/stdout.log</string>
    <key>StandardErrorPath</key><string>/Users/vincent/vncmcp/launchd_logs/stderr.log</string>
</dict>
</plist>
EOF
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.vincent.vncmcp.plist
```

Then control it from any folder with the `vncmcp` command (installed to
`~/.local/bin`, already on `PATH`):

```
vncmcp start     # load and start (also re-applies after editing the plist)
vncmcp stop
vncmcp restart   # e.g. after deploying new code with rsync
vncmcp status
vncmcp logs      # tail the current run's logs.txt
vncmcp open      # open the viewer in your default browser
vncmcp url       # print the viewer URL
```

**Reboot caveat:** a LaunchAgent only runs once its owning user's session is
active. A dedicated (non-console) user's Screen Sharing session does *not*
survive an actual full reboot the way it survives network drops or sleep —
after a reboot, someone needs to reconnect once via Screen Sharing to
re-establish that session, at which point `KeepAlive`/`RunAtLoad` takes it
from there automatically (including auto-restarting if the process ever
crashes). This is a limitation of any per-user LaunchAgent approach, not
something this project can work around without enabling automatic login
(which would show that account at the physical console on every boot —
exactly what the dedicated-account approach is meant to avoid).

**Running two instances at once:** if `viewer_daemon.py` is already running
(via the LaunchAgent) and Claude Desktop separately spawns its own
`server.py` over SSH, both open independent VNC connections (Screen Sharing
supports concurrent viewers, so this works) but only the first one to start
can bind the viewer's port — the second logs a warning and continues serving
MCP tools normally, just without its own viewer. Practically: the always-on
LaunchAgent viewer reflects its own idle connection, not necessarily live
Claude Desktop activity happening through a separate session at the same
moment.

## Connecting from Claude Desktop

Since `config.py` loads `.env` automatically, the SSH command just needs to
run the venv's Python — no secrets need to be inlined here. Add to
`claude_desktop_config.json` (this exact snippet, with your Mac's tailnet
name filled in, is also printed at the end of `install.sh`):

```json
{
  "mcpServers": {
    "mac-studio": {
      "command": "ssh",
      "args": [
        "automation@mac-studio.tailnet",
        "/Users/automation/vncmcp/.venv/bin/python3",
        "/Users/automation/vncmcp/server.py"
      ]
    }
  }
}
```

Restart Claude Desktop; the tools (`get_screen`, `click`, `type_text`,
`open_app`, etc.) should appear.

## Connecting over Tailscale

Both the SSH transport (port 22) and the viewer's HTTP port ride on Tailscale
— nothing is exposed to the public internet. Port 5900 (VNC) is **not**
exposed via Tailscale at all; the server always connects to its own
`localhost:5900`.

## Viewer usage

Browse to `http://<mac-studio-tailscale-ip>:<viewer-port>/` and supply the
`AUTH_TOKEN` either as `Authorization: Bearer <token>` or HTTP Basic auth
(any username, the token as the password). The page polls `/frame.jpg` and
`/status` every ~400ms and shows connection state, mouse position, last tool
run, and recording status.

### Manual tool tester

`/tools` (linked from the viewer's main page) is a plain HTML console for
trying tools by hand — useful for debugging without a full agent
conversation. It calls the exact same `@mcp.tool()` functions a real MCP
client (Claude Desktop, Codex) would call, grouped by category (Screen,
Mouse, Keyboard, Clipboard, Apps, Recording, Status), with a form per tool and
a Run button. Destructive tools (`run_shell`, `run_applescript`, `quit_app`)
ask for confirmation before running. It's an addon on top of the same auth
middleware and viewer process — no separate server, no new ports.

### Structured test reports

For agents that test applications, automate multi-step processes, or produce
demonstrations, three tools turn a sequence of actions into a persistent,
reviewable record: `start_test_run(project, goal)`, `log_test_step(run_id,
purpose, feature_tested, evaluation, evidence)` (captures a fresh screenshot
automatically), and `finish_test_run(run_id, status, summary)`. Browse the
results at `/reports` in the viewer — a list of projects, each with its runs,
each run showing an ordered table of steps with screenshot thumbnails.

**Important:** `evaluation`/`evidence` are the agent's own self-reported
assessment of what it observed — a useful record for a human to review
afterward, not a verified, ground-truth test result. The UI labels it as
such. Treat a `works` verdict as "the agent believed this worked and cited
this specific evidence for it," not as an assertion you can skip checking
yourself.

Reports are stored as JSON files under `REPORTS_DIR` (default
`~/vncmcp_reports`) at a **fixed path**, deliberately not nested in the
per-connection run directory — the tool calls that write a report run in
whichever process handled that particular MCP connection (e.g. a
`server.py` Claude Desktop spawned over SSH), while the viewer showing
`/reports` is typically the separate, always-on `viewer_daemon.py`. Both
need to agree on one shared location, or the UI would never see what a
session wrote.

#### Claude Code skill

`install_skill.sh` installs a Claude Code skill (`vnc-test-runner`) that
teaches an agent this workflow — when to call `start_test_run`, how granular
steps should be (one per feature interaction, not one per click), and to
require cited evidence rather than asserting a verdict from habit. Unlike
`install.sh`, **this runs on the machine driving Claude Code** (wherever you
talk to Claude from), not the Mac Studio — the skill is workflow guidance for
the calling agent, separate from the MCP server itself:

```bash
./install_skill.sh                 # installs for this user (~/.claude/skills)
./install_skill.sh --project DIR   # installs into DIR/.claude/skills instead
```

This is Claude-Code-specific; other MCP clients (Codex, etc.) don't read
`SKILL.md` files, but they still see the three tools themselves via normal
MCP tool discovery — their docstrings alone describe the same workflow.

## Security

- Never expose the viewer port or VNC port beyond Tailscale.
- Use a strong, unique password for the automation account — don't reuse an
  admin or Apple ID password (per Apple's own guidance: third-party VNC
  viewers don't always encrypt keystrokes as robustly as native Remote
  Desktop; classic RFB/VNC auth is weak by modern standards. This is
  acceptable here only because the connection never leaves the private,
  WireGuard-encrypted Tailscale network).
- The viewer requires Bearer-or-Basic auth on every route.
- `metadata.json` explicitly redacts `VNC_PASSWORD` and `AUTH_TOKEN`.

## Limitations

- Screenshot-polling viewer (~400ms) — not real-time video.
- Headless virtual sessions can behave unpredictably with GPU-heavy or
  display-attached-only apps.
- Legacy VNC/RFB auth is weak on its own merits; security here relies on
  Tailscale as the transport boundary, not on VNC auth strength.
- Single VNC connection, single automation session — not designed for
  multiple concurrent agents controlling the same Mac.
- Coordinate-based mouse/keyboard control only; no semantic UI understanding
  (Accessibility tree, OCR) in this MVP.

## Testing

```bash
.venv/bin/python test_integration.py
```

Exercises: connect, screenshot, click, type, viewer routes, an MCP tool
round-trip, recording start/stop, and reconnect-after-forced-drop. Without
`VNC_HOST`/`VNC_USER` set, connection-dependent checks are skipped rather than
failed, so the script is also useful as a quick post-change sanity check.

## Future architecture (design notes)

- **OCR** (Tesseract or a vision model) — a `screen_read_text()` tool for
  reading on-screen text without Accessibility APIs.
- **Accessibility API (`AXUIElement` via PyObjC)** — semantic click targets
  ("click the Save button") instead of raw coordinates; needs Accessibility
  permission granted to the automation account.
- **AppleScript recipe library** — named wrappers around common
  `run_applescript` queries (frontmost window bounds, etc.) instead of one
  raw escape hatch.
- **Playwright handoff** — for web-heavy tasks, detect a frontmost browser and
  hand control to a Playwright CDP session for reliable DOM-level automation.
- **CV/object detection** — locate icons/buttons in screenshots when there's
  no AX tree or OCR match (canvas apps, games, custom-drawn UI).
- **Semantic UI understanding** — combine AX tree + OCR + CV into one
  "describe what's on screen" tool returning structured, labeled elements.
- **Coordinate scaling** — a mapping layer in `vnc_client.py` between logical
  tool-call coordinates and physical VNC coordinates, for Retina/resize cases.
- **WebRTC streaming** — replace screenshot polling with a low-latency video
  track (e.g. `aiortc`) if 400ms polling proves too laggy for live babysitting.
- **Multiple simultaneous viewers** — the shared frame cache already serves
  any number of stateless `/frame.jpg` pollers; true multi-viewer gets
  interesting only with per-viewer cursors/annotations.
- **Human override / browser takeover** — a `/status` flag pausing tool
  execution so a human can take over via the viewer (CAPTCHAs, credentials).
- **Session persistence across restarts** — persist recent action history /
  window-layout context so a new conversation can "resume" awareness of what
  was left open.
- **Approval workflows** — gate destructive tools (`quit_app`, risky
  `run_applescript`/`run_shell` calls) behind a human approve/deny step
  surfaced in the viewer.