vincent-vnc-mcp

A self-hosted, VNC-backed MCP server that gives AI agents (Claude Desktop, Codex, etc.) real computer-use control over a dedicated macOS user session on a remote Mac Studio, reached over Tailscale — without disturbing anyone physically using that same Mac Studio at the console.

Getting started (2 steps, no technical setup required)

You'll need: a Mac you want the AI to control (this guide assumes it's a "Mac Studio", but any Mac works), Tailscale installed on both that Mac and your own computer, and a dedicated macOS user account on that Mac that isn't the one you normally sit in front of (System Settings → Users & Groups → Add Account) — this keeps the AI's session separate from whoever's using the Mac in person.

Step 1 — on the Mac Studio: open Terminal (logged in as that dedicated account, e.g. via Screen Sharing) and paste:

git clone https://github.com/cookiemonster0921/vincent-vnc-mcp.git ~/vncmcp && cd ~/vncmcp && ./install.sh

It'll ask a few plain-English questions (a password, where to save recordings) and set everything up automatically. At the end it prints a block of text to paste into Claude Desktop — keep that window open.

Step 2 — on your own computer: open Claude Desktop's settings, find its developer/MCP configuration file, and paste in the block from Step 1. Restart Claude Desktop. Tools like "take a screenshot", "click", "type", and "open an app" now appear, and Claude can control that Mac.

That's it — everything below is reference material for anyone who wants to understand how it works, adjust configuration, or troubleshoot.

Claude Desktop / Codex (your laptop)
        │  mcpServers config: ssh automation@mac-studio.tailnet python3 server.py
        ▼  (stdio JSON-RPC, tunneled over SSH via Tailscale)
┌───────────────────────────────────────────────────────────┐
│ Mac Studio — dedicated "automation" user, headless session │
│                                                             │
│  server.py  (one process, one asyncio event loop)          │
│    ├─ vnc_client.py — one persistent VNC connection to      │
│    │                  localhost:5900 (its own Screen        │
│    │                  Sharing session) + reconnect loop      │
│    │                  + shared in-memory frame cache         │
│    ├─ mcp_tools.py  — @mcp.tool() functions                  │
│    ├─ recorder.py   — ffmpeg + actions.jsonl                 │
│    └─ viewer.py     — FastAPI, bound to Tailscale interface  │
└───────────────────────────────────────────────────────────┘

Related MCP server: macinput

Why this works: macOS Screen Sharing and multiple sessions

macOS Screen Sharing has supported multi-user "Fast User Switching over the network" since Mac OS X 10.7 Lion, and this remains true on current macOS (verified against 15.6 Sequoia). When a VNC/Screen-Sharing client authenticates with a different user account than whoever is logged in at the physical console, macOS creates (or reattaches to) a separate, headless virtual WindowServer session for that account — exactly like Fast User Switching, just triggered remotely. The console user's physical session and screen are completely undisturbed.

(Connecting with the same credentials as the console user instead mirrors or shares their live screen — that's the interruptive case, and it's why this project requires a dedicated, non-console user account, not your own.)

Reference: Apple's VNC access/control guide for Remote Desktop.

Practical consequence: the dedicated account's virtual session is persistent. Disconnecting and reconnecting the VNC client re-attaches to the same running session — open apps, windows, and clipboard state are preserved — it is not recreated on every connection.

Known caveat: headless/virtual sessions can occasionally behave oddly with apps that expect an attached physical display or heavy GPU acceleration. This is a non-issue for typical automation targets (Finder, Safari, TextEdit, Terminal, most productivity apps).

One-time setup on the Mac Studio

1. Create a dedicated macOS user account for automation (e.g. automation). Give it a strong password that is not shared with any admin account or Apple ID.

2. System Settings → General → Sharing → Screen Sharing → Options/"Allow access for" → include the automation user (or "All users").

3. Log in once as automation via Screen Sharing ("log in as different user" when prompted, or open vnc://localhost while at the console as a different user) to establish its persistent virtual session. After this, leave it logged in — do not log it out at the console.

4. Install Tailscale on the Mac Studio and confirm it is reachable at a stable tailnet hostname.

5. Install ffmpeg (brew install ffmpeg) under the automation account.

Quick install (recommended)

On the Mac Studio, logged in as the automation user, get this repo onto disk (e.g. git clone or rsync it to ~/vncmcp), then run:

cd ~/vncmcp
./install.sh

It asks a few plain-English questions (the automation account's Screen Sharing password, where to save recordings, etc.), generates a viewer access token for you, installs everything into a local virtual environment, and runs the integration test automatically. At the end it prints the viewer URL, the access token, and a ready-to-paste Claude Desktop config snippet. Re-run it anytime to change your answers — it remembers what you entered last time.

Manual installation

If you'd rather not use install.sh, on the Mac Studio as the automation user:

cd ~/vncmcp
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

Then create a .env file in this directory (see the table below for the variables) — config.py loads it automatically — or export the same variables in your shell before running server.py.

Configuration (environment variables)

Running

If you used install.sh, your settings are already saved in .env and config.py loads it automatically:

.venv/bin/python server.py

Otherwise, pass the variables via your shell:

VNC_USER=automation VNC_PASSWORD=... AUTH_TOKEN=$(openssl rand -hex 24) \
VIEWER_HOST=<tailscale-ip-of-mac-studio> \
.venv/bin/python server.py

server.py runs the MCP server on stdio (for the SSH transport below) and starts the viewer's HTTP server as a background task in the same process. Logs go to <run_dir>/logs.txt and stderr — never stdout, since stdout carries the MCP JSON-RPC stream.

Each run creates a timestamped directory under RECORDINGS_DIR containing:

• recording.mp4 — screen recording (once start_recording is called)

• actions.jsonl — one JSON line per tool call

• screenshots/ — saved screenshots

• logs.txt — structured, human-readable logs

• metadata.json — run start time, screen size, redacted config snapshot

Running the viewer persistently (LaunchAgent)

server.py itself is meant to be spawned fresh per MCP client connection (see below) — its stdio transport needs a live client attached to stdin, so it's not suitable for launchd directly (with no client, stdin hits EOF immediately and the process would exit right after starting).

For a viewer/manual-tool-tester that's always reachable and survives crashes — viewer_daemon.py is a standalone entrypoint that runs the same VNC connection + viewer HTTP server (via the same app_lifespan) with no MCP session attached, so it can run under launchd indefinitely.

Set it up once:

mkdir -p ~/Library/LaunchAgents ~/vncmcp/launchd_logs
cat > ~/Library/LaunchAgents/com.vincent.vncmcp.plist <<'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key><string>com.vincent.vncmcp</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/vincent/vncmcp/.venv/bin/python3</string>
        <string>/Users/vincent/vncmcp/viewer_daemon.py</string>
    </array>
    <key>WorkingDirectory</key><string>/Users/vincent/vncmcp</string>
    <key>RunAtLoad</key><true/>
    <key>KeepAlive</key><true/>
    <key>ProcessType</key><string>Background</string>
    <key>StandardOutPath</key><string>/Users/vincent/vncmcp/launchd_logs/stdout.log</string>
    <key>StandardErrorPath</key><string>/Users/vincent/vncmcp/launchd_logs/stderr.log</string>
</dict>
</plist>
EOF
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.vincent.vncmcp.plist

Then control it from any folder with the vncmcp command (installed to ~/.local/bin, already on PATH):

vncmcp start     # load and start (also re-applies after editing the plist)
vncmcp stop
vncmcp restart   # e.g. after deploying new code with rsync
vncmcp status
vncmcp logs      # tail the current run's logs.txt
vncmcp open      # open the viewer in your default browser
vncmcp url       # print the viewer URL

Reboot caveat: a LaunchAgent only runs once its owning user's session is active. A dedicated (non-console) user's Screen Sharing session does not survive an actual full reboot the way it survives network drops or sleep — after a reboot, someone needs to reconnect once via Screen Sharing to re-establish that session, at which point KeepAlive/RunAtLoad takes it from there automatically (including auto-restarting if the process ever crashes). This is a limitation of any per-user LaunchAgent approach, not something this project can work around without enabling automatic login (which would show that account at the physical console on every boot — exactly what the dedicated-account approach is meant to avoid).

Running two instances at once: if viewer_daemon.py is already running (via the LaunchAgent) and Claude Desktop separately spawns its own server.py over SSH, both open independent VNC connections (Screen Sharing supports concurrent viewers, so this works) but only the first one to start can bind the viewer's port — the second logs a warning and continues serving MCP tools normally, just without its own viewer. Practically: the always-on LaunchAgent viewer reflects its own idle connection, not necessarily live Claude Desktop activity happening through a separate session at the same moment.

Connecting from Claude Desktop

Since config.py loads .env automatically, the SSH command just needs to run the venv's Python — no secrets need to be inlined here. Add to claude_desktop_config.json (this exact snippet, with your Mac's tailnet name filled in, is also printed at the end of install.sh):

{
  "mcpServers": {
    "mac-studio": {
      "command": "ssh",
      "args": [
        "automation@mac-studio.tailnet",
        "/Users/automation/vncmcp/.venv/bin/python3",
        "/Users/automation/vncmcp/server.py"
      ]
    }
  }
}

Restart Claude Desktop; the tools (get_screen, click, type_text, open_app, etc.) should appear.

Connecting over Tailscale

Both the SSH transport (port 22) and the viewer's HTTP port ride on Tailscale — nothing is exposed to the public internet. Port 5900 (VNC) is not exposed via Tailscale at all; the server always connects to its own localhost:5900.

Viewer usage

Browse to http://<mac-studio-tailscale-ip>:<viewer-port>/ and supply the AUTH_TOKEN either as Authorization: Bearer <token> or HTTP Basic auth (any username, the token as the password). The page polls /frame.jpg and /status every ~400ms and shows connection state, mouse position, last tool run, and recording status.

Manual tool tester

/tools (linked from the viewer's main page) is a plain HTML console for trying tools by hand — useful for debugging without a full agent conversation. It calls the exact same @mcp.tool() functions a real MCP client (Claude Desktop, Codex) would call, grouped by category (Screen, Mouse, Keyboard, Clipboard, Apps, Recording, Status), with a form per tool and a Run button. Destructive tools (run_shell, run_applescript, quit_app) ask for confirmation before running. It's an addon on top of the same auth middleware and viewer process — no separate server, no new ports.

Structured test reports

For agents that test applications, automate multi-step processes, or produce demonstrations, three tools turn a sequence of actions into a persistent, reviewable record: start_test_run(project, goal), log_test_step(run_id, purpose, feature_tested, evaluation, evidence) (captures a fresh screenshot automatically), and finish_test_run(run_id, status, summary). Browse the results at /reports in the viewer — a list of projects, each with its runs, each run showing an ordered table of steps with screenshot thumbnails.

Important: evaluation/evidence are the agent's own self-reported assessment of what it observed — a useful record for a human to review afterward, not a verified, ground-truth test result. The UI labels it as such. Treat a works verdict as "the agent believed this worked and cited this specific evidence for it," not as an assertion you can skip checking yourself.

Reports are stored as JSON files under REPORTS_DIR (default ~/vncmcp_reports) at a fixed path, deliberately not nested in the per-connection run directory — the tool calls that write a report run in whichever process handled that particular MCP connection (e.g. a server.py Claude Desktop spawned over SSH), while the viewer showing /reports is typically the separate, always-on viewer_daemon.py. Both need to agree on one shared location, or the UI would never see what a session wrote.

Claude Code skill

install_skill.sh installs a Claude Code skill (vnc-test-runner) that teaches an agent this workflow — when to call start_test_run, how granular steps should be (one per feature interaction, not one per click), and to require cited evidence rather than asserting a verdict from habit. Unlike install.sh, this runs on the machine driving Claude Code (wherever you talk to Claude from), not the Mac Studio — the skill is workflow guidance for the calling agent, separate from the MCP server itself:

./install_skill.sh                 # installs for this user (~/.claude/skills)
./install_skill.sh --project DIR   # installs into DIR/.claude/skills instead

This is Claude-Code-specific; other MCP clients (Codex, etc.) don't read SKILL.md files, but they still see the three tools themselves via normal MCP tool discovery — their docstrings alone describe the same workflow.

Security

• Never expose the viewer port or VNC port beyond Tailscale.

• Use a strong, unique password for the automation account — don't reuse an admin or Apple ID password (per Apple's own guidance: third-party VNC viewers don't always encrypt keystrokes as robustly as native Remote Desktop; classic RFB/VNC auth is weak by modern standards. This is acceptable here only because the connection never leaves the private, WireGuard-encrypted Tailscale network).

• The viewer requires Bearer-or-Basic auth on every route.

• metadata.json explicitly redacts VNC_PASSWORD and AUTH_TOKEN.

Limitations

• Screenshot-polling viewer (~400ms) — not real-time video.

• Headless virtual sessions can behave unpredictably with GPU-heavy or display-attached-only apps.

• Legacy VNC/RFB auth is weak on its own merits; security here relies on Tailscale as the transport boundary, not on VNC auth strength.

• Single VNC connection, single automation session — not designed for multiple concurrent agents controlling the same Mac.

• Coordinate-based mouse/keyboard control only; no semantic UI understanding (Accessibility tree, OCR) in this MVP.

Testing

.venv/bin/python test_integration.py

Exercises: connect, screenshot, click, type, viewer routes, an MCP tool round-trip, recording start/stop, and reconnect-after-forced-drop. Without VNC_HOST/VNC_USER set, connection-dependent checks are skipped rather than failed, so the script is also useful as a quick post-change sanity check.

Future architecture (design notes)

• OCR (Tesseract or a vision model) — a screen_read_text() tool for reading on-screen text without Accessibility APIs.

• Accessibility API (AXUIElement via PyObjC) — semantic click targets ("click the Save button") instead of raw coordinates; needs Accessibility permission granted to the automation account.

• AppleScript recipe library — named wrappers around common run_applescript queries (frontmost window bounds, etc.) instead of one raw escape hatch.

• Playwright handoff — for web-heavy tasks, detect a frontmost browser and hand control to a Playwright CDP session for reliable DOM-level automation.

• CV/object detection — locate icons/buttons in screenshots when there's no AX tree or OCR match (canvas apps, games, custom-drawn UI).

• Semantic UI understanding — combine AX tree + OCR + CV into one "describe what's on screen" tool returning structured, labeled elements.

• Coordinate scaling — a mapping layer in vnc_client.py between logical tool-call coordinates and physical VNC coordinates, for Retina/resize cases.

• WebRTC streaming — replace screenshot polling with a low-latency video track (e.g. aiortc) if 400ms polling proves too laggy for live babysitting.

• Multiple simultaneous viewers — the shared frame cache already serves any number of stateless /frame.jpg pollers; true multi-viewer gets interesting only with per-viewer cursors/annotations.

• Human override / browser takeover — a /status flag pausing tool execution so a human can take over via the viewer (CAPTCHAs, credentials).

• Session persistence across restarts — persist recent action history / window-layout context so a new conversation can "resume" awareness of what was left open.

• Approval workflows — gate destructive tools (quit_app, risky run_applescript/run_shell calls) behind a human approve/deny step surfaced in the viewer.