MCP VRoid — macOS & Linux

Control VRoid Studio through MCP on macOS and Linux/Hyprland. Launch VRoid, capture its window, locate controls with OCR, edit character parameters, and run save/export flows from an MCP client.

This is ohm41321/mcp-vroid, our version of nhodges/mcp-vroid. It adds a native macOS backend while retaining the original Linux/Hyprland backend. Both platforms use the same 18 MCP tools.

Platform support

The backend is selected automatically: macos on macOS, hyprland on Linux. Override it with MCP_VROID_BACKEND=macos or MCP_VROID_BACKEND=hyprland. Other Linux compositors and Windows are not currently supported.

Status: experimental GUI automation calibrated for VRoid Studio 2.14.0 (English UI). Run desktop actions attended.

Related MCP server: da-mcp

What's included in this version

• Native macOS window discovery, capture and mouse/keyboard input through Quartz, AppKit and CGEvent, with Accessibility and Screen Recording checks.

• A shared driver interface for macOS and Linux, including platform-aware shortcuts (cmd maps to Command on macOS and Control on Linux).

• OCR fallback for light-grey UI labels and Retina-aware UI anchors.

• Unicode handling for macOS text input, including emoji.

• Display-free unit tests and a GitHub Actions matrix for Linux and macOS.

Why

VRoid Studio has no scripting API, no CLI, no plugin surface. The only way in is the one a person uses: look at the window and move the mouse. So that is what this does — screenshot the window (grim on Hyprland, screencapture on macOS), locate things with OCR and colour matching, and inject real pointer and keyboard events at the compositor level. The MCP client's model is the eyes; these tools are the hands.

   grim / screencapture ──► PNG ──► tesseract / cv2 ──► (x, y) ──► virtual pointer + XTEST / CGEvent
    ▲                                                                          │
    └──────────────────────────  screenshot again  ◄───────────────────────────┘

Demo

These screenshots come from the upstream Linux/Hyprland demo, driven by an MCP client. They illustrate the shared tools; they are not evidence of a completed macOS export.

Setting body parameters by typing exact values into the Parameters panel (vroid_open_tab("Body") → vroid_set_slider("Head Size", -0.15)):

Inside the hair editor, tuning procedural hair guides (vroid_click on the group, vroid_set_slider on Height / Interval / Twist Intensity):

Filling the VRM Settings modal on the way to an export (vroid_export_vrm walks the whole flow, including Wine's save dialog):

Requirements

Two backends, picked by platform (MCP_VROID_BACKEND=hyprland|macos overrides): everything desktop-specific lives in src/mcp_vroid/driver/backends/.

Linux: Hyprland

The upstream Linux backend was developed and tested on Arch Linux + Hyprland, with VRoid Studio 2.14.0 (English UI) running under Steam/Proton. Requirements:

So: wlroots + Xwayland for the input and capture layer, Hyprland only for window management. On Arch:

sudo pacman -S grim tesseract tesseract-data-eng wayland gcc pkgconf

macOS

Tested on macOS 26 (Apple silicon, Retina) with the native VRoid Studio 2.14.0 build from vroid.com (bundle net.pixiv.vroid.macosx). No Steam, no Wine, no native helper to compile.

Permissions: open System Settings → Privacy & Security → Accessibility and → Screen Recording. Grant access to the responsible process shown by macOS: this may be your terminal or MCP client, or the Python executable used by uv. Granting only the host app is not always sufficient. Restart the host and MCP server after granting, then check that vroid_status reports helpers.accessibility and helpers.screen_recording as true. Acting tools refuse to run without Accessibility.

What "workspace 9" means here: vroid_launch brings VRoid to the front and maximises its window to the screen's visible frame (menu bar excluded). The Unity window has no native fullscreen — its zoom button is a plain AXZoomButton and AXFullScreen only re-zooms it — so the 28 pt title bar stays; the backend trims it from the reported geometry and from captures, so y = 0 is the tab strip like on Hyprland. vroid_release re-activates the app you were in before (switching the Space back if VRoid was in another one). Window captures normally use the window id (screencapture -l), which works from any Space without activating VRoid. If that capture does not match the reported bounds (for example, a Stage Manager thumbnail), the fallback may bring VRoid forward and switch Spaces before capturing its screen region.

Two things that bit during bring-up, both handled in the backend: macOS attributes permissions to the responsible process, which under some hosts is the python3.12 binary in uv's cache rather than the host app (check the Accessibility list for it); and Unity only sees ⌘/⇧ when the modifiers arrive as FlagsChanged events, so plain key-down events for ⌘ made Cmd+Shift+S a no-op while AppKit's save panel accepted them fine.

Quickstart

Both platforms require Python 3.11+, uv and an installed copy of VRoid Studio with its UI set to English.

git clone https://github.com/ohm41321/mcp-vroid.git
cd mcp-vroid
uv sync --locked        # virtualenv + platform-specific dependencies

On Linux/Hyprland, also build the required pointer helper:

bash native/build.sh

On macOS, install OCR and grant the permissions described above:

brew install tesseract

On Linux, native/build.sh compiles a ~150-line C client for the Wayland virtual-pointer protocol (the protocol XML is vendored under native/protocols/). Without it every pointer tool fails with native/vpointer missing; vroid_status tells you whether it is there. On macOS there is nothing to build — grant the two permissions instead.

Register it with Claude Code:

claude mcp add vroid -- uv run --directory /path/to/mcp-vroid mcp-vroid

…or with any client that takes an mcpServers block:

{
  "mcpServers": {
    "vroid": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/mcp-vroid", "mcp-vroid"]
    }
  }
}

Then ask your client to call vroid_status, and if it looks healthy, vroid_launch().

Clients often start servers with a sanitised environment. On Linux this server recovers XDG_RUNTIME_DIR, WAYLAND_DISPLAY, HYPRLAND_INSTANCE_SIGNATURE and DISPLAY from the runtime dir at startup (src/mcp_vroid/session_env.py), so hyprctl / grim / XTEST work anyway. vroid_status reports what it had to fill in; anything already in the environment wins. macOS needs nothing recovered.

Optional environment variables:

<state-home> is $XDG_STATE_HOME, or ~/.local/state when unset, on both platforms. Captures and exports are kept outside the checkout by default.

Tools

18 tools, in four groups.

Lifecycle

Seeing

Acting — raw input

Acting — flows

Every acting tool focuses VRoid first and refuses to act if the focused window is not VRoid Studio.

How it works

The loop is see → locate → act → see again:

1. vroid_launch()

2. vroid_screenshot() — the image goes to the client's model, which looks at it

3. vroid_find_text("Export") or vroid_find_button() for coordinates

4. vroid_click(x, y) — always with coordinates from a fresh capture

5. vroid_screenshot() to confirm what actually happened

Seeing is grim (Hyprland) on the window geometry or screencapture -l (macOS) on the window id, then tesseract for word boxes and OpenCV for solid-colour buttons (VRoid's primary pills are #0096FA, and OCR reliably loses white-on-blue labels). OCR runs a plain and an inverted (light-on-dark) pass, and a midtone-boosted pass for VRoid's light-grey captions when those two find nothing.

Acting on Hyprland goes down two different paths, for annoying reasons:

• Pointer — a small C client (native/vpointer.c) speaking zwlr_virtual_pointer_unstable_v1. It moves the real compositor cursor, so hover states and drags behave exactly as they do for a human, and it needs no permissions: ydotool's /dev/uinput route is 0600 root:root and would need sudo or a udev rule.

• Keyboard and wheel — X11 XTEST through Xwayland, because the virtual-pointer protocol has no keyboard counterpart and VRoid is an Xwayland client anyway.

On macOS everything is a CGEvent posted on the HID event tap: mouse moves/drags/clicks (with a real click count for double-clicks), scroll-wheel lines, and keyboard events that carry both an ANSI virtual key code (so ⌘-shortcuts land) and the Unicode string (so any character types).

Coordinate spaces. Three are in play and they are all different:

Tools take and return image px (space="image") by default and convert internally, so vroid_find_text output can be handed straight to vroid_click. If MCP_VROID_MAX_IMAGE_PX downscaled the picture you were shown, multiply coordinates read off it by the inverse of the reported downscale — or just ask vroid_find_text, which always reports native px.

Rules of thumb, learned the hard way:

• Read the whole frame, not a crop. A "Close Hairstyle Editor" confirm modal sat in the middle of the screen through six failed clicks because the check only OCR'd the top 60 px.

• Don't judge change by the 3D viewport. VRoid dithers every frame, so a full-window diff reads ~0.98 even when nothing happened. Watch a UI strip.

• Prefer numeric boxes to slider drags. vroid_set_slider types an exact value; dragging is for controls that have no box.

• Primary buttons are found by colour, not text. A grey pill where you expect blue is the app telling you a required field is empty.

A detailed map of VRoid's UI — tab strip, rails, panels, the export flow, the hair editor, with measured coordinates — is in docs/ui-map.md.

Limitations and brittleness

This is GUI automation with no API underneath. Be realistic about it:

• OCR is the whole locating story, and it is imperfect. Small, letter-spaced or light-on-dark labels get split or dropped (Export → E + xport). White-on-blue is lost entirely, which is why vroid_find_button exists. Icons have no text at all — those anchors are hard-coded in VRoid's UI points, measured from the nearest window edge.

• Coupled to the UI version. Needles and icon anchors were calibrated on VRoid Studio 2.14.0, English, at 2560×1440 / scale 1.25. A pixiv UI reflow, another language, or a different monitor can require re-measuring. (Japanese UI → kebab ⋮ → Settings → Language.) The macOS build draws the same UI at 2 px per point; the anchors are expressed in UI points from the window edges, and the toolbar icons, rail, parameter boxes and colour boxes were checked against a live 2940×1790 capture. Modal-centre regions are still window fractions.

• Modals appear outside your search region and swallow clicks silently.

• Timing is guessed. The 3D viewport takes ~5 s after a base is chosen; export takes 5–30 s, longer for heavy models.

• The save dialog is a separate window — Wine's, with its own class and geometry, or a centred NSSavePanel on macOS — use vroid_screenshot(whole_screen=true) there. On macOS the path goes in via ⌘⇧G ("Go to the folder") then the file name; verified up to the point of pressing Save, which an actual export has not yet exercised here.

• Save As to an existing file is not handled reliably. The replacement confirmation is unhandled and a stale file can satisfy the completion check. Use a new .vroid filename. VRM export deletes an existing target before starting; use a new .vrm path to preserve previous exports.

• Single instance, single session, single display. One VRoid window, one desktop, no headless mode, no parallelism. It drives your screen; on macOS the primary display (the one with the menu bar at 0,0) is assumed.

• The idle screensaver can grab the session mid-run (Linux). The guard refuses to type into it and closes that one window (and only that one) before acting.

• Attended use is recommended. See below.

Security

This server injects real mouse and keyboard events into your live desktop session and takes screenshots of it. That is the entire point, and it is also the risk:

• Screenshots may capture anything on the output — whole_screen=true captures everything, and captures are written to disk unencrypted.

• Keystrokes go to whatever holds keyboard focus. The driver refuses to act unless VRoid Studio is focused, but a careless or hostile prompt can still click anywhere inside VRoid.

• Screenshot and OCR tools can bring VRoid forward if the macOS window-id capture fails its geometry check while VRoid is on another Space.

• vroid_launch(restart=true) kills VRoid Studio and loses unsaved work.

• Nothing here is sandboxed and there is no confirmation step.

Run it attended, on a session you are watching. Don't run it on a shared or multi-user machine, don't leave an agent driving it unsupervised, and treat the captures directory as sensitive. vroid_release() gives the desktop back when you're done.

Development

uv run pytest -q                                 # display-free unit tests (gestures, key maps, both backends)
uv run python scripts/smoke_test.py              # start the server, list tools, call vroid_status
uv run python scripts/smoke_test.py --screenshot # + one passive capture if VRoid is open
uv run vroid-driver shot                         # the original driver CLI, still here

The unit tests require no running VRoid instance or desktop permissions. The smoke test checks MCP startup and status on a configured desktop; --screenshot also needs Screen Recording on macOS. These checks do not prove that a save or export completes successfully.

vroid-driver (mcp_vroid.driver.cli) is a shell interface to the same engine — launch, shot, find, click, tab, slider, export, cam, apply-params, … — handy for debugging without an MCP client in the loop.

Layout:

src/mcp_vroid/server.py       MCP tool definitions (stdio)
src/mcp_vroid/session_env.py  recovers the Wayland/X session env (no-op on macOS)
src/mcp_vroid/driver/         the engine
  window.py                   find / launch / focus / workspaces (platform-neutral surface)
  capture.py                  Shot + coordinate spaces
  locate.py                   tesseract OCR + colour button matching
  input.py                    gestures: glide-click, drag, wheel, type, keys
  actions.py                  the VRoid-specific flows
  backends/hyprland.py        hyprctl + grim + vpointer + XTEST
  backends/macos.py           Quartz window list + screencapture + CGEvent + AppKit
native/vpointer.c             zwlr_virtual_pointer client (Linux)
tests/                        display-free unit tests

Contributing

Open issues and pull requests in this repository. Useful contributions include:

• A port to another compositor or OS. Implement the module-level functions in driver/backends/hyprland.py (a Sway port is a swaymsg rewrite of the window half; grim and zwlr_virtual_pointer already work there) and register the name in backends/__init__.py.

• A pixel-by-pixel re-measure of the anchors on macOS, and a report of the export flow end to end there.

• Anchors for other resolutions or DPI scales, or for the Japanese UI.

• Bug reports — include your compositor, VRoid Studio version, monitor resolution and scale, and the output of vroid_status. A capture from the failing step helps enormously.

GitHub Actions runs the display-free unit tests on Linux and macOS with Python 3.11 and 3.12. Desktop smoke tests and live VRoid flows must be run locally. There is no formatter or linter configured; match the surrounding style.

Credits and licence

Original project and Linux implementation: nhodges/mcp-vroid by Nuri Hodges. This repository adds native macOS support, cross-platform driver structure, tests and setup documentation. The original MIT copyright notice is preserved in LICENSE.

MIT — see LICENSE. VRoid Studio is a product of pixiv Inc.; this project is unaffiliated with pixiv and simply drives the app's UI.