mcp-vroid

An MCP server that drives VRoid Studio's GUI. It gives any MCP client (Claude Code, or anything else that speaks the protocol) a set of tools to launch the app, look at it, find widgets in the picture, click and type, set parameters, and export a .vrm — on Arch + Hyprland (Wayland), with VRoid Studio running under Steam/Proton.

There is no scripting API in VRoid Studio, so this works the only way that is available: screenshot the window, locate things with OCR and colour matching, and inject real pointer and keyboard events.

   grim ──► PNG ──► tesseract / cv2 ──► (x, y) ──► virtual pointer / XTEST
    ▲                                                        │
    └────────────────────  screenshot again  ◄───────────────┘

The engine under the server is the tools/vroid-driver spike from my arrakis project, vendored here as mcp_vroid.driver — same code, repackaged so it can be installed and started by an MCP client.

Requirements

Python deps (uv sync installs them): mcp, pillow, numpy, opencv-python-headless, pytesseract, python-xlib.

Related MCP server: blockout-mcp

Install

git clone https://github.com/nhodges/mcp-vroid
cd mcp-vroid
uv sync                 # virtualenv + dependencies
bash native/build.sh    # builds native/vpointer  <-- REQUIRED, not optional

native/build.sh compiles a ~150-line C client for zwlr_virtual_pointer_unstable_v1 (the protocol XML is vendored under native/protocols/). Without it, every pointer tool fails with native/vpointer missing. vroid_status reports whether it is present.

Why a C helper: ydotool is not installed on the reference machine and /dev/uinput is 0600 root:root, so evdev injection would need sudo or a udev rule. The Wayland virtual-pointer protocol needs neither, moves the real compositor cursor, and works over any window.

Register it with a client

Claude Code:

claude mcp add vroid -- uv run --directory /path/to/mcp-vroid mcp-vroid

Generic mcpServers JSON:

{
  "mcpServers": {
    "vroid": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/mcp-vroid", "mcp-vroid"]
    }
  }
}

Clients often launch servers with a sanitised environment. This server recovers XDG_RUNTIME_DIR, WAYLAND_DISPLAY, HYPRLAND_INSTANCE_SIGNATURE and DISPLAY from the runtime dir at startup (src/mcp_vroid/session_env.py) so hyprctl/grim/XTEST work anyway; vroid_status shows what it had to fill in. Anything already in the environment wins.

Optional environment variables:

Tools

Lifecycle

Seeing

Acting (raw input)

Acting (flows)

Every acting tool focuses VRoid first and refuses to act if the focused window is not VRoid Studio.

How to drive it

Mostly: screenshot → look → locate → act → screenshot again.

1. vroid_launch()

2. vroid_screenshot() and look at the image

3. vroid_find_text("Export") (or vroid_find_button()) for coordinates

4. vroid_click(x, y) — coordinates from a fresh capture, always

5. vroid_screenshot() to confirm what actually happened

Rules of thumb learned the hard way in the original spike:

• Read the whole frame, not a crop. A "Close Hairstyle Editor" confirm modal sat in the middle of the screen for six failed clicks because the check only OCR'd the top 60 px.

• Don't judge change by the 3D viewport. VRoid dithers every frame, so a full-window diff reads ~0.98 even when nothing happened. Watch a UI strip.

• Prefer numeric boxes to slider drags. vroid_set_slider types an exact value; dragging is for controls that have no box.

• Primary buttons are found by colour, not text. A grey pill where you expect blue is the app telling you a required field is empty.

• OCR of a full 2560×1440 frame takes ~10 s. Pass a region.

Coordinate spaces

Three spaces are in play, and they are all different:

Tools take and return image px (space="image") by default and convert internally, so hand vroid_find_text output straight to vroid_click. If MCP_VROID_MAX_IMAGE_PX downscaled the picture you were shown, multiply coordinates you read off it by the inverse of the reported downscale factor first — or just ask vroid_find_text, which always reports native px.

UI map (VRoid Studio 2.14.0, English)

Coordinates are image px on a 2560×1440 capture of the fullscreen window. Treat them as hints — the tools locate by OCR first.

Start screen — Create New + card at ≈ (118, 218), caption at (118, 328); New / Open top-right at (2439, 99) / (2495, 100); Sample Models grid below. Create New opens a modal "Select a base to start with" with captions Fem (1199, 862) and Masc (1359, 862) — click the thumbnail ~100 px above the caption.

Editor — tab strip at y ≈ 23: Face 97 · Hairstyle 198 · Body 302 · Outfit 392 · Accessories 509 · Look 622. Hamburger ☰ at (29, 23) → Save (Ctrl+S), Save As… (Ctrl+Shift+S), import/bulk export, undo/redo, back to model selection — Escape does not close this menu, click elsewhere. Toolbar top-right: camera (2415, 23), share/export (2464, 23), kebab ⋮ (2512, 23). Left icon rail (x ≈ 24, first icon y ≈ 77, then every ~48 px) = sub-category for the current tab. Left panel = preset grid with Presets/Custom at y ≈ 120. Right panel = Customize, then Parameters.

Right-panel controls

Body parameters start Model's Height : 161.2 cm, then Fem Height, Masc Height, Body Size, Head Size, Head Width, Head Tip (Y), Neck Length/Thickness/Width, Soften Collarbone, … Face parameters: Eye Size X/Y, Eyes Position (X/Y), Rotate Eye Socket, Inner/Outer Eye Slant, Iris Size X/Y, Gaze (Y), … (~40 rows; the tools scroll for you).

Hair editor — Hairstyle tab → left-rail part icon → Custom sub-tab → + Create New → right panel Edit Hairstyle. Inside: Add Freehand Hair Guides / Add Procedural Hair Guides, a Hair Groups list, a tool palette at (330 / 365 / 398 / 432, 83), undo/redo at (76, 23) / (133, 23). Leaving asks first: the ✕ at (23, 23) pops a Close Hairstyle Editor modal with Save as new item / Overwrite / Close without saving.

Export as VRM — share icon (2464, 23) → Export as VRM → full-screen export page with the blue Export pill at ≈ (2412, 197) → VRM Settings modal (centred, ~x 1000–1560, scrollable): Export Format radios VRM1.0/VRM0.0, Avatar Name required, Version, Creators required, copyright/contact/references, usage checkboxes; the Export pill stays grey and dead until both required fields are filled → Wine save dialog (its own window, title Export): the File name: field opens focused and selected, so typing a Windows path replaces it and Return fires the default button. The Proton prefix maps Z:\ to /, so /home/nuri/x is Z:\home\nuri\x. Do *not* click a Save located by OCR: the Save in: label matches the same needle.

What is brittle

• OCR is the whole locating story. Small, letter-spaced or light-on-dark labels get split or dropped (Export → E + xport). Icons have no text at all — those anchors are hard-coded fractions of the window and will move if pixiv reflows the UI.

• Fixed anchors are fractions calibrated on 2560×1440 at scale 1.25. A different monitor may need them re-measured.

• Modals appear outside your search region and swallow clicks silently.

• Timing. The 3D viewport takes ~5 s to appear after a base is chosen; export takes 5–30 s (longer for heavy models).

• The Wine dialog is a separate window with its own class and geometry — use vroid_screenshot(whole_screen=true) there.

• Language. These needles assume the English UI. If VRoid comes up Japanese, switch it in kebab ⋮ → Settings → Language.

• The idle screensaver can grab the session mid-run. The guard refuses to type into it and closes that one window (and only that one) before acting.

Security note

This server injects real mouse and keyboard events into your live desktop session and takes screenshots of it. That is the entire point, and it is also the risk:

• Screenshots may capture anything on the output — whole_screen=true captures everything, and captures are written to disk unencrypted.

• Keystrokes go to whatever holds focus. The driver refuses to act unless VRoid Studio is the focused window, but a compromised or careless prompt can still click anywhere inside VRoid.

• vroid_launch(restart=true) kills VRoid Studio and loses unsaved work.

• Nothing here is sandboxed and there is no confirmation step.

Run it attended, on a session you are watching, and don't leave an agent driving it unsupervised. vroid_release() gives the desktop back when you're done.

Development

uv run python scripts/smoke_test.py             # start the server, list tools, call vroid_status
uv run python scripts/smoke_test.py --screenshot # + one passive capture if VRoid is open
uv run vroid-driver shot                        # the original driver CLI, still here

vroid-driver (mcp_vroid.driver.cli) is the spike's shell interface — launch, shot, find, click, tab, slider, export, cam, apply-params, … — handy for debugging without an MCP client in the loop.

Credits and licence

The driver (src/mcp_vroid/driver/, native/) started life as the tools/vroid-driver spike in my own arrakis project and is included here with the MCP server wrapped around it.

MIT — see LICENSE.