nested-display-mcp
Summary: Run and control GUI apps inside visible nested Wayland/Xwayland displays via MCP, with exact screenshots and reliable input without touching the host desktop focus.
Open, list, resize, and close nested displays; optionally launch apps with args, env, and cwd.
Configure display size, host window title, backend (
x11/wayland), and X11 host window position.Capture full-display, cropped-region, scaled, or per-window PNG screenshots.
List windows with ids, titles, app_id/class, shell, rects, focus, PIDs, floating/fullscreen state.
Focus, close, and fullscreen windows.
Move the pointer absolutely or relative to a window, and send relative deltas for games/emulators.
Click, drag, and scroll with configurable button, count, position, and hold duration.
Type text and send key chords such as
ctrl+alt+ReturnorF10.Wait for windows by regex over title/app_id/class, or sleep for a duration.
Work with multiple displays at once, using optional display ids when more than one is open.
Drives Wine and Windows 9x applications and installers inside a nested Wayland/Xwayland display with exact screenshots and reliable keyboard/pointer control, e.g. launching wine setup.exe and interacting with its windows via absolute clicks, chords and typed text.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@nested-display-mcp open a 1024x768 display, launch the installer, and screenshot it"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
nested-display-mcp
An MCP server that runs an application inside a visible nested Wayland display and gives an agent exact screenshots and reliable keyboard/pointer control, without touching the host desktop's focus, cursor acceleration or window management.
Built for driving GUI apps, installers, emulators and old games (Wine/Windows 9x) where injecting input into the real desktop is fragile. Each task gets a real compositor, so screenshots are exact and input never depends on host focus.
┌─────────────┐ stdio ┌────────────────────┐ spawn ┌──────────────────────────────┐
│ MCP client │ ───────► │ nested-display-mcp │ ──────► │ sway (nested, one per display)│
└─────────────┘ └────────────────────┘ │ ├─ WAYLAND_DISPLAY=wayland-N │
▲ │ │ └─ Xwayland :M (Wine/X11) │
│ PNG + JSON │ └──────────────────────────────┘
└────────────────────────┴── grim / wlrctl / wtype / xdotoolWhy sway, and why this shape
Need | How |
A window the user can watch and move | nested sway with |
Exact, palette-correct screenshots |
|
Absolute pointer, no host focus |
|
Keyboard without focus games |
|
Wine / X11 apps | sway starts its own Xwayland; each display gets its own |
Knowing what is on screen |
|
Cage was tested and rejected: it refuses external Xwayland (xwayland_shell_v1 permission
denied), so it cannot run Wine. Gamescope works but has no screencopy (no grim) and only one
window; it stays a possible future "game mode" backend.
Related MCP server: deskwright
Requirements
System packages (Fedora names):
sudo dnf install -y sway grim wlrctl wtype xdotoolOn Debian/Ubuntu: sudo apt install -y sway grim wlrctl wtype xdotool; on Arch:
sudo pacman -S sway grim wlrctl wtype xdotool. Node.js 20 or newer is also required.
The host must have a Wayland session (sway nests inside it) or an X11 session with XWayland;
the default x11 backend needs a host DISPLAY. The MCP server inherits WAYLAND_DISPLAY,
DISPLAY and XDG_RUNTIME_DIR from the client that starts it.
On a host with the proprietary NVIDIA driver, sway refuses to start unless it is launched with
--unsupported-gpu. The server detects that driver (/proc/driver/nvidia/version) and adds the
flag itself; a nested display never scans out through DRM, so nothing else is affected.
Build
npm ci # or npm install
npm run buildInstall
The repo is self-contained: the root is both the agent skill (SKILL.md) and the MCP server
(dist/index.js), so a single checkout gives you both.
git clone https://github.com/gashcrumb/nested-display-mcp.git ~/.agents/skills/nested-display
cd ~/.agents/skills/nested-display
npm ci && npm run buildThen register the server with your client (below) and restart it. Some clients look in their own
skills directory instead of ~/.agents/skills; see
docs/installation.md for client-specific setup, updating and uninstalling.
Smoke-test without an agent:
npm run smoke # opens a real nested display, clicks/types, then closes itRegister with your MCP client
The server is a stdio MCP server: run node /path/to/nested-display-mcp/dist/index.js and speak
MCP on stdin/stdout. For opencode, that looks like this in ~/.config/opencode/opencode.jsonc:
"mcp": {
"servers": {
"nested-display": {
"type": "local",
"command": [
"/usr/bin/node",
"/home/<you>/.agents/skills/nested-display/dist/index.js"
],
"disabled": false
}
}
}Use command -v node for the right binary (nvm paths differ per machine). Other MCP clients use
the same command/args pair in their own config format. Restart the client so the tools are
discovered.
Tools
Tool | Purpose |
| Start a display ( |
| Inspect or tear down displays ( |
| Full-frame PNG, optional region ( |
| Crop to one window's rect |
|
|
| Window and display management |
| Absolute output coordinates (or |
| Delta motion - for PS/2 guests, captured pointers and camera controls |
|
|
| Press, move in steps, release (X11 targets only) |
|
|
| Text entry and chords ( |
| Wait on a regex over title/app_id/class |
| Sleep (e.g. after launching an app) |
All tools take an optional id; with a single display open it can be omitted.
Example
display_open { title: "setup", width: 1024, height: 768, app: "wine", args: ["setup.exe"], env: { WINEDLLOVERRIDES: "ddraw=n" } }
wait_for_window { pattern: "Setup" }
screenshot {}
click { x: 512, y: 384 }
type { text: "hello" }
key { combo: "alt+Return" }
display_close {}Input routing (important)
A target whose
shellisxwaylandgets XTest keyboard input: focus first, then type globally.xdotool type --windowdrops characters in some apps (observed with Wine Notepad).A Wayland target gets wtype / wlrctl. Do not mix the two paths in the same session: a virtual-keyboard keymap upload changes Xwayland's keymap and XTest text then arrives garbled. Never use
wtypeon an X11 target - a stray mapping can arrive as Escape and close the app.Pointer moves always go through the compositor:
wlrctl pointer move -100000 -100000(clamp to the origin), then move by the exact delta.wlrctlmotion is unaccelerated, so this is absolute and exact. XTest warps are not used for positioning because polling toolkits (raylib/GLFW, SDL) never see a motion event from them, leaving their cached cursor position stale. For X11 targets the position is verified withxdotool getmouselocationand a warp is used only as a fallback.The first pointer event after a display is created is swallowed while Xwayland takes pointer focus; the server sends one throwaway move per display before the first real interaction.
Clicks hold the button for 100 ms by default (
hold_ms). Polling UIs checkIsMouseButtonPressedonce per frame, so an atomic press+release between frames is lost.Key chords hold for 80 ms by default (
hold_msonkey) for the same reason: a tap that lands between two polls is invisible toIsKeyPressed.dragis X11-only:zwlr_virtual_pointer_v1has no button-hold action.
Testing
npm run build
npm run smoke # X11 client: open, wait, screenshot, type, key, click, resize, close
node scripts/wayland-test.mjs # Wayland client (foot): wtype + virtual pointer
node scripts/multi-test.mjs # two displays at once, distinct Xwayland displays
node scripts/title-test.mjs # host window title/positionThe smoke test drives the server over stdio exactly like a real client. foot and xterm are
needed for the example clients (not for the server itself).
Machine-specific regression scripts (which drive a particular local app and hardcode its
coordinates) are kept out of the shared tree in scripts/local/ and ignored by git.
Layout
src/index.ts MCP tool definitions
src/display.ts sway lifecycle, sockets, Xwayland detection, host window handling
src/sway.ts swaymsg helpers + window tree flattening
src/input.ts pointer/keyboard routing (XTest vs wlr virtual input)
src/capture.ts grim capture, crop, PNG size
src/util.ts exec helpers, types, PNG header parsing
SKILL.md agent skill (same directory as the server)
docs/ installation guide
scripts/ portable end-to-end tests
scripts/local/ machine-specific tests (gitignored)Known gaps
No audio routing for the nested display (apps use the host PipeWire default).
Clipboard bridging between host and nested display is not implemented.
No OCR / click-text helper yet.
gamescope backend (exact-size game mode, libei injection) is deliberately out of v1.
wlrctl keyboard typehas no named-key action;wtype -kcovers chords.
Available Tools
19 toolsclickClickC
Clicks a mouse button, optionally at absolute output coordinates (or window-relative with relative=true). Uses XTest for X11 targets and the virtual pointer for Wayland targets.
Inputs: x, y, id, count, +4 more.
display_closeClose a nested displayA
Terminates the app and the nested compositor. Pass all=true to close every display.
Inputs: id, all.
display_listList nested displaysA
Lists open displays with their ids, sizes, sockets and app pids.
Inputs: no named parameters.
display_openOpen a nested displayA
Starts a nested sway compositor (visible window on the host desktop, with its own Xwayland) and optionally launches an app inside it. Input and screenshots are routed by this server, so no host focus is ever required. Returns the display id and its inner WAYLAND_DISPLAY/DISPLAY values.
Inputs: id, app, cwd, env, +6 more.
dragDragA
Presses the left button at 'from', moves to 'to' in steps, releases. X11/Xwayland targets only (the virtual pointer protocol cannot hold a button).
Inputs: to_x (required), to_y (required), from_x (required), from_y (required), +3 more.
focusFocus a windowB
Focuses a window inside the display (needed before typing into some apps).
Inputs: window_id (required), id.
fullscreenToggle fullscreenC
Makes the focused window fullscreen (or restores it).
Inputs: id, on.
keyPress a key combinationA
Presses a key or chord, e.g. 'Return', 'ctrl+alt+Return', 'shift+Tab', 'F10'. Uses xdotool names for X11 targets and xkb keysyms for Wayland targets.
Inputs: combo (required), id, hold_ms, window_id.
pointer_moveMove the pointerA
Moves the pointer to absolute output coordinates. X11 targets use XTest; Wayland targets use the compositor's virtual pointer (unaccelerated).
Inputs: x (required), y (required), id, relative, +1 more.
pointer_move_relativeMove the pointer by a deltaA
Relative pointer motion (compositor-side, unaccelerated). Use this for relative-only consumers: emulated PS/2 mice (86Box, QEMU), games that capture the pointer, or camera controls.
Inputs: dx (required), dy (required), id.
resizeResize the displayB
Changes the display resolution. The host window follows, and screenshots/coordinates use the new size.
Inputs: width (required), height (required), id.
screenshotScreenshot the nested displayA
Captures the composited frame (grim) as PNG - exact pixels, palette already resolved, includes Wayland and X11 clients. Coordinates in the image are the same coordinates used by click/pointer_move.
Inputs: x, y, id, scale, +2 more.
screenshot_windowScreenshot one windowC
Captures the window's rectangle from the composited frame.
Inputs: window_id (required), id, scale.
scrollScrollC
Scrolls vertically (dy) and horizontally (dx) at the given position or the current pointer.
Inputs: x, y, dx, dy, +2 more.
typeType textA
Types text into the focused (or given) window. X11/Xwayland targets use XTest, Wayland targets use wtype. Do not mix the two paths in one session: keymap uploads can garble XTest text.
Inputs: text (required), id, window_id.
waitWaitA
Sleeps for the given milliseconds (use sparingly, e.g. after launching an app).
Inputs: ms (required).
wait_for_windowWait for a windowA
Waits until a window whose title, app_id or class matches the regex exists. Returns the window.
Inputs: pattern (required), id, timeout_ms.
window_closeClose a windowB
Asks the compositor to close a window (as if clicking its close button).
Inputs: window_id (required), id.
windowsList windowsA
Lists windows inside the display: id, title, app_id/class, shell (xwayland or xdg_shell), rect, focus and pids. Rects are in output coordinates, matching screenshots.
Inputs: id.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 19 tools
Every tool targets a distinct action or resource: display lifecycle (open/list/close), capture (full frame vs window rect), pointer motion (absolute vs relative), button actions (click/drag/scroll), text input vs key chords, and window management. The descriptions explicitly resolve the few near-neighbors (screenshot vs screenshot_window, pointer_move vs pointer_move_relative, type vs key) by stating when each path applies.
The set mixes conventions: noun_verb prefixed lifecycle tools (display_open/display_list/display_close), bare verbs/nouns (click, drag, scroll, type, key, focus, windows, screenshot), and multiword forms like screenshot_window, pointer_move_relative, wait_for_window. It remains readable and somewhat grouped, but there is no single predictable pattern an agent can rely on.
Nineteen tools is on the heavy side but each maps to a real capability in a rich domain (display lifecycle, screenshots, window management, four input modalities, timing). No obvious duplicates or filler, though a couple of tools (wait, fullscreen) are modest additions to an otherwise dense input surface.
The surface covers the full lifecycle: create/list/close displays, capture, enumerate/focus/close/wait for windows, and drive pointer, buttons, scroll, text, and keys, plus resolution changes. Minor gaps remain—no per-window move/resize (only display resize), and drag is documented as X11-only, limiting Wayland interaction workflows.
Maintenance
Related MCP Connectors
Cloud desktops for AI agents: create a Linux computer, see its screen, click, type and run tasks.
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
A cloud Linux desktop for your AI: browse, click, type, read the screen and run commands.
Launch, inspect, control, and share isolated cloud browsers for your agents.
Related MCP Servers
- AlicenseNot gradedqualityAmaintenanceEnables AI agents to automate Linux desktop GUI by launching and interacting with Wayland applications in isolated virtual KWin sessions, or connecting to live desktops for collaborative automation.159 PyPI81MIT
- AlicenseAqualityBmaintenanceEnables AI agents to operate a real GNOME Wayland desktop through accessibility-tree widget actions, pointer/keyboard input, OCR, window management, and screen capture, optionally on a private headless session.3328 PyPI11Apache 2.0
- AlicenseAqualityCmaintenanceEnables an agent to see and operate a GNOME Wayland desktop like a human, including screenshots, mouse/keyboard control, accessibility tree access, OCR, clipboard, and audio listening.13MIT
- AlicenseNot gradedqualityBmaintenanceGives each agent its own isolated nested Hyprland desktop with a private seat, clipboard and accessibility bus, so it can see the screen and drive a mouse, keyboard, touchpad gestures, windows and apps without touching the user's pointer, focus or windows. Exposes screenshots, zoom, OCR, accessibility-tree control, window management and batched actions as MCP tools, and can optionally act read-only, or with input, on the real desktop.2MIT