Skip to main content
Glama
nested-display-mcp

nested-display-mcp

by gashcrumb

nested-display-mcp

An MCP server that runs an application inside a visible nested Wayland display and gives an agent exact screenshots and reliable keyboard/pointer control, without touching the host desktop's focus, cursor acceleration or window management.

Built for driving GUI apps, installers, emulators and old games (Wine/Windows 9x) where injecting input into the real desktop is fragile. Each task gets a real compositor, so screenshots are exact and input never depends on host focus.

┌─────────────┐  stdio   ┌────────────────────┐  spawn  ┌──────────────────────────────┐
│ MCP client  │ ───────► │ nested-display-mcp │ ──────► │ sway (nested, one per display)│
└─────────────┘          └────────────────────┘         │  ├─ WAYLAND_DISPLAY=wayland-N │
        ▲                        │                      │  └─ Xwayland :M (Wine/X11)     │
        │  PNG + JSON            │                      └──────────────────────────────┘
        └────────────────────────┴── grim / wlrctl / wtype / xdotool

Why sway, and why this shape

Need

How

A window the user can watch and move

nested sway with WLR_BACKENDS=x11: Mutter draws a real title bar and we can position/retitle it

Exact, palette-correct screenshots

grim captures the composited frame, so Wine's 8bpp palette translation is already resolved

Absolute pointer, no host focus

wlrctl (virtual pointer) for Wayland clients, XTest for Xwayland clients

Keyboard without focus games

wtype for Wayland, xdotool for Xwayland

Wine / X11 apps

sway starts its own Xwayland; each display gets its own DISPLAY

Knowing what is on screen

swaymsg -t get_tree gives titles, classes, geometry, focus and shell per window

Cage was tested and rejected: it refuses external Xwayland (xwayland_shell_v1 permission denied), so it cannot run Wine. Gamescope works but has no screencopy (no grim) and only one window; it stays a possible future "game mode" backend.

Related MCP server: deskwright

Requirements

System packages (Fedora names):

sudo dnf install -y sway grim wlrctl wtype xdotool

On Debian/Ubuntu: sudo apt install -y sway grim wlrctl wtype xdotool; on Arch: sudo pacman -S sway grim wlrctl wtype xdotool. Node.js 20 or newer is also required.

The host must have a Wayland session (sway nests inside it) or an X11 session with XWayland; the default x11 backend needs a host DISPLAY. The MCP server inherits WAYLAND_DISPLAY, DISPLAY and XDG_RUNTIME_DIR from the client that starts it.

On a host with the proprietary NVIDIA driver, sway refuses to start unless it is launched with --unsupported-gpu. The server detects that driver (/proc/driver/nvidia/version) and adds the flag itself; a nested display never scans out through DRM, so nothing else is affected.

Build

npm ci          # or npm install
npm run build

Install

The repo is self-contained: the root is both the agent skill (SKILL.md) and the MCP server (dist/index.js), so a single checkout gives you both.

git clone https://github.com/gashcrumb/nested-display-mcp.git ~/.agents/skills/nested-display
cd ~/.agents/skills/nested-display
npm ci && npm run build

Then register the server with your client (below) and restart it. Some clients look in their own skills directory instead of ~/.agents/skills; see docs/installation.md for client-specific setup, updating and uninstalling.

Smoke-test without an agent:

npm run smoke     # opens a real nested display, clicks/types, then closes it

Register with your MCP client

The server is a stdio MCP server: run node /path/to/nested-display-mcp/dist/index.js and speak MCP on stdin/stdout. For opencode, that looks like this in ~/.config/opencode/opencode.jsonc:

"mcp": {
  "servers": {
    "nested-display": {
      "type": "local",
      "command": [
        "/usr/bin/node",
        "/home/<you>/.agents/skills/nested-display/dist/index.js"
      ],
      "disabled": false
    }
  }
}

Use command -v node for the right binary (nvm paths differ per machine). Other MCP clients use the same command/args pair in their own config format. Restart the client so the tools are discovered.

Tools

Tool

Purpose

display_open

Start a display (width, height, title, backend) and optionally launch app

display_list / display_close

Inspect or tear down displays (all: true for everything)

screenshot

Full-frame PNG, optional region (x/y/width/height) and scale

screenshot_window

Crop to one window's rect

windows

[{id, name, app_id, class, shell, rect, focused, pid, floating, fullscreen}]

focus / fullscreen / window_close / resize

Window and display management

pointer_move

Absolute output coordinates (or relative to a window)

pointer_move_relative

Delta motion - for PS/2 guests, captured pointers and camera controls

click

button, count, optional x/y + window_id

drag

Press, move in steps, release (X11 targets only)

scroll

dx/dy at an optional position

type / key

Text entry and chords (ctrl+alt+Return, F10, shift+Tab)

wait_for_window

Wait on a regex over title/app_id/class

wait

Sleep (e.g. after launching an app)

All tools take an optional id; with a single display open it can be omitted.

Example

display_open { title: "setup", width: 1024, height: 768, app: "wine", args: ["setup.exe"], env: { WINEDLLOVERRIDES: "ddraw=n" } }
wait_for_window { pattern: "Setup" }
screenshot {}
click { x: 512, y: 384 }
type { text: "hello" }
key { combo: "alt+Return" }
display_close {}

Input routing (important)

  • A target whose shell is xwayland gets XTest keyboard input: focus first, then type globally. xdotool type --window drops characters in some apps (observed with Wine Notepad).

  • A Wayland target gets wtype / wlrctl. Do not mix the two paths in the same session: a virtual-keyboard keymap upload changes Xwayland's keymap and XTest text then arrives garbled. Never use wtype on an X11 target - a stray mapping can arrive as Escape and close the app.

  • Pointer moves always go through the compositor: wlrctl pointer move -100000 -100000 (clamp to the origin), then move by the exact delta. wlrctl motion is unaccelerated, so this is absolute and exact. XTest warps are not used for positioning because polling toolkits (raylib/GLFW, SDL) never see a motion event from them, leaving their cached cursor position stale. For X11 targets the position is verified with xdotool getmouselocation and a warp is used only as a fallback.

  • The first pointer event after a display is created is swallowed while Xwayland takes pointer focus; the server sends one throwaway move per display before the first real interaction.

  • Clicks hold the button for 100 ms by default (hold_ms). Polling UIs check IsMouseButtonPressed once per frame, so an atomic press+release between frames is lost.

  • Key chords hold for 80 ms by default (hold_ms on key) for the same reason: a tap that lands between two polls is invisible to IsKeyPressed.

  • drag is X11-only: zwlr_virtual_pointer_v1 has no button-hold action.

Testing

npm run build
npm run smoke          # X11 client: open, wait, screenshot, type, key, click, resize, close
node scripts/wayland-test.mjs   # Wayland client (foot): wtype + virtual pointer
node scripts/multi-test.mjs     # two displays at once, distinct Xwayland displays
node scripts/title-test.mjs     # host window title/position

The smoke test drives the server over stdio exactly like a real client. foot and xterm are needed for the example clients (not for the server itself).

Machine-specific regression scripts (which drive a particular local app and hardcode its coordinates) are kept out of the shared tree in scripts/local/ and ignored by git.

Layout

src/index.ts     MCP tool definitions
src/display.ts   sway lifecycle, sockets, Xwayland detection, host window handling
src/sway.ts      swaymsg helpers + window tree flattening
src/input.ts     pointer/keyboard routing (XTest vs wlr virtual input)
src/capture.ts   grim capture, crop, PNG size
src/util.ts      exec helpers, types, PNG header parsing
SKILL.md         agent skill (same directory as the server)
docs/            installation guide
scripts/         portable end-to-end tests
scripts/local/   machine-specific tests (gitignored)

Known gaps

  • No audio routing for the nested display (apps use the host PipeWire default).

  • Clipboard bridging between host and nested display is not implemented.

  • No OCR / click-text helper yet.

  • gamescope backend (exact-size game mode, libei injection) is deliberately out of v1.

  • wlrctl keyboard type has no named-key action; wtype -k covers chords.

Available Tools

19 tools
clickClickC

Clicks a mouse button, optionally at absolute output coordinates (or window-relative with relative=true). Uses XTest for X11 targets and the virtual pointer for Wayland targets.

Inputs: x, y, id, count, +4 more.

display_closeClose a nested displayA

Terminates the app and the nested compositor. Pass all=true to close every display.

Inputs: id, all.

display_listList nested displaysA

Lists open displays with their ids, sizes, sockets and app pids.

Inputs: no named parameters.

display_openOpen a nested displayA

Starts a nested sway compositor (visible window on the host desktop, with its own Xwayland) and optionally launches an app inside it. Input and screenshots are routed by this server, so no host focus is ever required. Returns the display id and its inner WAYLAND_DISPLAY/DISPLAY values.

Inputs: id, app, cwd, env, +6 more.

dragDragA

Presses the left button at 'from', moves to 'to' in steps, releases. X11/Xwayland targets only (the virtual pointer protocol cannot hold a button).

Inputs: to_x (required), to_y (required), from_x (required), from_y (required), +3 more.

focusFocus a windowB

Focuses a window inside the display (needed before typing into some apps).

Inputs: window_id (required), id.

fullscreenToggle fullscreenC

Makes the focused window fullscreen (or restores it).

Inputs: id, on.

keyPress a key combinationA

Presses a key or chord, e.g. 'Return', 'ctrl+alt+Return', 'shift+Tab', 'F10'. Uses xdotool names for X11 targets and xkb keysyms for Wayland targets.

Inputs: combo (required), id, hold_ms, window_id.

pointer_moveMove the pointerA

Moves the pointer to absolute output coordinates. X11 targets use XTest; Wayland targets use the compositor's virtual pointer (unaccelerated).

Inputs: x (required), y (required), id, relative, +1 more.

pointer_move_relativeMove the pointer by a deltaA

Relative pointer motion (compositor-side, unaccelerated). Use this for relative-only consumers: emulated PS/2 mice (86Box, QEMU), games that capture the pointer, or camera controls.

Inputs: dx (required), dy (required), id.

resizeResize the displayB

Changes the display resolution. The host window follows, and screenshots/coordinates use the new size.

Inputs: width (required), height (required), id.

screenshotScreenshot the nested displayA

Captures the composited frame (grim) as PNG - exact pixels, palette already resolved, includes Wayland and X11 clients. Coordinates in the image are the same coordinates used by click/pointer_move.

Inputs: x, y, id, scale, +2 more.

screenshot_windowScreenshot one windowC

Captures the window's rectangle from the composited frame.

Inputs: window_id (required), id, scale.

scrollScrollC

Scrolls vertically (dy) and horizontally (dx) at the given position or the current pointer.

Inputs: x, y, dx, dy, +2 more.

typeType textA

Types text into the focused (or given) window. X11/Xwayland targets use XTest, Wayland targets use wtype. Do not mix the two paths in one session: keymap uploads can garble XTest text.

Inputs: text (required), id, window_id.

waitWaitA

Sleeps for the given milliseconds (use sparingly, e.g. after launching an app).

Inputs: ms (required).

wait_for_windowWait for a windowA

Waits until a window whose title, app_id or class matches the regex exists. Returns the window.

Inputs: pattern (required), id, timeout_ms.

window_closeClose a windowB

Asks the compositor to close a window (as if clicking its close button).

Inputs: window_id (required), id.

windowsList windowsA

Lists windows inside the display: id, title, app_id/class, shell (xwayland or xdg_shell), rect, focus and pids. Rects are in output coordinates, matching screenshots.

Inputs: id.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.4/5.0

    Scored across 19 tools

    Disambiguation5/5

    Every tool targets a distinct action or resource: display lifecycle (open/list/close), capture (full frame vs window rect), pointer motion (absolute vs relative), button actions (click/drag/scroll), text input vs key chords, and window management. The descriptions explicitly resolve the few near-neighbors (screenshot vs screenshot_window, pointer_move vs pointer_move_relative, type vs key) by stating when each path applies.

    Naming Consistency3/5

    The set mixes conventions: noun_verb prefixed lifecycle tools (display_open/display_list/display_close), bare verbs/nouns (click, drag, scroll, type, key, focus, windows, screenshot), and multiword forms like screenshot_window, pointer_move_relative, wait_for_window. It remains readable and somewhat grouped, but there is no single predictable pattern an agent can rely on.

    Tool Count4/5

    Nineteen tools is on the heavy side but each maps to a real capability in a rich domain (display lifecycle, screenshots, window management, four input modalities, timing). No obvious duplicates or filler, though a couple of tools (wait, fullscreen) are modest additions to an otherwise dense input surface.

    Completeness4/5

    The surface covers the full lifecycle: create/list/close displays, capture, enumerate/focus/close/wait for windows, and drive pointer, buttons, scroll, text, and keys, plus resolution changes. Minor gaps remain—no per-window move/resize (only display resize), and drag is documented as X11-only, limiting Wayland interaction workflows.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents to automate Linux desktop GUI by launching and interacting with Wayland applications in isolated virtual KWin sessions, or connecting to live desktops for collaborative automation.
      159 PyPI
      81
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables AI agents to operate a real GNOME Wayland desktop through accessibility-tree widget actions, pointer/keyboard input, OCR, window management, and screen capture, optionally on a private headless session.
      33
      28 PyPI
      11
      Apache 2.0
    • A
      license
      A
      quality
      C
      maintenance
      Enables an agent to see and operate a GNOME Wayland desktop like a human, including screenshots, mouse/keyboard control, accessibility tree access, OCR, clipboard, and audio listening.
      13
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Gives each agent its own isolated nested Hyprland desktop with a private seat, clipboard and accessibility bus, so it can see the screen and drive a mouse, keyboard, touchpad gestures, windows and apps without touching the user's pointer, focus or windows. Exposes screenshots, zoom, OCR, accessibility-tree control, window management and batched actions as MCP tools, and can optionally act read-only, or with input, on the real desktop.
      2
      MIT