# nested-display-mcp

An MCP server that runs an application inside a **visible nested Wayland display** and gives an
agent exact screenshots and reliable keyboard/pointer control, without touching the host desktop's
focus, cursor acceleration or window management.

Built for driving GUI apps, installers, emulators and old games (Wine/Windows 9x) where injecting
input into the real desktop is fragile. Each task gets a real compositor, so screenshots are exact
and input never depends on host focus.

```
┌─────────────┐  stdio   ┌────────────────────┐  spawn  ┌──────────────────────────────┐
│ MCP client  │ ───────► │ nested-display-mcp │ ──────► │ sway (nested, one per display)│
└─────────────┘          └────────────────────┘         │  ├─ WAYLAND_DISPLAY=wayland-N │
        ▲                        │                      │  └─ Xwayland :M (Wine/X11)     │
        │  PNG + JSON            │                      └──────────────────────────────┘
        └────────────────────────┴── grim / wlrctl / wtype / xdotool
```

## Why sway, and why this shape

| Need | How |
| --- | --- |
| A window the user can watch and move | nested sway with `WLR_BACKENDS=x11`: Mutter draws a real title bar and we can position/retitle it |
| Exact, palette-correct screenshots | `grim` captures the composited frame, so Wine's 8bpp palette translation is already resolved |
| Absolute pointer, no host focus | `wlrctl` (virtual pointer) for Wayland clients, XTest for Xwayland clients |
| Keyboard without focus games | `wtype` for Wayland, `xdotool` for Xwayland |
| Wine / X11 apps | sway starts its own Xwayland; each display gets its own `DISPLAY` |
| Knowing what is on screen | `swaymsg -t get_tree` gives titles, classes, geometry, focus and shell per window |

Cage was tested and rejected: it refuses external Xwayland (`xwayland_shell_v1` permission
denied), so it cannot run Wine. Gamescope works but has no screencopy (no grim) and only one
window; it stays a possible future "game mode" backend.

## Requirements

System packages (Fedora names):

```bash
sudo dnf install -y sway grim wlrctl wtype xdotool
```

On Debian/Ubuntu: `sudo apt install -y sway grim wlrctl wtype xdotool`; on Arch:
`sudo pacman -S sway grim wlrctl wtype xdotool`. Node.js **20 or newer** is also required.

The host must have a **Wayland session** (sway nests inside it) or an X11 session with XWayland;
the default `x11` backend needs a host `DISPLAY`. The MCP server inherits `WAYLAND_DISPLAY`,
`DISPLAY` and `XDG_RUNTIME_DIR` from the client that starts it.

## Build

```bash
npm ci          # or npm install
npm run build
```

## Install

The repo is self-contained: the root is **both** the agent skill (`SKILL.md`) and the MCP server
(`dist/index.js`), so a single checkout gives you both.

```bash
git clone https://github.com/gashcrumb/nested-display-mcp.git ~/.agents/skills/nested-display
cd ~/.agents/skills/nested-display
npm ci && npm run build
```

Then register the server with your client (below) and restart it. Some clients look in their own
skills directory instead of `~/.agents/skills`; see
[`docs/installation.md`](docs/installation.md) for client-specific setup, updating and uninstalling.

Smoke-test without an agent:

```bash
npm run smoke     # opens a real nested display, clicks/types, then closes it
```

## Register with your MCP client

The server is a stdio MCP server: run `node /path/to/nested-display-mcp/dist/index.js` and speak
MCP on stdin/stdout. For opencode, that looks like this in `~/.config/opencode/opencode.jsonc`:

```jsonc
"mcp": {
  "servers": {
    "nested-display": {
      "type": "local",
      "command": [
        "/usr/bin/node",
        "/home/<you>/.agents/skills/nested-display/dist/index.js"
      ],
      "disabled": false
    }
  }
}
```

Use `command -v node` for the right binary (nvm paths differ per machine). Other MCP clients use
the same command/args pair in their own config format. Restart the client so the tools are
discovered.

## Tools

| Tool | Purpose |
| --- | --- |
| `display_open` | Start a display (`width`, `height`, `title`, `backend`) and optionally launch `app` |
| `display_list` / `display_close` | Inspect or tear down displays (`all: true` for everything) |
| `screenshot` | Full-frame PNG, optional region (`x/y/width/height`) and `scale` |
| `screenshot_window` | Crop to one window's rect |
| `windows` | `[{id, name, app_id, class, shell, rect, focused, pid, floating, fullscreen}]` |
| `focus` / `fullscreen` / `window_close` / `resize` | Window and display management |
| `pointer_move` | Absolute output coordinates (or `relative` to a window) |
| `pointer_move_relative` | Delta motion - for PS/2 guests, captured pointers and camera controls |
| `click` | `button`, `count`, optional `x/y` + `window_id` |
| `drag` | Press, move in steps, release (X11 targets only) |
| `scroll` | `dx`/`dy` at an optional position |
| `type` / `key` | Text entry and chords (`ctrl+alt+Return`, `F10`, `shift+Tab`) |
| `wait_for_window` | Wait on a regex over title/app_id/class |
| `wait` | Sleep (e.g. after launching an app) |

All tools take an optional `id`; with a single display open it can be omitted.

## Example

```
display_open { title: "setup", width: 1024, height: 768, app: "wine", args: ["setup.exe"], env: { WINEDLLOVERRIDES: "ddraw=n" } }
wait_for_window { pattern: "Setup" }
screenshot {}
click { x: 512, y: 384 }
type { text: "hello" }
key { combo: "alt+Return" }
display_close {}
```

## Input routing (important)

* A target whose `shell` is `xwayland` gets **XTest keyboard** input: focus first, then type
  globally. `xdotool type --window` drops characters in some apps (observed with Wine Notepad).
* A Wayland target gets **wtype** / **wlrctl**. Do not mix the two paths in the same session: a
  virtual-keyboard keymap upload changes Xwayland's keymap and XTest text then arrives garbled.
  Never use `wtype` on an X11 target - a stray mapping can arrive as Escape and close the app.
* Pointer moves always go through the compositor: `wlrctl pointer move -100000 -100000` (clamp
  to the origin), then move by the exact delta. `wlrctl` motion is unaccelerated, so this is
  absolute and exact. XTest warps are not used for positioning because polling toolkits
  (raylib/GLFW, SDL) never see a motion event from them, leaving their cached cursor position
  stale. For X11 targets the position is verified with `xdotool getmouselocation` and a warp is
  used only as a fallback.
* The first pointer event after a display is created is swallowed while Xwayland takes pointer
  focus; the server sends one throwaway move per display before the first real interaction.
* Clicks hold the button for 100 ms by default (`hold_ms`). Polling UIs check
  `IsMouseButtonPressed` once per frame, so an atomic press+release between frames is lost.
* Key chords hold for 80 ms by default (`hold_ms` on `key`) for the same reason: a tap that lands
  between two polls is invisible to `IsKeyPressed`.
* `drag` is X11-only: `zwlr_virtual_pointer_v1` has no button-hold action.

## Testing

```bash
npm run build
npm run smoke          # X11 client: open, wait, screenshot, type, key, click, resize, close
node scripts/wayland-test.mjs   # Wayland client (foot): wtype + virtual pointer
node scripts/multi-test.mjs     # two displays at once, distinct Xwayland displays
node scripts/title-test.mjs     # host window title/position
```

The smoke test drives the server over stdio exactly like a real client. `foot` and `xterm` are
needed for the example clients (not for the server itself).

Machine-specific regression scripts (which drive a particular local app and hardcode its
coordinates) are kept out of the shared tree in `scripts/local/` and ignored by git.

## Layout

```
src/index.ts     MCP tool definitions
src/display.ts   sway lifecycle, sockets, Xwayland detection, host window handling
src/sway.ts      swaymsg helpers + window tree flattening
src/input.ts     pointer/keyboard routing (XTest vs wlr virtual input)
src/capture.ts   grim capture, crop, PNG size
src/util.ts      exec helpers, types, PNG header parsing
SKILL.md         agent skill (same directory as the server)
docs/            installation guide
scripts/         portable end-to-end tests
scripts/local/   machine-specific tests (gitignored)
```

## Known gaps

* No audio routing for the nested display (apps use the host PipeWire default).
* Clipboard bridging between host and nested display is not implemented.
* No OCR / click-text helper yet.
* gamescope backend (exact-size game mode, libei injection) is deliberately out of v1.
* `wlrctl keyboard type` has no named-key action; `wtype -k` covers chords.