ultranix-mcp

**ultranix-mcp is the enterprise-grade, secure Linux desktop-automation layer for AI agents.**It gives Model Context Protocol (MCP) clients - Claude Desktop, Claude Code, Cursor, Windsurf, and any MCP-enabled assistant - the ability to see, click, type, and drive a Wayland desktop: mouse, keyboard, screenshots, OCR, icon finding, window management, and accessibility-tree inspection.

ultranix-mcp is Wayland-native by design: compositor protocols first, uinput/evdev second, XDG Desktop Portals last - behind the same governance-and-trust surface as its siblings (ultramac on macOS, ultrawin on Windows): audit logging, rate limiting, input sanitization, and AES-256-GCM-encrypted action history. It is the first Linux desktop MCP to combine a cross-compositor fallback ladder, a full governance surface, and a tri-OS sibling contract - organisations can let agents control a Linux desktop without giving up control themselves.

**Status:**v1.4.0 implemented. Phases 0-5 of ROADMAP.md have shipped, plus the v1.1.0 wave (layer-shell overlay, X11-native providers, PipeWire portal capture, opt-in Sentry, OCR cache, additional metrics), the v1.2.0 breadth wave (clipboard tools, plugin tool-macros, screen_record, sway/Wayfire/river/KDE/GNOME session detection, per-backend cargo features, framed history v2), the v1.3.0 policy wave (runtime access-control policy with per-key roles, --readonly/--allow-tools/--deny-tools, per-backend invocation metrics, optional HMAC-signed audit lines), and the v1.4.0 reach wave (Wayfire/river/GNOME window rungs, live screen_stream capture, plugin-exposed dynamic tools, OCI image + -bin package) - see CHANGELOG.md for per-release notes. The verified target environment is CachyOS (Arch) + Hyprland on Wayland, PipeWire, xdg-desktop-portal-hyprland, and a live AT-SPI2 bus, on Rust 1.98.1.

Features

• Precision Mouse Control- click, double-click, drag, scroll, button-state control, position queries, and smooth path movement via the zwlr_virtual_pointer_v1 protocol. No root, no helper daemons.

• Advanced Keyboard Input- type text and drive key states through virtual-keyboard-unstable-v1, with full modifier and keymap handling.

• Intelligent Vision- in-process wlr-screencopy-unstable-v1 capture, ONNX Runtime OCR (ort crate), and OWL-ViT icon finding. Region screenshots, color sampling (color_at), and session spatial focus (set_spatial_focus scopes screenshot/find_text_on_screen/find_icon), and real screen_highlight overlays via zwlr_layer_shell_v1 (translucent, click-through; -32010 ProviderUnavailable on compositors/sessions without layer-shell - see docs/TOOLS.md).

• Window Management- list, focus, move, resize, close, and inspect windows through Hyprland's hyprctl IPC socket (hyprctl -j JSON: clients, activewindow, dispatch, workspaces), sway's own IPC protocol on $SWAYSOCK, Wayfire's ipc/ipc-rules plugins on $WAYFIRE_SOCKET, the pinned riverctl subprocess + wlroots foreign-toplevel composite on river (riverctl keeps focused-view close and relative move/resize deltas; the toplevel protocol supplies enumeration and per-window focus/close/min/max/ fullscreen), wlr-toplevel as the shared wlroots fallback rung and the sole rung on unknown wlroots sessions, kdotool on KDE, the "Window Calls" Shell extension on GNOME (when installed), or wmctrl on X11 sessions.

• UI Inspection- full accessibility-tree access over AT-SPI2 (atspi crate): UI-tree dumps, focused-element queries, element search, and wait-for-element synchronization.

• Clipboard & Plugins- clipboard_get/clipboard_set/ clipboard_clear over wl-clipboard (Wayland) or xclip/xsel (X11), writes consent-gated; and declarative plugin tool-macros - ~/.ultranix-mcp/plugins/*.json manifests of catalog-tool steps run via plugin_list/plugin_run/plugin_reload, each step re-entering the secured dispatch path. A manifest tool section (v1.4.0) registers the plugin as a first-class tools/list entry with a generated inputSchema, dispatched through the same secured plugin_run pipeline.

• Screen Recording & Live Capture- screen_record captures a frame every interval_ms for up to duration_ms into a fresh rec-<ulid> dir plus a manifest.json (hard caps: 600 frames, 512 MiB); screen_stream (v1.4.0) runs a continuous start/status/latest/ stop rolling-window capture under stream-<ulid> (≤1800 frames, ≤512 MiB, oldest evicted) with latest returning the newest frame in screenshot's image shape - since/wait_ms turn it into a long-poll (park up to 30 s for a frame newer than the watermark) instead of busy-polling.

• Enterprise Security- uxcp_* API-key auth on HTTP, 10 req/s token bucket, input sanitization, command/path whitelists, AES-256-GCM-encrypted action history, and JSONL audit logging. See SECURITY.md.

Related MCP server: deskwright

Architecture

ultranix-mcp is a single Rust 2024 binary on the tokio runtime, built on rmcp - the official Model Context Protocol Rust SDK - with native stdioand streamable-HTTP (:3010) transports.

The desktop-automation layer is organised as provider traits behind dependency injection(the pattern proven in ultrawin's src/traits.rs): every capability is an Option<Arc<dyn Trait>>, so missing compositor features, absent portals, or headless CI degrade gracefully instead of failing hard. Mock providers implement the same traits, which keeps the full tool surface testable without a Wayland session.

1. wlroots-native- in-process Wayland protocols (Hyprland, sway, Wayfire, river, and most unknown wlroots compositors; no root)

2. grim/slurp (capture) + uinput/evdev (input)- whitelisted helper binaries and kernel-level input for non-wlroots sessions

3. XDG Desktop Portal- Screenshot and RemoteDesktop over zbus (universal fallback, subject to portal consent; the only route on KDE/GNOME Wayland, which implement neither wlr-screencopy nor the wlr virtual-input protocols)

Window management rides compositor IPC where it exists: hyprctl on Hyprland, sway's i3-flavoured IPC ($SWAYSOCK, shipped at v1.2.0) on sway, Wayfire's ipc/ipc-rules plugins ($WAYFIRE_SOCKET, v1.4.0) on Wayfire, riverctl + zwlr_foreign_toplevel_manager_v1 on river (riverctl drives focused-view close and relative-delta move/resize; foreign-toplevel enumerates windows and addresses them as wlr-toplevel-N for focus/close/min/max/fullscreen), and wlr-toplevel as the shared wlroots fallback rung - the sole window rung on unknown wlroots sessions (niri, labwc, ...), kdotool (KWin scripting - Wayland and X11 alike) on KDE, the "Window Calls" Shell extension over D-Bus on GNOME (v1.4.0, extension required - org.gnome.Shell.Eval is deliberately unused), and wmctrl on other X11 sessions.

On X11 sessions the X11-native rungs shipped at v1.1.0 resolve instead: scrot capture, xdotool input (both still ahead of portal/uinput), and wmctrl window management on non-Hyprland X11.

graph TB
    subgraph "Client Layer"
        AI[AI Assistant / Agent]
        MCP[MCP Client]
    end

    subgraph "Transport Layer"
        STDIO[STDIO<br/>never requires auth]
        HTTP[Streamable HTTP :3010<br/>uxcp_* API key]
    end

    subgraph "Security Layer"
        AUTH[API Key Auth]
        RATE[Rate Limiter<br/>10 req/s token bucket]
        SAN[Input Sanitization<br/>command + path whitelists]
    end

    subgraph "Core (rmcp + tokio)"
        SERVER[ultranix-mcp server]
        TOOLS[40 tools - 6 categories]
    end

    subgraph "Providers - Option&lt;Arc&lt;dyn Trait&gt;&gt;"
        CAP[CaptureProvider]
        INP[InputProvider]
        UIA[UIAutomationProvider]
        WIN[WindowProvider]
        VIS[VisionProvider]
        BRW[BrowserProvider]
        CLIP[ClipboardProvider]
    end

    subgraph "Backends - priority order"
        WLR[wlroots-native<br/>screencopy - virtual-pointer - virtual-keyboard]
        UIN[uinput / evdev]
        PORTAL[XDG Desktop Portal<br/>zbus]
        HYPR[hyprctl IPC]
        SWAY[sway IPC - SWAYSOCK]
        WF[Wayfire IPC - WAYFIRE_SOCKET]
        RIV[riverctl + foreign-toplevel - river composite]
        WTOP[wlr-foreign-toplevel - shared wlroots rung]
        GS[gnome-shell - Window Calls ext]
        KDOT[kdotool - KDE]
        ATSPI[AT-SPI2 bus]
        ORT[ONNX Runtime - ort]
        CDP[CDP 127.0.0.1:9222]
        WLC[wl-clipboard - xclip/xsel]
    end

    AI --> MCP
    MCP --> STDIO
    MCP --> HTTP
    STDIO --> SERVER
    HTTP --> AUTH --> RATE --> SAN --> SERVER
    SERVER --> TOOLS
    TOOLS --> CAP
    TOOLS --> INP
    TOOLS --> UIA
    TOOLS --> WIN
    TOOLS --> VIS
    TOOLS --> BRW
    TOOLS --> CLIP
    CAP --> WLR
    INP --> WLR --> UIN --> PORTAL
    WIN --> HYPR
    WIN --> SWAY
    WIN --> WF
    WIN --> RIV
    WIN --> GS
    WIN --> KDOT
    UIA --> ATSPI
    VIS --> ORT
    BRW --> CDP
    CLIP --> WLC

**Token efficiency.**Tool definitions cost context window. ultranix-mcp supports --category= filtering so you expose only the tools you need:

# Serve only mouse + keyboard tools
# (`--stdio` is an alias for `--transport stdio`)
ultranix-mcp --transport stdio --category=mouse,keyboard

Categories: mouse, keyboard, vision (capture/OCR/UI-tree/recording), automation (misc), admin (window/history/metrics/plugins), clipboard. Default: all.

Why ultranix-mcp?

Installation

Option 1: AUR (packaging shipped, submission pending)

Arch-family PKGBUILDs (ultranix-mcp, ultranix-mcp-git, and the prebuilt-binary ultranix-mcp-bin) ship in packaging/ - AUR submission is tracked on ROADMAP.md. A cargo install ultranix-mcp path is supported once the crate is published; see docs/PACKAGING.md §2 for the build-time ort network-fetch caveat.

Option 1b: OCI image (v1.4.0)

ghcr.io/jxoesneon/ultranix-mcp is published on every v* tag by .github/workflows/oci.yml - intended for headless/CI use (bind-mount $XDG_RUNTIME_DIR, the session bus, and /dev/uinput to reach real providers; see docs/HEADLESS.md).

Option 2: Build from source

Prerequisites:

• Linux with a Wayland session - verified on CachyOS (Arch) + Hyprland

• Rust 1.98+ (2024 edition; verified on 1.98.1)

• Session tools used at runtime: hyprctl, grim, slurp

• Optional: xdg-desktop-portal-hyprland (portal fallback path), an AT-SPI2 accessibility bus (UI inspection), Chromium/Chrome with --remote-debugging-port=9222 (browser tools), wl-clipboard (wl-copy/wl-paste - clipboard tools on Wayland), xclip + xsel (clipboard tools on X11/XWayland), kdotool (window tools on KDE), riverctl (river focused-view geometry; enumeration and per-window control come from wlroots foreign-toplevel when advertised), the GNOME "Window Calls" Shell extension (window tools on GNOME)

Steps:

1. Clone the repository:

   git clone https://github.com/jxoesneon/ultranix-mcp.git
   cd ultranix-mcp

2. Build the project:

   cargo build --release

3. Start the server:

   # Stdio transport (recommended for local MCP clients)
   # (`--stdio` is accepted as an alias for `--transport stdio`)
   ./target/release/ultranix-mcp --transport stdio
   
   # Streamable HTTP transport on :3010 (requires ULTRANIX_MCP_API_KEY)
   ./target/release/ultranix-mcp --transport http --bind 127.0.0.1:3010
   
   # Filter to a subset of tool categories (reduce context overhead)
   ./target/release/ultranix-mcp --transport stdio --category=mouse,keyboard

4. Run tests(mock providers - no Wayland session required):

   cargo test

**Cargo features (v1.2.0).**Every backend group is a feature, all on by default so cargo install is unchanged:

Option 3: Nix flake (unverified)

A flake.nix ships at the repo root: nix build produces the package, nix develop enters a devShell with the Rust toolchain and native deps (pipewire, libxkbcommon, libclang for bindgen, session helper binaries), and nix run launches the server. **Note:**the flake was written by review and has not been evaluated in our toolchain - treat it as unverified; fixes and confirmations welcome.

MCP client configuration

Point your MCP client at the binary over stdio. Example for Claude Desktop / Cursor (claude_desktop_config.json / mcp.json):

{
  "mcpServers": {
    "ultranix": {
      "command": "ultranix-mcp",
      "args": ["--transport=stdio"]
    }
  }
}

(--stdio is accepted as a shorthand alias for --transport stdio.)

Token-efficient variant - expose only the mouse, keyboard, and vision categories:

{
  "mcpServers": {
    "ultranix": {
      "command": "ultranix-mcp",
      "args": ["--transport=stdio", "--category=mouse,keyboard,vision"]
    }
  }
}

Configuration

ultranix-mcp works out of the box over stdio (which never requires authentication). For the HTTP transport and production environments, the following variables are supported:

**Data directory.**Runtime state lives under ~/.ultranix-mcp/:

Session Requirements & Security

Wayland automation replaces macOS-style permission prompts with compositor capabilities. ultranix-mcp selects the least-privileged backend that works:

1. wlroots-native (Hyprland)- zwlr_virtual_pointer_v1, virtual-keyboard-unstable-v1, and wlr-screencopy-unstable-v1 are exposed to regular clients. No root, no udev rules, no consent dialogs.

2. uinput/evdev- requires write access to /dev/uinput via the packaged udev rule (GROUP="ultranix-input" - a dedicated group holding only the service user; never the broad input group, which grants keylogger-level read access to every evdev node). Setup is opt-in and documented in packaging/README-uinput.md.

3. XDG Desktop Portal- Screenshot/RemoteDesktop via zbus; the portal mediates a per-app consent dialog through xdg-desktop-portal-hyprland.

**Security Note:**ultranix-mcp ships with built-in safeguards against injection attacks - an arg-constrained, absolute-path-pinned command whitelist (grim, slurp, scrot, hyprctl without dispatch exec/exec-once; xdotool/wmctrl on X11 sessions only), a path whitelist ($XDG_RUNTIME_DIR, /tmp, ~/.ultranix-mcp/**), strict input validation, rate limiting, fail-closed API-key auth on HTTP, a consent gate on destructive tools (-32015 ConsentRequired -> retry with consent_token; --allow-destructive bypass), and encrypted action history. Read the full SECURITY.md and threat model.

Tool Reference

Summary mirror - docs/TOOLS.md is the canonical tool catalog (full schemas, per-tool errors, and consent semantics).

Mouse (--category=mouse)

mouse_click, mouse_double_click, mouse_move, mouse_get_position, mouse_scroll, mouse_drag, mouse_button_control

Keyboard (--category=keyboard)

type_text, key_control

Vision (--category=vision)

screenshot, screen_info, screen_highlight, color_at, set_spatial_focus, get_ui_tree, get_focused_element, find_element, find_text_on_screen, find_icon, wait_for_ui_element, invoke_element, screen_record, screen_stream

Automation (--category=automation)

sleep, mouse_move_path, system_command, web_query

Admin (--category=admin)

window_control, get_windows, get_active_window, metrics, get_action_history, replay_action, clear_action_history, plugin_list, plugin_run, plugin_reload

Clipboard (--category=clipboard)

clipboard_get, clipboard_set, clipboard_clear

Roadmap

See ROADMAP.md for the delivery plan - all of Phases 0-5 (scaffold -> Hyprland I/O -> AT-SPI2 -> vision/CDP -> enterprise -> portability/packaging) shipped as of v1.0.0, the v1.1.0 wave added the layer-shell screen_highlight overlay, X11-native providers, PipeWire portal capture, opt-in Sentry, the OCR result cache, and four more Prometheus metrics, and the v1.2.0 wave added clipboard tools, plugin tool-macros, bounded screen_record, sway/Wayfire/river/KDE/GNOME session detection with a sway window provider, per-backend cargo features, and the framed v2 action-history format, and the v1.3.0 wave added the runtime policy layer (TOML roles, per-key scoping, --readonly and tool allow/deny flags), per-backend invocation metrics, and optional HMAC-signed audit records, and the v1.4.0 wave closed the window-provider coverage (Wayfire IPC, river riverctl, GNOME Window Calls), shipped live screen_stream rolling capture and plugin-exposed dynamic tools, and added the OCI image and ultranix-mcp-bin distribution artifacts - see CHANGELOG.md for release notes.

Documentation

Design and governance documents live in docs/, including architecture decision records under docs/adr/. Start with CONTRIBUTING.md for the development workflow.

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for the mock-provider testing pattern, the ADR process, and pull-request conventions, and CODE_OF_CONDUCT.md for community expectations.

License

This project is licensed under the ISC License.