Identify the active display protocol (Wayland/X11) and desktop environment from environment variables to understand the GUI session the server runs under.
Enables AI assistants to automate Wayland desktop environments through screenshot analysis, mouse control, and keyboard input simulation. It supports visual context via VLM providers like Gemini and OpenRouter to perform complex, multi-step desktop actions.
Type text into the focused VRoid Studio form field after clicking it, with optional clear-first to replace existing content, ensuring correct entry before proceeding.
Clicks the mouse at specified screen coordinates, using a closed-loop convergence to ensure accurate cursor placement before clicking. Supports relative deltas and optional glide movement.
List whitelisted system tools that Windy Word can install on this machine to expand capabilities. Returns platform check, distro detection, and install commands per tool.
Identify the desktop environment and input backends before automating. Call this tool to detect Wayland/X11, active window, and safety warnings without capturing screen or sending input.
Scroll vertically in small increments. Provide a positive step count to scroll up or negative to scroll down, enabling precise navigation in Linux desktop applications.
Install missing system tools like wtype or xdotool from the whitelist using pkexec. A graphical sudo prompt appears; use dryRun=true to preview the command.
Retrieves the current pointer coordinates, preferring the compositor's answer on Wayland. When unsupported, it returns the server's last known position and indicates the source, avoiding stale X11 data.
Record screen or window to H.264 MP4 to capture animations, transitions, scrolls, stutters, and hover states that stills cannot show. Uses Wayland screen cast without consent dialog.
Move the mouse cursor to an exact screen position using absolute coordinates. Coordinates match window and screen maps, enabling precise targeting on GNOME Wayland desktops.
Place text or a file's bytes on the GNOME Wayland clipboard and verify the write by reading back the data, ensuring the compositor holds the correct content.
List all 12 paste strategies with per-machine availability, default fallback chain, and hotkey collision detection for macOS, Windows, Linux X11, and Wayland.
Capture the Raspberry Pi Wayland desktop or a selected region, with optional downscaling, and return a view_id that maps mouse actions to image-pixel coordinates.