hypruse is an MCP server that gives AI agents native computer-use control on Hyprland through IPC, accessibility trees, screenshots, and virtual input.
Semantic desktop snapshots and window/workspace operations (desktop, hypr, launch).
Read and click accessible controls by name with exact coordinates and current values (ui, marks, click_ui).
Capture screenshots and zoom into regions for coordinate-mapped visual targeting (screenshot, zoom).
Control the mouse and keyboard, including typing unicode text and app shortcuts (pointer, keyboard).
Discover and run the user's own Hyprland keybinds and workflows (binds, use_bind).
Wait on real compositor events instead of polling (wait_for).
Run multi-step micro-sequences in one call with optional post-action observation (sequence, then).
Use shell verbs and an installable agent skill outside MCP.
Work in background mode on hidden workspaces without taking the user's cursor, focus, or workspace.
Apply safety layers: read-only, confinement, auth guard, strict seat checks, marking, journaling, dry-run, and replay.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@hypruse Take a screenshot of my focused window."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Computer use for Hyprland. An MCP server that gives AI agents native hands on your Wayland desktop: workspaces, windows, mouse, keyboard, screenshots.
No ydotool daemon. No root. No portals. No X11.
Computer use exists on macOS and Windows, and the vendors' Linux apps still ship without it: Claude Desktop's Linux docs say computer use is not available there, and ChatGPT's computer use covers macOS and Windows only. Anthropic's reference implementation is an X11 container. Other Wayland tools lean on ydotool and uinput, on GNOME-only portals, or on a compositor plugin built for one exact Hyprland version.
Meanwhile Hyprland already exposes everything an agent needs, better than any accessibility bridge: a complete IPC surface for state and window management, and first-class Wayland protocols for input. hypruse just wires them to MCP:
Semantic first. desktop returns the real window/workspace tree (addresses, classes, titles, geometry) in one call. The agent switches workspaces and focuses windows the way you do (instantly, over IPC), not by squinting at pixels.
Vision when it matters. Screenshots of a monitor, one window (with grim 1.5+ on Hyprland 0.54+, captured as itself even on a hidden workspace or under another window, though a hidden app may show how it last looked), or a zoomed region, with the geometry/scale metadata to map any pixel back to a clickable coordinate: a coarse-to-fine loop grounded in the GUI-agents research.
Native input. Clicks and scrolls are spoken directly over the Wayland wire (zwlr_virtual_pointer_v1); typing goes through wtype's virtual keyboard with a proper XKB keymap, unicode-safe on any layout.
Works beside you. The agent can work in apps on its own workspaces while you keep working on yours. It presses controls through the accessibility tree and types through it or with Hyprland's sendshortcut, so your cursor, keyboard focus and workspace stay where you left them. Stock Hyprland, no compositor plugin: see Background mode.
Related MCP server: kwin-mcp
agent (Claude Code, or any MCP client)
│ stdio
▼
hypruse
├── hyprctl -j ········▶ desktop state: monitors, workspaces, windows, layers
├── hyprctl dispatch ··▶ focus / move / close / launch / movecursor,
│ and sendshortcut for keys to a hidden window
├── grim ··············▶ screenshots: monitor, window, region
├── D-Bus (AT-SPI) ····▶ accessibility tree: named controls, current
│ values, exact coords (ui / marks / click_ui),
│ and presses and text in the background
├── wtype ·············▶ keyboard (zwp_virtual_keyboard_v1, real XKB keymap)
└── raw Wayland wire ··▶ click & scroll (zwlr_virtual_pointer_v1)Optional binaries gate two more tools: imagemagick draws the numbered
overlay for marks, and wl-clipboard backs the opt-in clipboard tool.
Design decisions:
No ydotool / uinput. That path needs a daemon, udev rules or root, and types US scancodes that break on other layouts. hypruse is just another Wayland client of your compositor, same standing as wlrctl.
No portals. xdg-desktop-portal-hyprland does not implement the RemoteDesktop portal (InputCapture is capture, not injection), so anything built on libei/portals silently degrades on Hyprland. hypruse doesn't try.
Cursor positioning through the compositor's own cursor dispatcher (global logical coordinates, exact on any monitor layout), with only button/axis events on the virtual pointer, sidestepping the known multi-monitor bugs of absolute virtual-pointer motion (hyprwm/Hyprland#6749).
tool | what it does |
| One-call semantic snapshot: monitors, workspaces, windows (address/class/title/geometry), active window, cursor, and layer surfaces (launchers, bars, notification popups) with a best-effort kind and geometry. A listed layer is one the compositor tracks, not one you can see: transparent and dormant surfaces are reported too, so screenshot when visibility matters |
| Focused monitor, one window by address (with grim 1.5+ on Hyprland 0.54+, even hidden or covered; metadata |
| Native-resolution re-capture around an estimated point (optionally clamped to a window): the precision step before clicking small controls, same metadata contract |
| Read a window's accessibility tree (AT-SPI, GTK/Qt apps that expose one) and return clickable elements by name with exact global coordinates, no screenshot; reports current values too (typed text, slider position, checkbox state); falls back to vision when an app exposes nothing |
| Set-of-Marks capture: the window screenshot with every accessible control drawn as a numbered mark, plus a JSON legend (role, name, current value, exact click point per number); needs ImageMagick for the drawing, degrades to the legend alone without it |
| Click a control by accessible NAME or by a |
| move / click / drag / scroll (discrete wheel notches) in global coordinates. Your real cursor, so by default it is refused where the input would land on a window other than yours |
| Type literal text (unicode-safe) or press app-level combos ( |
| Switch workspace, focus/move/close windows, fullscreen, floating (pure IPC, milliseconds; |
| Start an app, silently on the next of the agent's own workspaces unless you name one (or pass |
| The user's own keybinds, decoded ( |
| Execute a keybind by combo ( |
| Run an ordered list of actions (pointer/keyboard/click_ui/hypr/wait_for) in one call; stops the moment the desktop changes in a way the current step did not expect, so a click/type/enter micro-sequence costs one round-trip instead of several |
| Block on real compositor events (window open/close, workspace change, title change, layer surfaces appearing/closing, urgency, screen sharing on/off) with a match filter and timeout; a filtered wait whose condition already holds (window already gone, workspace already active, layer already mapped) answers instantly with |
| Read or write the text clipboard via |
The acting tools (pointer, keyboard, click_ui, hypr, use_bind, sequence) take an optional then argument that appends the result to the same call, so the agent sees the effect without a second round-trip: then='desktop' adds a fresh semantic snapshot (~20 ms, cheap, best for window/focus changes), then='screenshot' a stable capture (best for visual changes), then='ui' the acted-on window's controls with their current values (a few hundred exact tokens, best after typing or toggling; click_ui reads the window it clicked even if the click handed focus to a dialog, the others read the focused window), then='none' nothing (the default everywhere except sequence, which defaults to 'desktop').
Every tool is also a shell verb (hypruse desktop, hypruse click_ui Save --window 0x...) for agents that run commands instead of MCP, with a skill that teaches them: see Shell verbs and the agent skill.
The tools group into five capabilities, ordered most-reliable-and-cheapest first. An agent that reaches for them in this order is both faster and more accurate, and many tasks never need a screenshot at all.
desktop returns the entire window and workspace tree in one call: every window's address, class, title, and geometry, the active window, the cursor, and any layer surfaces the compositor is tracking (launchers, bars, notification popups). hypr and launch then act on it over IPC in milliseconds: switch workspace, focus/move/close/fullscreen/float a window by address, or start an app.
Use it well: never take a screenshot to find or arrange windows. Read desktop, act on the address you want. launch blocks on the real openwindow event and hands back the new window's address, so there is nothing to poll or guess; it also relocates single-instance apps (browsers) that ignore workspace rules.
When an app exposes an accessibility tree (most GTK and Qt apps), you can target controls by name instead of by pixel. ui lists every control with its exact global coordinate, and reports the current value of the controls that carry one: the text in a field, a slider's percentage, a checkbox's state. click_ui resolves a name and clicks it in one call, through the real cursor (so the beacon and every safety guarantee still apply), or, in a window you are not in, through the control's own accessibility action without touching your cursor. marks draws numbered marks over a screenshot with a legend, for when you want to see the options first and then click_ui(mark=N).
Use it well: reach for click_ui name="Save" before estimating any pixel, since it is exact and spends no image. Read a form's state with ui (did the box actually tick?) instead of screenshotting it. An ambiguous name returns the candidates rather than guessing. When an app exposes no tree (terminals, canvas apps, Electron/Chrome without --force-renderer-accessibility) the tool says so, and you fall back to vision.
For everything the accessibility tree cannot name, screenshot (monitor, window, or region) and zoom (a native-resolution re-capture around a point) carry a strict coordinate contract, global = geometry + pixel / scale, that stays exact on every monitor and fractional scale.
Use it well: don't guess a small control from a full-screen image. Work coarse-to-fine: screenshot the window, estimate the target, zoom there, re-estimate on the sharp crop, then click. This two-step loop is the research-backed way to hit small targets.
For an agent the model calls dominate task latency, not the desktop, so the real speedups are structural. sequence runs an ordered micro-plan (click, type, press enter, wait) in a single call, stopping the moment the desktop changes structurally in a way a step did not intend (a window opening, closing, or moving, an unexpected workspace switch, or a seat-taking launcher or on-screen keyboard; it deliberately ignores bare focus changes and notification popups). then='desktop' | 'screenshot' | 'ui' fuses a fresh view of the result into the acting call itself. wait_for blocks on real compositor events (a window or launcher opening, a title changing, a workspace switch, an urgency hint, screen-sharing starting) instead of sleeping and hoping.
Use it well: collapse a known click/type/enter flow into one sequence. After typing into a form, add then='ui' to read the effect back in a few hundred exact tokens instead of a screenshot. After a launch or a shortcut that opens something, wait_for the event rather than sleeping.
hypruse hands an agent your real seat, so it ships the controls to bound what that agent can do. Beyond the always-on approval prompts and the Waybar activity beacon, opt-in env flags narrow what an agent can touch: HYPRUSE_READONLY exposes only the observation tools; HYPRUSE_CONFINE restricts input to the windows the agent launched, or a class/workspace allowlist; HYPRUSE_AUTH_GUARD (on by default) refuses to drive authentication dialogs; HYPRUSE_STRICT refuses to act if you took the seat back; HYPRUSE_MARK tags agent-owned windows and announces when the agent opens a window or captures the screen. Two more record rather than restrict: HYPRUSE_JOURNAL writes an auditable NDJSON line per tool call, refusals included, and HYPRUSE_DRYRUN runs every check and delivers nothing, so you can watch an agent plan the work before it touches your desktop.
Use it well: run read-only for the first week. When you trust a workflow, allowlist its tools and, if you want to walk away, confine the agent to a scope so it cannot click or type into your password manager on another workspace. Keep a panic bind handy (hypruse stop, or pkill -f hypruse). The Security model has the full story.
An agent can work in apps on workspaces you are not looking at while you keep working on your own. On stock Hyprland, with no compositor plugin, hypruse does not move your cursor, take your keyboard focus or switch your workspace to do it. An app can still bring one of its own windows forward in response to something the agent did; when that changes your focused window, the result says so.
It is on by default, in automatic mode. An action on the window you are in runs on your screen, as it always has. An action on any other window runs in the background:
click_ui presses the control through its accessibility action. A text field gets the app's own focus instead (except in GTK 4, which does not allow it). When no button, box or field matches the name, click_ui also takes a named element of another role that has an action, such as a row in a web app's list. Some of those actions only focus the element (Qt WebEngine's rows offer nothing else). The result then says it focused the element and whether the app reports it focused. That does not open the row: in testing, Enter sent to a focused Qt WebEngine row did not open it either.
keyboard with window= types into the window's focused text field (or its only one) through the accessibility interface. Otherwise it sends the keys to that window alone with Hyprland's sendshortcut, one key at a time on your keyboard layout. Newlines, tabs and backspaces are always sent as keys. Keys go to the control the app itself has focused, which is not necessarily the field text went into, so the agent focuses a field with click_ui before sending keys to it.
launch with no workspace opens the app on one of the agent's own workspaces, out of your sight.
The pointer is your real cursor, so it cannot work in the background. It is refused where its input would land on a window other than yours, and the refusal points the agent at click_ui.
What the background cannot do is refused with the reason and what to do instead. background=false on a call (--foreground on the CLI) does it on your screen.
The guarantee overrides everything. Without it, a call's own background argument wins over the automatic rule.
setting | what it does |
| The guarantee. The agent works only on windows on its own workspaces. The pointer, workspace switches, |
| Per call, on |
unset | Automatic, as above. |
| Off: actions run on your screen as in 0.12, unless a call passes |
The guarantee bounds hypruse's own input. It cannot bound what a program does once launch starts it: the command runs as given, through a shell, so keep launch behind your client's approval prompt even with HYPRUSE_BACKGROUND=1.
The agent's workspaces are HYPRUSE_AGENT_WORKSPACES, 99 down to 90 by default, at the far end of the numbers so they stay clear of the ones you use. A-B counts in either direction, and a comma list (99,98,50) or named workspaces (name:research) work too. HYPRUSE_AGENT_FILL (default 1) is how many of the agent's apps share one workspace before the next one is used. With 1, every app gets a workspace to itself, which keeps hidden windows at their full size. A workspace in the range that holds one of your windows is skipped, so yours and the agent's never mix, and so is one you are looking at. When the whole range is full the next app goes where the agent has the fewest. A window an earlier session opened counts as the agent's when HYPRUSE_MARK tagged it. With HYPRUSE_CONFINE=workspace:..., apps open only on agent workspaces inside that scope, and on yours as before when there is none.
launch waits for its own window: one that opens on the workspace it asked for, or one of the launched program's own class. A window of another app that you open meanwhile is left alone; one of the same app can be taken for the launched one. An app that is already running (a browser, often) opens its new window from the running process, which ignores the workspace rule, so it appears on your screen and takes your focus until hypruse moves it; the result says so, and under the guarantee it carries a warning. Start a separate instance for the agent (for a browser, a separate profile) to avoid it.
Every background result says how it was done, whether your focused window and workspace were the same before and after, and, for a control that carries a value, what the accessibility tree read back:
pressed check box 'Agree' through its accessibility action (no pointer, the human's focus untouched) in yad; now: checked=true
typed 12 characters into the text field in yad through the accessibility interface (the human's focus untouched); the field now holds the typed text (12 characters)
sent ctrl+a to 0x5648592451c0 (yad) with sendshortcut, to the control the app has focused (the human's focus untouched)A press on a check box or slider that the app ignored reads no change seen. A button press, a focused field or keys sent with sendshortcut carry no value to read back, so check their effect with ui. If your focused window changed during the action (because the app brought a window forward, or because you switched), the result ends with a WARNING naming the window that has focus now. Typed results never repeat the text, since the journal keeps every result line and typed text is only ever recorded as a length plus a digest. Screenshots are not used to confirm anything, because many apps stop drawing while hidden and a hidden window's pixels can be stale; ui shows its current state.
Only controls the accessibility tree exposes can be pressed in the background, and only with their primary action: a right or double click is refused there. Clicking an arbitrary point in a hidden window would need a compositor plugin or an input seat of its own, which this design avoids. Apps without a tree (terminals, canvas apps, Electron and Chrome without --force-renderer-accessibility) can still take keys.
The accessibility tree tells an app's windows apart by title or size. When neither tells the target from the app's other windows (a second window of a single-instance app with the same title and size, say), or the app's accessibility connection belongs to another process than its window, hypruse cannot be sure which window a control belongs to: click_ui refuses, keyboard sends keys, which Hyprland aims at the window itself, and ui says its controls may be another window's.
A hidden window's ui points are where its controls sit inside it, not on your screen, and the screen there shows your own windows. ui ends with a note saying so, because a pointer click at those points would land on whatever you are looking at.
A GTK 3 app brings its window forward when a field takes focus over the accessibility interface. With Hyprland's misc:focus_on_activate on, that would switch you to it, so focusing a GTK 3 field is refused then. A per-window focus_on_activate rule cannot be seen; the after-check reports it.
A character your keyboard layout has no key for (an emoji or CJK on most layouts) can only go in through the accessibility interface. A hidden GTK window does not report which of several text fields has its focus, so in a multi-field form such text is refused before any key is sent, and so is typing into a form with a password field among several, unless allow_auth=true.
For each key, sendshortcut moves keyboard focus to the target and back inside one compositor step, so your own typing cannot mix in. Your app does see its focus leave and come back, which most apps ignore; a few close an open menu.
sendshortcut is refused for XWayland windows, because Hyprland delivers the key to the focused window instead when both are XWayland.
With several keyboard layouts, sendshortcut works only while you are on the first one. Hyprland finds each key on the layout in use but the app reads it on the first, so on another layout different characters would arrive, and hypruse refuses instead. Text through the accessibility interface does not depend on the layout.
Web pages in the background need the browser bridge, which comes in a later release.
While the session is locked, background actions are refused like all other input.
Every trust layer applies to the background paths: confinement checks the target window, the auth guard refuses a background press or key in an authentication dialog, and typing into a password field the accessibility tree shows needs allow_auth=true (an app with no tree cannot be checked, as in the foreground). You cannot see background work, so for anything unattended set HYPRUSE_BACKGROUND=1: input stays on the agent's workspaces, hypruse's own input never takes your screen (an app the agent launches can, and the result warns), the journal keeps every result with its method, and HYPRUSE_MARK tells you when the agent opens a window.
Not every agent speaks MCP, and the ones that do increasingly prefer a command line: a tool list costs a few thousand tokens per session before the first call, a shell verb costs nothing until it runs. So the same fifteen tools are verbs, going through the same functions the MCP server registers, with the same trust guards, journal and activity beacon:
hypruse desktop # one line per monitor, workspace, window, layer
hypruse launch --workspace 2 -- firefox --new-window https://example.org
hypruse ui --window 0x55a4479a1200 --name Save # [0] push button "Save" @1204,88
hypruse click_ui Save --window 0x55a4479a1200 --then ui
hypruse screenshot --window 0x55a4479a1200 # a file path, then {"geometry":[x,y,w,h],"scale":1.0,...}
hypruse zoom 1180 840 --window 0x55a4479a1200
hypruse keyboard type "hello" --window 0x55a4479a1200
hypruse wait_for title_change --match Inbox --timeout 10
hypruse sequence '[{"op":"click_ui","name":"Search","window":"0x55a4479a1200"},{"op":"keyboard","action":"type","text":"btop","window":"0x55a4479a1200"},{"op":"keyboard","action":"key","keys":"enter","window":"0x55a4479a1200"}]'The contract is built for a program reading the output: the verbs are the tool names, a tool's action is a positional sub-verb (pointer click 800 60, hypr workspace 3, clipboard read), output is compact plain text (one line per fact, --json for the raw result as one line), a capture prints its file path and coordinate metadata rather than bytes, and errors are one line on stderr with a meaningful exit code: 0 delivered, 1 error, 2 usage, 3 refused by a trust layer (or by read-only mode), 4 ran but found nothing to act on (a wait_for timeout, no accessibility tree, an ambiguous name). hypruse --help lists every verb and hypruse <verb> --help its flags; --dry-run rehearses an acting verb. A verb starts in a few hundred milliseconds: the MCP stack is never imported on that path.
Between one-shot processes hypruse keeps the little state a verb would otherwise lose in $XDG_RUNTIME_DIR/hypruse/cli-state.json, keyed by the compositor instance: the marks numbering (so click_ui --mark N works), the launched confinement set, and the HYPRUSE_STRICT seat baseline, which is the one that would otherwise fail open.
The skill that teaches an agent the verbs, the desktop-first workflow and the safety rules ships inside the package. Install it into the skill directories of the agents on your machine (Claude Code, Codex, Pi, Hermes, OpenClaw, OpenCode, Gemini, Antigravity, Cursor, Copilot), or via the skills CLI:
hypruse skill install # ~/.agents/skills/hypruse, linked into each agent that is installed
hypruse skill install --agent codex # one agent, created if absent
npx skills add IlyasKhallouki/hypruse -ghypruse init offers the same install. The skill pre-approves only the observation verbs, doctor, --help and reading the capture files; acting verbs stay behind your agent's own approval prompt, which is the boundary the Security model leans on.
Requirements: Hyprland (both config managers: hyprland.conf and the Lua hyprland.lua that 0.56 introduced), grim (grim 1.5+ on Hyprland 0.54+ lets a window screenshot capture the window itself, and hypruse doctor says whether yours can), wtype (most Hyprland setups already have both), and uv. The accessibility tools (ui/marks/click_ui) use busctl, which ships with systemd, to find the accessibility bus, then talk to it over one connection of their own. Optional: wl-clipboard for the opt-in clipboard tool, imagemagick for numbered marks captures.
Arch Linux, from the AUR:
yay -S hypruse # or hypruse-git for mainThen let it set itself up and verify the environment:
hypruse init # detects your MCP clients, registers (asks first), offers the agent skill, runs doctor
hypruse doctor # just the diagnosticsManual registration, Claude Code:
claude mcp add -s user hypruse -- uvx hypruseFrom a source checkout:
claude mcp add -s user hypruse -- uv run --directory /path/to/hypruse hypruseAny other MCP client: run uvx hypruse as a stdio server. hypruse is also in
the official MCP registry as
io.github.IlyasKhallouki/hypruse, so clients that browse the registry can
install it from there.
Read-only mode: set HYPRUSE_READONLY=1 in the server config to expose only the observation tools (desktop, screenshot, zoom, ui, marks, binds, wait_for). The agent can see and narrate but cannot click, type, or launch. A good first week.
The Linux beta ships without Anthropic's first-party computer use, but stdio MCP servers work in chat, which makes hypruse the workaround. In ~/.config/Claude/claude_desktop_config.json:
{
"mcpServers": {
"hypruse": {
"command": "uvx",
"args": ["hypruse"],
"env": { "HYPRUSE_SCREENSHOT_MODE": "image" }
}
}
}Two Desktop-specific notes: use image mode (Desktop renders inline MCP images and has no file-read tool), and the app must run natively inside your Hyprland session so the server inherits WAYLAND_DISPLAY/HYPRLAND_INSTANCE_SIGNATURE; from a VM or container it cannot reach your compositor. If your Desktop install bypasses tool-approval prompts, treat the Waybar indicator + panic keybind as mandatory, not optional.
Read this section before installing. hypruse hands an agent your mouse, your keyboard, your screen contents, and an app launcher. The layers that keep that sane:
Approval: MCP clients gate tool calls. In Claude Code, allowlist the read-only tools (desktop, screenshot) and leave pointer/keyboard/hypr/launch on ask-first until you trust a workflow.
Visibility: the server maintains an activity beacon ($XDG_RUNTIME_DIR/hypruse/state.json); the shipped Waybar module is invisible when idle and shows a robot indicator while an agent has hands on your desktop.
Interruption: click the indicator, or bind a panic key. The portable form works for every install: bind = SUPER SHIFT, BackSpace, exec, pkill -f hypruse. If hypruse is on your PATH (the AUR or a pipx install), bind = SUPER SHIFT, BackSpace, exec, hypruse stop is nicer: it signals the server to shut down gracefully, releasing any held pointer button and clearing the beacon. For a uvx install use exec, uvx hypruse stop; for a source checkout, exec, uv run --directory /path/to/hypruse hypruse stop. Killing it mid-action is safe either way: button press/release pairs never span tool calls, and even a long drag's held button is released on the way out.
The seat is shared. There is one cursor and one keyboard focus, and Hyprland's focus-follows-mouse means a cursor move alone can retarget keystrokes. Don't type while an agent is driving your window; watch the indicator. Background mode keeps the agent off your seat for every window you are not in.
Scope: stdio only (no network listener), nothing persisted except the beacon and the capped screenshot cache in $XDG_RUNTIME_DIR (tmpfs, newest 20) and, if you turn it on, the action journal on disk under $XDG_STATE_HOME (rotated, one generation, and text-redacted by default). No clipboard access unless you opt in: HYPRUSE_CLIPBOARD=1 registers a clipboard tool (never in read-only mode); clipboards hold passwords, so leave it off unless a workflow needs it. A screenshot sees everything visible, and a window screenshot also reaches windows on hidden workspaces, in scratchpads and under other windows: treat an agent session like sharing your whole desktop. Read-only mode and HYPRUSE_CONFINE limit what the agent can do, not what it can see. Window screenshots are refused while the session is locked.
What the agent reads is untrusted. Window titles, accessibility names and values, and clipboard text flow verbatim into the agent's context, and any web page, filename, or document can put instructions there (prompt injection). hypruse cannot sanitize meaning, so the approval layer is the backstop: keep consequential tools (launch, keyboard, clipboard) on ask-first when the agent will look at untrusted windows, and treat "the screen told me to" as attacker input when reviewing an approval prompt.
Input never lands where it silently would not work. Three always-on checks (no env flag) refuse or annotate rather than report a phantom success: a click aimed under a launcher or on-screen keyboard layer surface (which sits above windows and would swallow it), typing while a launcher holds the keyboard grab, and any input while the session is locked. A lock screen is an ext-session-lock client, invisible to the window and layer lists, so hypruse asks Hyprland itself (hyprctl locked), which covers hyprlock, swaylock, Omarchy's built-in lock, and a locker that crashed and left the session locked. While locked, pointer and a window-less keyboard refuse unless allow_auth=true says a human wants the agent driving the unlock prompt (never under HYPRUSE_CONFINE); keyboard with window= and click_ui always refuse, since input aimed at a window cannot reach it behind the lock. These are truthfulness aids, not a sandbox: the lock check fails open on an unreadable system state, so they harden the common case without being a boundary you can lean on. (While HYPRUSE_AUTH_GUARD is on, an unreadable layer list makes pointer, click_ui and keyboard refuse, since an authentication prompt cannot be ruled out.)
Seven opt-in env flags. The first five narrow what an agent can touch, and each fails toward less action; the last two record and rehearse rather than restrict. All compose with the layers above:
HYPRUSE_CONFINE restricts input to a scope of windows: launched (only windows hypruse itself opened this session), class:firefox,kitty, or workspace:3,special:notes. Keyboard, click_ui, and hypr window ops are refused outside the scope; a pointer click is refused when any window under the point is out of scope (Hyprland's window list is not z-ordered, so hypruse fails closed rather than guess which window is on top). This is what lets you leave an agent working while your password manager sits on another workspace, out of reach of its clicks and keys (not of its screenshots). use_bind is refused outright while confinement is set, because a keybind runs an arbitrary compositor action that cannot be scoped to a window.
HYPRUSE_BACKGROUND=1 is the background guarantee: input is confined to the windows on the agent's own workspaces (and to HYPRUSE_CONFINE's scope too, when both are set), and any hypruse action that would take your cursor, focus or workspace is refused. It cannot bound what a command started with launch does.
HYPRUSE_AUTH_GUARD (default on) refuses to click or type into a system authentication dialog (polkit agents, whether a window like hyprpolkitagent or a prompt the shell draws as a layer, as Omarchy 4 and end-4's dots do; the GNOME keyring prompt; Hyprland's own dialogs, which are its permission prompt and also its app-not-responding and safe-mode notices, since all three share one window class), so a manipulated agent cannot approve a privilege escalation. click_ui also refuses when such a dialog floats over the control it was asked to click. When the layer list cannot be read, it refuses rather than guess. A prompt a shell opens as an ordinary window of its own class, as DankMaterialShell does, is not recognised. Set HYPRUSE_AUTH_GUARD=strict to also refuse typing into a password field inside an ordinary window (a browser login), detected via the accessibility tree. A per-call allow_auth=true on pointer/keyboard/click_ui overrides it, and because it changes the tool's arguments the override surfaces distinctly in the approval prompt. HYPRUSE_AUTH_GUARD=0 disables it.
HYPRUSE_STRICT refuses to act when the cursor or focused window moved since hypruse's last action (the human, or a popup, took the seat): the agent must re-read desktop/screenshot and retry, so it never types into a window you just switched to.
HYPRUSE_MARK makes the agent's presence legible on the desktop: it tags every window the agent opens hypruse-owned and flashes an on-screen notice when the agent opens a window or captures the screen. It also installs a border_color window rule on that tag so owned windows get a colored outline, but whether a runtime rule renders depends on your Hyprland version and config precedence (on some setups it does not take effect); when it cannot be installed at all, hypruse says so on stderr rather than leaving you with a marking layer that is quietly not running. For a guaranteed outline, add the rule to your own config, which hypruse's tagging then matches: windowrule = border_color rgb(ff5555), tag hypruse-owned in hyprland.conf (older Hyprland: tag:hypruse-owned), or hl.window_rule({ match = { tag = "hypruse-owned" }, border_color = "rgb(ff5555)" }) in hyprland.lua.
HYPRUSE_JOURNAL records what the agent did: one NDJSON line per tool call in $XDG_STATE_HOME/hypruse/journal.ndjson (HYPRUSE_JOURNAL=1), or a path of your own. Read it with hypruse journal, re-run it with hypruse replay. See The record below.
HYPRUSE_DRYRUN is a rehearsal: every argument check and every guard above runs, then the call reports what it would have done and delivers nothing.
The flags above decide what an agent may do in the moment and then forget it happened. HYPRUSE_JOURNAL is the memory. One JSON object per line, so tail -f, grep, and jq all work on a live file:
{"v":1,"seq":7,"ts":"2026-08-02T09:12:13.456Z","kind":"act","tool":"pointer",
"args":{"action":"click","x":800,"y":60},"outcome":"ok","ms":37,"result":"click ok"}kind splits the two questions people actually ask: act is input delivered to your desktop, observe is the agent looking, which is what a privacy audit wants (when the screen was captured, when the clipboard was read). Observation results are never recorded, only that they happened, so the journal never becomes a second copy of everything the agent saw. Refusals are recorded too, with the guard's own message: it is the only place the history of your trust layers doing their job exists.
Typed and copied text is recorded as a length plus a short digest, not as text, because keystrokes are passwords. HYPRUSE_JOURNAL_TEXT=1 keeps it verbatim, which you need only to replay typing. Note what a digest is and is not: it proves two entries typed the same thing and it will not hand a reader your password, but it is an unsalted SHA-256 prefix next to an exact character count, so a four-digit PIN is trivially recovered from it. Treat the journal as sensitive either way. It is written 0600 in a 0700 directory, and rotated at HYPRUSE_JOURNAL_MAX_BYTES (8 MiB, one previous generation kept as .1; set 0 to never rotate).
The journal is a recorder, not a guard: if it cannot be written the action still happens and hypruse warns once on stderr, because failing your desktop over a log line is the wrong trade. What it does not record is a then= observation as its own entry: the acting call that carried it is recorded, then=screenshot and all, but the capture does not get a second line of its own.
Read it back with hypruse journal (--acts for actions only, --refused for what the guards stopped, -n N to tail, -v to include each result):
1 09:12:10 session start pid 4211 0.10.0 confine=launched auth_guard=1 strict=True
14 09:12:13 act pointer action=click x=800 y=60
16 09:12:15 act keyboard action=type text=<11 chars>
19 09:12:19 act hypr action=close_window target=0x5f2a10
REFUSED TrustError: 0x5f2a10 (Signal) is outside the agent's confinement scope
312 actions (0 dry), 604 observations, 1 refused by a trust layer, 0 errorsHYPRUSE_DRYRUN=1 turns the same session into a rehearsal. Every acting tool validates its arguments and runs every trust guard, then reports the plan instead of executing it, so a dry run refuses exactly what a real run would:
DRY RUN, nothing was delivered: would click push button 'Send' at (1204, 880) in signalNothing reaches the desktop: not the click, not the keystroke, not even the window focus that normally precedes typing. Enforced twice, once at each tool and once at the input path itself, so a code path nobody thought of fails loudly rather than quietly acting during a simulation. The agent is told dry run is on, so it reports a plan instead of retrying an action that "did not work". The scope is the agent's actions, not the server's own startup: HYPRUSE_MARK, if you set it, still installs its window rule when the server starts.
hypruse replay <journal> re-issues a journal's actions through the same tool functions, so the same guards apply to the replay. It prints the plan and stops there unless you pass --execute. Before it takes the seat it refuses outright, rather than failing halfway and leaving your desktop part-way through someone else's plan, when: an action was recorded by a newer hypruse, a recorded window no longer exists (--skip-missing runs the rest), typed text was recorded as a digest, the entry is a click_ui(mark=N) whose numbering died with the session that drew it, the entry is a clipboard write and HYPRUSE_CLIPBOARD is not set, or HYPRUSE_READONLY or HYPRUSE_DRYRUN is set. It paces itself from the recorded timing, capped by --max-gap and scaled by --speed, and its own actions are recorded and marked, so replaying the same file again runs the original plan rather than the plan plus the replay of it.
The honest limit is window addresses: they are heap pointers, so yesterday's journal mostly names windows that are gone, and an address can even be reused by a different window later, which no pre-flight can catch. Replay is for re-running a flow on a desktop that still looks like the one recorded.
Dry run and replay compose in the obvious direction: let the agent work with HYPRUSE_DRYRUN=1, read the journal, and replay it with --execute once the plan is one you like.
Measured on a live session (Hyprland 0.55, 1080p, 20 windows): desktop
~20 ms (one batched hyprctl call), workspace/window dispatch ~10-20 ms, full-monitor screenshot ~65 ms
(fast JPEG default; ~800 ms if you ask for lossless PNG, grim's zlib
path dominates), region/zoom captures well under that. If tool
calls feel slow, it is almost certainly the MCP approval prompt in
front of each call, not the server. Allowlist the tools you trust and the
latency disappears. Claude Code (.claude/settings.json):
{
"permissions": {
"allow": [
"mcp__hypruse__desktop",
"mcp__hypruse__screenshot",
"mcp__hypruse__hypr"
// add pointer/keyboard/launch once you trust your workflows
]
}
}Everything speaks Hyprland's global logical coordinates, the space hyprctl cursorpos and window at use. Screenshots are pixel-space; each capture returns geometry and scale so global = origin + pixel / scale. On scale 1.0 monitors (most setups) image pixels are global coordinates.
The zoom tool does the precision arithmetic for the agent: give it an estimated global point and it captures a native-resolution box around it, clamped to the screen (or to a window), with the same metadata contract. That two-step loop, estimate on the full view then re-estimate on the zoom, is the research-backed way to hit small controls.
Captures default to JPEG q90: on a 1080p frame that is roughly 12x faster to encode than PNG (grim's zlib path dominates capture time, measured ~65 ms vs ~800 ms) and 3-4x smaller, while full-res q90 reads UI text well. Pass lossless=true for exact pixels (PNG). In image mode, captures also fit the host's result-size limit (Claude Desktop caps tool results at 1 MB) by degrading quality before resolution, since grim's downscale filter is slower than a full-res capture, and cap the long edge at HYPRUSE_MAX_IMAGE_EDGE pixels (default 1568) so the host never downscales the image under the model. The applied scale is folded into the returned metadata, so coordinate mapping stays exact; tune with HYPRUSE_MAX_IMAGE_BYTES, or pass scale for a deliberate zoom-out.
By default the screenshot tool writes the image under $XDG_RUNTIME_DIR/hypruse/ and returns its path; MCP hosts with a file reader (Claude Code's Read) render it natively. This default exists because some hosts (including Claude Code 2.1.x) serialize inline MCP image blocks to base64 text the model cannot see. HYPRUSE_SCREENSHOT_MODE=image switches to inline image content blocks for hosts that render them correctly.
uv sync --group dev
uv run pytest # unit tests, no compositor needed
uv run pytest -m e2e --override-ini addopts= # live seat-safe checks
uv run pytest -m background --override-ini addopts= # live: drives yad forms on the agent's workspaces
uv run python scripts/e2e_input.py # supervised: takes the seat ~10sThe background suite opens yad forms on the agent's workspaces, drives them in the background, and checks after every step that your active window, workspace and cursor are exactly as they were, so leave the mouse and keyboard alone while it runs. It also checks Hyprland's own events, which hold even while you work: no window it opened ever takes focus, and no agent workspace is switched to.
The input e2e is deliberately manual: it borrows your cursor and keyboard, counts down, proves click/scroll/type delivery by reading the target terminal's screen back over kitty remote control, and restores your focus.
Grounded in measured hot-path latencies and the finding that LLM calls are 76 to 96% of computer-use task latency (OSWorld-Human), so cutting round-trips beats shaving milliseconds. The round-trip work that framing motivated has largely shipped: sequence, act-and-observe then= (including then='ui'), and the accessibility-tree tools (ui, marks, click_ui) that target controls by name with no screenshot. What remains:
Faster
In-process wlr-screencopy over the raw wire (as input already works): drop grim's fork floor from small captures and add damage-tracked wait-for-stable that returns the instant the screen settles.
Fewer round-trips
Semantic screen diff: after an action, return only what changed (window topology from the event stream, or a bounded changed-region crop) instead of a full frame. The socket2 event expansion behind wait_for already tracks most of the topology; the missing piece is folding it into a post-action delta.
Deeper reading
Wider accessibility coverage: close the gaps the ui and marks tools hit today, chiefly GTK's newer combo boxes that publish neither their text nor selection (so a rendered dropdown value still needs a screenshot), plus AT-SPI value-change events so then='ui' can report a control settling without a poll.
Notifications: read recent desktop-notification content and history, not just wait for the popup to appear (wait_for already matches layer_open on the notification namespace).
Background
Web pages in the background, through the browser bridge (an extension and a native messaging host).
Trust
record tool: a scoped GIF or mp4 of the agent driving the desktop, via wf-recorder (a wlroots-family binary like grim), a visual companion to the journal.
Platform
sway / niri support: the wire client already speaks the wlr protocols; what remains is an IPC layer alongside hyprctl.py (contributions welcome).
Headless end-to-end tests in CI: needs a QEMU virtio-gpu VM, since Hyprland's aquamarine backend requires a real GPU render node that hosted runners lack.
Measurement
Zoom-loop precision benchmark: measure click accuracy of the coarse-to-fine loop against known targets.
End-to-end task-success benchmark on Hyprland (OSWorld-style): the deferred bigger sibling of the zoom-loop microbenchmark, scoring full multi-step tasks through the real MCP surface so the a11y-versus-vision and round-trip work is judged on task completion, not latency alone.
project | approach | on Hyprland |
computer-use-linux | AT-SPI + portals, ydotool fallback | GNOME-first; the RemoteDesktop portal it prefers is not implemented by xdg-desktop-portal-hyprland |
hyprmcp | hyprctl wrapper | window management only; no screenshots or input |
wayland-mcp | evemu input, VLM analysis | requires elevated setup for input; no Hyprland semantics |
Anthropic computer-use-demo | X11 + xdotool in Docker | a sandboxed reference environment rather than a live desktop |
Cua (cua-driver) | cross-platform driver; an optional Hyprland compositor plugin gives the agent its own input seats | the plugin must be built for the exact installed Hyprland version, and its README calls it experimental and unreleased |
hypruse ships no OCR engine; its universal precision mechanism is the coarse-to-fine zoom loop: screenshot a window, re-capture the target region at native resolution, click through the exact coordinate mapping. That choice follows what the GUI-agents field converged on. Anthropic's computer use grounds clicks from raw pixels and ships a zoom action as the documented fix for small text; its troubleshooting guidance for near-miss clicks prescribes zooming and region cropping, never OCR [1]. OpenAI's CUA is likewise pure pixel grounding under resolution discipline, with no OCR layer at all [2]. Zoom is also the measured lever: training-free iterative zooming roughly doubles high-resolution grounding accuracy (OS-Atlas-7B, 18.9 → 49.7 on ScreenSpot-Pro) [3], and the benchmark's official harness implements a dozen grounding-model adapters plus four zoom/crop strategies, but zero OCR baselines [4]. Vision-only agents match or beat agents that additionally consume HTML or accessibility trees [5], substrates Wayland doesn't guarantee anyway, and state-of-the-art native agents run from screenshots alone [7]. OCR was rejected because it is blind to icons, the element class every grounding model handles worst (SeeClick: 30-52% on icons vs 56-78% on text) [6]; where OCR survives in modern stacks it is a text-disambiguation sidecar, not the targeting mechanism [8].
Where an app exposes an accessibility tree, hypruse also reads it (the ui tool, AT-SPI over D-Bus) to target controls by name with exact coordinates and no screenshot. This follows the strongest Linux precedent: OSWorld, the standard computer-use benchmark, exposes the desktop accessibility tree (obtained on Ubuntu through AT-SPI) as a first-class observation alongside screenshots, and reports the accessibility-tree-plus-screenshot combination as its best configuration [9]. The tree gives exact element identity and coordinates that current models cannot reliably infer from pixels: Agent-S tags each element with an id because MLLMs "lack an internal coordinate system," lifting OSWorld success from 11.2 to 20.6 percent [10]; Microsoft's UFO drives Windows through the UI Automation tree and fuses it with vision [11]; browser agents read the accessibility tree, which Playwright serializes to compact YAML, rather than pixels for the same reason [12]. hypruse's Wayland-specific trick is coordinate mapping: AT-SPI screen coordinates are unreliable on Wayland because an app does not know its global position, so hypruse uses window-relative extents plus the window position it already has from hyprctl. Coverage is uneven by nature: canvas, games, terminals, and Electron/Chrome without a flag expose little or nothing [13]. Testing this implementation against real GTK and Qt apps sharpened that in both directions: a typed entry reads back its exact contents, and sliders and toggles report their position and state, but GTK's newer combo boxes publish neither their text nor a selection, so reading a rendered dropdown value still needs a screenshot. Toolkits also describe widgets they never laid out (zero-height scroll arrows, unrendered tab pages reporting origins in the millions), which hypruse rejects against the window rect hyprctl knows authoritatively. A strong visual grounder can substitute for the tree entirely [5], so the accessibility tree complements the zoom loop rather than replacing it, and vision stays the guaranteed fallback.
Anthropic, Computer use tool: platform docs (computer_20251124, enable_zoom, resolution guidance)
OpenAI, Computer-Using Agent and the computer use guide
DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning, EMNLP 2025. arXiv:2507.00008
Li et al., ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. arXiv:2504.07981; official harness
Gou et al., Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (UGround), ICLR 2025 Oral. arXiv:2410.05243
Cheng et al., SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents, ACL 2024. arXiv:2401.10935
Qin et al., UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv:2501.12326
Agyeya et al., Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents (Tesseract as a textual-grounding sidecar). arXiv:2504.00906
Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv:2404.07972; the a11y tree is obtained on Ubuntu through AT-SPI (code)
Agashe et al., Agent S: An Open Agentic Framework that Uses Computers Like a Human. arXiv:2410.08164
Zhang et al., UFO: A UI-Focused Agent for Windows OS Interaction (UI Automation + vision). arXiv:2402.07939
Playwright, ARIA snapshots: the accessibility tree as compact structured text
Wang et al., GUI Agents: A Survey (accessibility-API coverage gaps; a11y complements vision). arXiv:2412.13501
The user's own Hyprland keybinds: combo, action, arg, and a
description when the config provides one. This is how the desktop's
owner drives it: to perform one of these workflows, call use_bind
with the combo (it runs the bound action). An action of lua means the
bind is a closure in a Lua Hyprland config, which nothing can run from
outside: read its description and do the same thing with hypr or
launch. NOTE: the keyboard tool canNOT trigger these compositor
binds (synthetic keys reach apps, not Hyprland's bind matcher), so do
not try to press them.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only give readOnlyHint/openWorldHint; the description adds the crucial behavioral facts — that these binds are the desktop owner's own config, that lua binds are opaque closures unrunnable from outside, and that synthetic keystrokes cannot reach Hyprland's bind matcher. These are non-obvious constraints an agent would otherwise get wrong.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads what the tool returns, then pivots to the workflow routing and the caveat. Slightly dense with three separate admonitions in the back half, but every sentence carries distinct information; nothing is filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-param list tool with an output schema and annotations already covering the safety profile, the description closes the remaining gaps: what the entries mean, how to act on them via siblings, and the one trap (keyboard can't reach compositor binds). An agent has everything needed to call it and use the result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Zero parameters, so the baseline is 4. The description still adds value by naming the fields the result carries (combo, action, arg, description), which helps the agent interpret what comes back rather than guessing.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Identifies the resource (the user's own Hyprland keybinds) and enumerates the returned fields (combo, action, arg, description), which distinguishes it from use_bind, keyboard and the other desktop siblings. It lacks an explicit verb like 'list', but the framing and title make the read-only enumerating nature unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit routing: to run a bind, call use_bind with the combo; for `lua` binds that nothing can execute externally, read the description and reimplement with `hypr` or `launch`; and it explicitly warns that `keyboard` canNOT trigger these binds. That is when-to-use, when-not-to-use, and named alternatives all in one.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Click a control by its accessible NAME, or by a mark number from
the last marks capture, in ONE call: the control comes from the
accessibility tree, so no screenshot and no pixel estimation is spent.
On the window the human is in, the window is focused and the click
goes through the real pointer (visible cursor, same panic-kill
guarantees). On any other window it runs in the background: the
control's accessibility action presses it (a text field gets the app's
own focus), the human's cursor and focus stay put, and the result reads
the control back; a control with no action is refused there. When no
control matches, a name also finds an element in another role that has
an action (a web app's list row); if its action only focuses it, the
result says so and whether the app reports it focused, which does not
show what a key sent next will do: check with ui.
background true or false forces one way (false takes over the human's
screen). Pass exactly one of
name (matched against window's controls, exact accessible name
preferred, substring otherwise) or mark. An ambiguous name returns
the candidates instead of guessing: disambiguate with index (0-based
into that list) or a more specific name. Falls back with a note when
the app exposes no tree (use screenshot + zoom + pointer then).
then ('desktop'|'screenshot'|'ui'|'none') appends the result;
'ui' shows the click's effect on the controls in the same call.
allow_auth=true overrides the refusal to click a system
authentication dialog.
| Name | Required | Description | Default |
|---|---|---|---|
| mark | No | ||
| name | No | ||
| then | No | none | |
| index | No | ||
| button | No | left | |
| double | No | ||
| window | No | ||
| allow_auth | No | ||
| background | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=false and destructiveHint=true; the description goes well beyond them by disclosing focus/cursor side effects per window, panic-kill guarantees, the refusal of actionless controls in the background, the ambiguity-returns-candidates behavior, and the `allow_auth` override for system dialogs. No contradiction with the destructive/not-readOnly annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the core action ('Click a control by its accessible NAME... in ONE call') and packs substantial content with little filler. However, several clauses are dense and hard to parse in one pass (e.g. the sentence about an action that 'only focuses it'), and word count is high.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 9-parameter, 0%-schema-coverage tool with an output schema (so return shape needn't be explained), the description covers routing, fallbacks, side effects, and the auth override thoroughly. The main gaps are the unmentioned `button` and `double` click modifiers.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description carries the load and does define `name` matching semantics (exact preferred, substring otherwise), the exclusivity of `name`/`mark`, `index`, `then` values, `background`, and `allow_auth`. It omits `button`, `double`, and the scoping role of `window`, which are only inferable from their bare titles.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (click) and the two exact addressing modes (accessible name or `mark`), plus the mechanism (accessibility tree, no screenshot/pixel estimation). This contrast with `screenshot`/`zoom`/`pointer` makes the tool distinguishable from its siblings at a glance.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit routing rules: exactly one of `name` or `mark`, `background` true/false to force foreground vs background, `index` to disambiguate an ambiguous name, and a stated fallback ('use screenshot + zoom + pointer then') when no tree is exposed. These are named when-to-use conditions, not inferences.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Semantic desktop snapshot: monitors, workspaces, windows (address,
class, title, at + size in global coords), active window, cursor,
and layers: launchers (wofi/rofi), bars, notification popups, and
on-screen keyboards are NOT windows and appear only there, with a
best-effort kind and global geometry you can screenshot by region or
click into. A listed layer is one the compositor tracks, not one you
can see: it may be transparent or dormant, so screenshot when
visibility matters. Call first; act on the addresses it returns.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, so safety is covered; the description adds non-obvious behavior: layers are compositor-tracked but may be transparent or dormant, `kind` is best-effort, and layer geometry is usable for region screenshots or clicks. That is real disclosure beyond the structured fields.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the resource and what it contains, then the layer caveat, then the workflow directive — no wasted sentences. The middle sentence is dense and heavily nested, which slightly hurts scanability but every clause carries distinct information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-shape detail is not needed; the description still covers snapshot contents, layer semantics, and the intended call-first workflow. For a parameterless, read-only snapshot tool, nothing an agent needs to invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the baseline is 4. The description correctly implies a parameterless snapshot call and adds no misleading parameter claims.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb+resource ('Semantic desktop snapshot') and enumerates exactly what it returns (monitors, workspaces, windows with address/class/title/geometry, active window, cursor, layers). It also implicitly separates itself from the pixel-based `screenshot` sibling by stressing addresses and layer semantics rather than images.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Call first; act on the addresses it returns' gives explicit ordering guidance, and 'screenshot when visibility matters' names the alternative tool and the condition that selects it. An agent knows both when to reach for this tool and when to fall back to `screenshot`.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Window/workspace ops over IPC (instant, no vision).
action='workspace' (workspace: number/name/'special:name') |
'focus_window' (target: address) | 'move_window' (target + workspace,
silent) | 'close_window' (target) | 'fullscreen' (target?) |
'toggle_floating' (target?). then ('desktop'|'screenshot'|'ui'|'none')
appends the result to this call. While the background guarantee is on
(HYPRUSE_BACKGROUND=1), workspace, focus_window and fullscreen are
refused because they change the human's view, and move_window moves
only to the agent's own workspaces.
| Name | Required | Description | Default |
|---|---|---|---|
| then | No | none | |
| action | Yes | ||
| target | No | ||
| workspace | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover the safety profile (readOnlyHint=false, destructiveHint=true), and the description adds real context beyond that: the background-mode refusal rules, the workspace confinement of move_window, the 'silent' move, and how 'then' appends output to the call. It stops short of saying which actions are irreversible or what closing a window loses.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the instant/no-vision premise, then uses a compact pipe-delimited action list, then appends the constraint sentence last. Dense but every clause carries information; the backtick/pipe notation is scannable rather than wasteful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a multi-action IPC tool with an output schema (so return values need not be explained), the description covers the action vocabulary, the shared parameters, and the environment-dependent refusals. Nothing critical is missing, though the dispatch rules between hypr and its ui/desktop siblings remain implicit.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% and the schema has no enums, so the description carries the full burden and largely succeeds: it enumerates action values, target as an address, workspace as number/name/'special:name', and then as desktop|screenshot|ui|none. The terse 'address' wording and the mention of a 'silent' flag not present in the schema keep it from a 5.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource: window/workspace operations over IPC, and explicitly notes it is instant and vision-free, which nudges it apart from vision siblings like screenshot/click_ui. It does not name or contrast with siblings such as desktop or ui, so an agent still has to infer the boundary.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-not conditions: under HYPRUSE_BACKGROUND=1, workspace, focus_window and fullscreen are refused, and move_window is restricted to the agent's own workspaces. It lacks explicit guidance on choosing hypr over siblings like desktop or launch, but the conditional-use rules are solid.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Keyboard to an app. action='type' (text, unicode-safe) |
'key' (keys combo: 'ctrl+shift+t', 'esc', 'F5'; aliases
enter/esc/tab/backspace/pgup/pgdn/arrows, else XKB keysyms). Pass
window (an address from desktop) so keystrokes land in the intended
app rather than whatever currently holds focus. The window the human
is in is focused and typed into on their screen. Any other window gets
the keys in the background, with the human's focus left alone: text
goes into its text field through the accessibility interface, or key
by key through Hyprland's sendshortcut (a character the keyboard layout
has no key for is refused on that path). Keys sent that way, and key
combos, go to the control the app itself has focused, which is not the
field text went into through accessibility: focus a field first with
click_ui. Without window, keys go to
the focused window. background true or false forces one way. This
drives shortcuts the focused application handles
(ctrl+t, ctrl+l). It does NOT trigger Hyprland's own keybinds
(super+...): those go through use_bind, and workspace/window actions
through hypr. then ('desktop'|'screenshot'|'ui'|'none') appends the
result to this call. allow_auth=true overrides the default refusal to
type into a password field or a system authentication dialog (only when
a human intends that credential entry).
| Name | Required | Description | Default |
|---|---|---|---|
| keys | No | ||
| text | No | ||
| then | No | none | |
| action | Yes | ||
| window | No | ||
| allow_auth | No | ||
| background | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Well beyond the annotations (readOnlyHint=false, destructiveHint=true), the description discloses the foreground-vs-background split, that background text goes through the accessibility interface while key-by-key goes through Hyprland sendshortcut (with per-character refusal when no keysym exists), that background focus is separate from the clicked field, and that password/auth dialogs are refused unless allow_auth=true. This is unusually rich behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with action semantics and each sentence carries real information, but it is dense and the middle passage on accessibility vs sendshortcut is a long run-on that could be tightened. No filler, though.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter, high-risk input tool with an output schema present, the description covers targeting, focus semantics, safety refusals, and result chaining (`then`). Output values need not be explained given the output schema. An agent has everything needed to call it correctly and safely.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description carries the full burden — and it does: it defines `action` values, `text` (unicode-safe), `keys` with combo syntax and aliases, `window` (an address from desktop), `background` as a forcing flag, `then` as a result-appending selector, and `allow_auth` as an override. All 7 parameters gain meaning beyond their bare titles.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific statement of the resource ('Keyboard to an app') and enumerates the action modes ('type', 'key'), so the agent knows exactly what capability this is. It further distinguishes itself from siblings by name: it does NOT trigger Hyprland keybinds (use_bind) or window/workspace actions (hypr).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit routing rules are given: pass `window` to target a specific app, focus a field with click_ui before sending keys via the accessibility path, use use_bind for Hyprland keybinds and hypr for workspace actions. When-not guidance ('does NOT trigger Hyprland's own keybinds') is spelled out rather than implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Open an app: run command via Hyprland exec. Optional workspace placement
(silent, works even for single-instance apps like browsers, whose
window gets moved after it appears) and wait_s (1-30, default 8;
raise for slow apps). Without workspace, the app opens silently on
the next of the agent's own workspaces (HYPRUSE_AGENT_WORKSPACES, 99
down to 90 by default) that is not on screen, so the agent can work on
it in the background; background=false opens it on the human's
workspace instead. It waits for its own window (one on that workspace,
or of the program's own class), never one the human opens meanwhile.
An app that is already running opens its window from the running
process, on the human's screen, before it is moved: the note says so.
Returns the new window's address/class/title/workspace, or a timeout
note.
| Name | Required | Description | Default |
|---|---|---|---|
| wait_s | No | ||
| command | Yes | ||
| workspace | No | ||
| background | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Far exceeds what annotations provide: it discloses silent background launching, single-instance app window-moving behavior, that it waits only for its own window and never one the human opens, and the timeout outcome. This is unusually rich behavioral context for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action, and each sentence carries real information. However the single dense paragraph is long and slightly run-on, revisiting workspace placement behavior more than once.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a behaviorally complex launcher, the description covers the mechanism, placement logic, waiting semantics, edge cases, and even the return values (which the output schema already supplies). Nothing essential for correct invocation is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite 0% schema description coverage, the description fully annotates all four params: command semantics, workspace placement effect, wait_s range (1-30, default 8) and when to raise it, and the meaning of background. It compensates completely for the empty schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Open an app') and the exact mechanism ('run `command` via Hyprland exec'). An agent can distinguish this from siblings like screenshot, click_ui, or keyboard without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear conditional guidance for each option: use `workspace` for silent placement, raise `wait_s` for slow apps, set `background=false` to open on the human's workspace. It stops short of explicitly naming or excluding sibling tools, so no full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Set-of-Marks capture: a screenshot of the window WITH its accessible
controls drawn as numbered red marks, plus a JSON legend mapping each
number to the control's role, name, current value, and exact global
click point. One glance replaces the estimate-zoom-estimate loop for
every control the accessibility tree knows: read the number off the
image and call click_ui(mark=N) (or pointer at the legend's x,y).
window is an address from desktop (default: focused); name filters
the marked controls. Falls back to the plain legend when ImageMagick is
not installed, and to a fall-back-to-vision note when the app exposes
no tree (then use screenshot + zoom).
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | ||
| window | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, so the bar is lower, yet the description adds genuine degradation behavior: falling back to a plain legend when ImageMagick is missing, and to a fall-back-to-vision note when no tree exists. It does not cover return shape edge cases or performance costs, keeping it short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core purpose and the payoff (one glance replaces the zoom loop); the trailing sentences on params and fallbacks are information-dense. It is slightly long with stacked parentheticals, but each sentence carries actionable content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given an output schema exists, the description need not detail the legend's exact return structure, and it still sketches it. Combined with sibling routing, param semantics, and both failure modes, an agent has everything needed to call `marks` correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does: `window` is explained as an address from desktop with focused as default, and `name` as a filter over the marked controls. Both parameters gain meaning not present in the bare schema, though format syntax for the address string is not spelled out.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and artifact: a screenshot of a window with accessible controls drawn as numbered red marks, plus a JSON legend mapping numbers to role/name/value/click point. This is clearly distinguishable from the sibling `screenshot` (plain image) and `zoom` (magnified view) because it names the marking behavior explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly routes the agent: 'read the number off the image and call `click_ui(mark=N)` (or `pointer` at the legend's x,y)', replacing the estimate-zoom-estimate loop. It also states the when-not case by naming the fallback to `screenshot` + `zoom` when the app exposes no accessibility tree.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Mouse in global coordinates. action='move' (x,y) | 'click' (optional
x,y first; button left/right/middle; double=true) | 'drag' (x,y →
to_x,to_y holding button) | 'scroll' (scroll_dy notches, positive =
content down; optional x,y first). then appends the result to this
call so you skip a round-trip: 'desktop' a fresh snapshot, 'screenshot'
a stable capture, 'ui' the focused window's elements with current
values, 'none' (default) nothing. allow_auth=true overrides the
refusal to click over a system authentication dialog. The pointer is
the human's own cursor, so it never runs in the background: it is
refused where the input would land on a window other than the one the
human is in (press controls there with click_ui), unless
background=false.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | ||
| y | No | ||
| then | No | none | |
| to_x | No | ||
| to_y | No | ||
| action | Yes | ||
| button | No | left | |
| double | No | ||
| scroll_dx | No | ||
| scroll_dy | No | ||
| allow_auth | No | ||
| background | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only say destructiveHint=true; the description adds the crucial behavioral facts: the pointer is the human's own cursor and never runs in the background, calls are refused for non-focused windows unless background=false, and clicks over system auth dialogs are refused unless allow_auth=true. That is exactly the kind of context annotations cannot express.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the action menu, and the per-action parentheses keep it tight despite a lot of content; no sentence is filler. It is dense to the point of being hard to parse on first read, which costs a point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need no explanation; annotations cover the safety profile; the description covers refusal conditions, mode trade-offs, and round-trip avoidance via `then`. For a destructive 12-parameter mutation tool this is complete enough to call correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage across 12 parameters, the description carries the whole load and mostly succeeds: x/y, button values, double, to_x/to_y, scroll_dy direction, then-options, allow_auth and background are all explained. scroll_dx is never mentioned and the coordinate system is only implied by 'global coordinates'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Names the specific resource (the mouse/pointer) and scope (global coordinates), then enumerates every action with its argument shape: 'move' (x,y), 'click', 'drag', 'scroll'. It also distinguishes itself from the sibling click_ui by naming it as the fallback for other windows.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a concrete routing rule: where the pointer would land on a window the human is not in, use click_ui instead, and use allow_auth to override the authentication-dialog refusal. Clear context, though it never states when a caller should prefer ui/desktop navigation generally over raw pointer actions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Capture the focused monitor, a window (window: "active" or an
address from desktop, cheapest for reading one app), or a region
"x,y,WxH". Returns the image (or a file path to read) + JSON metadata
with geometry/scale for pixel→global mapping. A window on a hidden
workspace or under another window is still captured as itself
(metadata source "window"; without grim -T a shown window is cropped
from the screen, source "screen", and a hidden one is refused);
visible says whether the compositor draws it, and
a hidden one may show how it last looked (many apps stop drawing while
hidden). Window captures are refused while the session is locked. scale 0.1-1.0:
optional deliberate downscale, usually leave unset. stable=true
waits (up to 2s) until two consecutive frames match, so a capture
right after an action is not taken mid-animation; metadata gains
stable. Captures are fast JPEG by default; lossless=true returns
PNG for pixel-exact work. Before clicking a small control, follow with
zoom at the estimated point.
| Name | Required | Description | Default |
|---|---|---|---|
| scale | No | ||
| region | No | ||
| stable | No | ||
| window | No | ||
| lossless | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, so safety is covered, yet the description adds substantial non-obvious behavior: hidden-workspace and occlusion handling, the `source` window-vs-screen distinction, refusal while the session is locked, the 2s `stable` two-frame wait, and JPEG-vs-PNG defaults. This is exactly the operational context annotations cannot carry.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
It is front-loaded on the capture modes, which is good, but the middle becomes a dense thicket of parentheticals and implementation detail (grim -T, compositor internals) that is harder to parse than an agent needs. Several clauses could be tightened without losing actionable meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-parameter, zero-coverage schema the description is comprehensive: capture modes, failure conditions (locked session, hidden-window refusal), timing behavior, and output format are all addressed. Since an output schema exists it is not obligated to detail returns, and it stays appropriately brief on that front.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% (only titles and defaults), so the description must carry all five parameters — and it does: `window` ('active' or a desktop address), `region` ('x,y,WxH'), `scale` (0.1-1.0, usually unset), `stable` (two-frame wait), and `lossless` (PNG). Every parameter gets meaning the bare schema lacks.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb (capture) and enumerates the three selectable resources (focused monitor, window, region) with the exact syntax for each. It also routes to the `zoom` sibling for the follow-up case, so an agent can distinguish it from adjacent tools without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives real selection guidance: window is flagged as 'cheapest for reading one app', and the closing sentence tells the agent to follow with `zoom` before clicking small controls. What is missing is explicit when-not-to-use framing or named alternatives for full-screen vs region tradeoffs beyond the cost hint.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run an ordered list of actions in ONE call, so a click/type/enter
micro-sequence costs one round-trip instead of several. Each step is
{"op": "pointer"|"keyboard"|"click_ui"|"hypr"|"wait_for", ...that tool's
args}, e.g. [{"op":"pointer","action":"click","x":800,"y":60},
{"op":"keyboard","action":"type","text":"hello","window":"0x.."},
{"op":"keyboard","action":"key","keys":"enter"}]. With stop_on_change
(default) the run stops, best-effort, when it notices a STRUCTURAL
change between steps that the step did not intend: a window opening
(e.g. a dialog), closing, or moving, a switch to an unexpected
workspace, or a seat-taking layer surface (a launcher or on-screen
keyboard) appearing, so later steps do not act on stale state.
Notification popups and bars are not treated as changes. It does NOT catch
a bare focus change, so to type into a specific window reliably give
that keyboard step a window= address (the keys go to that window).
Bounded to 20
steps and ~30s total. then observes the final state ('desktop'
default, 'screenshot', 'ui', 'none'). background applies to every
pointer, keyboard and click_ui step that does not set its own, and
background=true refuses a hypr step that takes the human's view.
| Name | Required | Description | Default |
|---|---|---|---|
| then | No | desktop | |
| steps | Yes | ||
| background | No | ||
| stop_on_change | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the destructiveHint/readOnlyHint annotations: stop_on_change default behavior, precisely which events count as a structural change (window open/close/move, unexpected workspace, layer surface) and which don't (notification popups/bars, bare focus change), the window= workaround, hard limits (20 steps, ~30s), and background's refusal rule for view-taking hypr steps. This is exactly the behavioral context an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the value proposition before the mechanics, and every clause roughly earns its place. It is a dense single block with many nested parentheticals, which is slightly harder to scan than it could be, but no sentence is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive, multi-step, bounded tool this covers everything: limits, stop conditions, parameter inheritance, per-step addressing, and awaiting behavior. An output schema exists, so return values need not be described, and the remaining gaps are negligible.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description carries the full burden and does: it defines stop_on_change's detection semantics, `then`'s observed-state role and its 'desktop' default alongside 'screenshot'/'ui'/'none', and background's inheritance across pointer/keyboard/click_ui steps plus its hypr constraint. It also specifies the steps item shape via example even though the schema is a bare array.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource: it runs an ordered list of actions in one call, explicitly to collapse a click/type/enter micro-sequence into one round-trip. It names the exact ops the steps can carry (pointer, keyboard, click_ui, hypr, wait_for) and gives a concrete example, so an agent can distinguish it from the individual sibling action tools immediately.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clear context for use: batching several related actions to avoid multiple round-trips. The boundaries are given indirectly (steps reference the individual sibling tools' args, implying single actions belong there), but there is no explicit statement of when NOT to batch or when to prefer a single pointer/keyboard call.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read a window's accessibility tree (AT-SPI) and return its elements
with GLOBAL click points, so you can target a control by NAME with no
screenshot and no pixel guessing. window is an address from desktop
(default: the focused window). name filters to elements whose
accessible name contains it (case-insensitive); actionable (default)
keeps only interactive roles (buttons, entries, menu items, ...); with
a name that matches none of those, it keeps named elements in other
roles that have an accessibility action (a web app's list row), with
the action's name as action.
Returns [{role, name, x, y, clickable}] where x,y is the click point:
focus the window, then click it with pointer (the window must be
visible to receive the click, and focusing can move it, so read again
after focusing), or do both in one call with click_ui.
Controls that carry a CURRENT VALUE also
report it: value (text typed into an entry, or a slider/spinner
number), percent for a slider's position, checked for a box or
toggle. Password fields never report contents, and many dropdowns
expose no value at all, so read the screen with screenshot when a
rendered value matters. Not every app exposes a tree (terminals, and
Electron/Chrome without --force-renderer-accessibility, expose little
or nothing); when it does not, fall back to screenshot + zoom.
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | ||
| window | No | ||
| actionable | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint and openWorldHint, so the description carries real extra weight: password fields never report contents, many dropdowns expose no value, terminals and Electron/Chrome without --force-renderer-accessibility expose little or nothing, and focusing can move the window (requiring a re-read). These are non-obvious operational traits beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The core purpose and targeting model are front-loaded, and most sentences earn their place by preventing a failed click. It is dense and somewhat overloaded, however — the value/percent/checked enumeration and the parenthetical about dropdown values push it toward an inline manual rather than a crisp definition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-required-parameter read tool with rich annotations and an output schema, nothing an agent needs is missing: it knows the return shape, the interaction sequence, the scoping defaults, and the failure modes with fallbacks. Return-value detail is technically redundant given the output schema but does no harm.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% (titles and defaults only), so the description must compensate and does: `window` is an address from desktop defaulting to the focused window, `name` is a case-insensitive substring filter with defined fallback behavior for non-interactive roles, and `actionable` defaults true and keeps only interactive roles. This adds meaning well beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource ('Read a window's accessibility tree (AT-SPI)') and states the payoff ('GLOBAL click points ... no screenshot and no pixel guessing'), which cleanly separates it from the screenshot and zoom siblings. An agent can identify the tool's job without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It routes explicitly to alternatives and states conditions: click the returned point with `pointer`, do both in one call with `click_ui`, read the screen with `screenshot` when a rendered value matters, and fall back to 'screenshot + zoom' when an app exposes no tree. Both when-to-use and when-not-to-use are covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run one of the user's own Hyprland keybinds by its combo (from the
binds tool), e.g. 'SUPER+F'. This executes the bound action directly
(the only reliable way: synthetic keypresses do not trigger compositor
binds). Use it to drive the owner's configured workflows: launchers,
layout shortcuts, scratchpads. then ('desktop'|'screenshot'|'ui'|'none')
appends the result to this call (handy after a launcher bind). Refused
while HYPRUSE_CONFINE is set or the background guarantee is on
(HYPRUSE_BACKGROUND=1): a bind runs an arbitrary compositor action
that cannot be scoped to a window.
| Name | Required | Description | Default |
|---|---|---|---|
| then | No | none | |
| combo | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark it destructive/non-readOnly, but the description adds substantial behavior beyond them: the refusal conditions tied to environment variables, the justification that it executes an arbitrary compositor action that cannot be scoped to a window, and how the `then` result is appended to the call.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose and example, then rationale, use cases, the secondary parameter, and finally refusal conditions. Every sentence adds distinct information with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given an output schema exists, return values needn't be described. The description covers purpose, params, refusals, and safety posture fully; an agent has everything needed to invoke it correctly or know it will be rejected.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must carry the load and largely does: `combo` is explained as a combo string sourced from the `binds` tool with an example, and `then` is enumerated ('desktop'|'screenshot'|'ui'|'none') with its append behavior. It stops short of documenting the default ('none') or exact combo syntax rules.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb+resource: 'Run one of the user's own Hyprland keybinds by its combo', with a concrete example ('SUPER+F'). It explicitly distinguishes itself from synthetic-key approaches and points to the sibling `binds` tool as the source of valid combos.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
States when to use it ('to drive the owner's configured workflows: launchers, layout shortcuts, scratchpads') and when it is refused (HYPRUSE_CONFINE set, HYPRUSE_BACKGROUND=1), plus the rationale that synthetic keypresses cannot trigger compositor binds. Both the use case and the exclusion conditions are explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Block until a desktop event happens (real compositor events, not
polling). event: 'window_open' | 'window_close' | 'workspace' |
'title_change' | 'layer_open' | 'layer_close' (layer-shell surfaces:
launchers, notification popups; match on the namespace, e.g. 'wofi') |
'urgent' (a window demands attention) | 'screencast' (screen sharing
started/stopped). match: optional case-insensitive substring filter
over the event's fields (class/title/workspace name/address/namespace).
timeout_s 1-60, default 10. Returns the event payload, or a timeout
note; a filtered wait whose condition ALREADY holds (the window is
already gone, the workspace already active, the layer already mapped)
returns instantly with already: true instead of missing an event
that fired before it could subscribe. Use it after actions with
delayed effects: app startups, page loads that change a window title,
a launcher bind that pops a layer.
| Name | Required | Description | Default |
|---|---|---|---|
| event | Yes | ||
| match | No | ||
| timeout_s | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint/openWorldHint annotations, the description discloses non-polling semantics, the timeout range and default, the return shape (event payload or timeout note), and the important 'already: true' fast-path for conditions that already hold — a subtle behavior that would otherwise cause spurious misses. This is exactly the kind of context annotations cannot carry.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core behavior and information-dense throughout; every sentence earns its place. It is, however, one long run-on paragraph with deeply nested parentheticals, which costs some readability for an agent parsing it.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers the event vocabulary, filtering semantics, timeout bounds, the already-true edge case, and invocation context. An output schema exists, yet the return behavior is still summarized usefully — nothing needed to call this correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description carries the full burden and does: it enumerates all seven valid event values with explanations, defines match as a case-insensitive substring filter over class/title/workspace name/address/namespace, and pins timeout_s to 1-60 with default 10. This is richer than what an annotated schema would typically provide.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a precise verb and resource ('Block until a desktop event happens') and immediately scopes it as real compositor events rather than polling, then enumerates the exact event set. An agent can tell this apart from action-oriented siblings like launch, click_ui, or binds without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives concrete when-to-use triggers: 'after actions with delayed effects: app startups, page loads that change a window title, a launcher bind that pops a layer.' That routes the agent from siblings like launch/binds into this tool. It does not name an explicit alternative or state a when-not-to-use condition, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Native-resolution re-capture around a point: the precision step of
the coarse-to-fine loop. Screenshot first, estimate the target's global
x,y, zoom there, re-estimate on the zoomed image (scale ~1.0, so
global = geometry[:2] + image_pixel), then click. size "WxH" in
logical pixels (default 480x360) is clamped to the screen; window
(an address from desktop, on screen) clamps to that window instead. The metadata
echoes the requested point back as point; stable=true waits for
the frame to settle first; lossless=true returns PNG.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| size | No | ||
| stable | No | ||
| window | No | ||
| lossless | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint and openWorldHint; the description adds real behavioral context beyond them: size defaults to 480x360 and is clamped to the screen (or to `window`), metadata echoes the requested point back as `point`, stable=true waits for the frame to settle, and lossless=true returns PNG. These are exactly the traits an agent needs to predict output and side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose is front-loaded and every sentence carries information about a parameter or behavior. It is dense and slightly run-on, with workflow instructions for the agent folded into the tool description, but there is little genuine waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be enumerated, yet the description still notes that metadata echoes the point back as `point`. Combined with clamping rules, defaults, and the settle/PNG flags, an agent has everything needed to invoke it correctly in the coarse-to-fine loop.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must carry the burden, and it does: size is described as 'WxH' in logical pixels with a default and clamping rule, window is defined as an address from desktop that changes the clamp target, and stable/lossless are both explained with their effects. Only x/y are left implicit, but the workflow sentence pins them as global coordinates.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource: 'Native-resolution re-capture around a point'. It also positions the tool in a workflow as 'the precision step of the coarse-to-fine loop', which implicitly distinguishes it from the coarse sibling (screenshot). An agent can tell what this does without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The embedded sequence ('Screenshot first, estimate the target's global x,y, zoom there, re-estimate..., then click') gives clear context for when this tool is used in the loop. It does not name alternatives or exclusions explicitly, so it stops short of a 5 despite being highly actionable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.13.0click_ui1 field changedInput schema / properties / backgroundAdded value: +{
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "default": null,
+ "title": "Background"
+}keyboard1 field changedInput schema / properties / backgroundAdded value: +{
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "default": null,
+ "title": "Background"
+}launch1 field changedInput schema / properties / backgroundAdded value: +{
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "default": null,
+ "title": "Background"
+}pointer1 field changedInput schema / properties / backgroundAdded value: +{
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "default": null,
+ "title": "Background"
+}sequence1 field changedInput schema / properties / backgroundAdded value: +{
+ "anyOf": [
+ {
+ "type": "boolean"
+ },
+ {
+ "type": "null"
+ }
+ ],
+ "default": null,
+ "title": "Background"
+}v0.11.0click_uihyprlaunchuse_bindwait_forv0.9.4desktophyprkeyboard1 field changedInput schema / properties / allow_authAdded value: +{
+ "default": false,
+ "title": "Allow Auth",
+ "type": "boolean"
+}launchmarkspointerscreenshotuiuse_bindwait_forzoomv0.6.0pointerscreenshotsequencezoomv0.5.0desktophypr3 fields changedInput schema / properties / thenAdded value: +{
+ "default": "none",
+ "title": "Then",
+ "type": "string"
+}Output schema / properties / result / anyOfAdded value: +[
+ {
+ "items": {},
+ "type": "array"
+ },
+ {
+ "type": "string"
+ }
+]Output schema / properties / result / typeRemoved value: -"string"keyboard4 fields changedInput schema / properties / thenAdded value: +{
+ "default": "none",
+ "title": "Then",
+ "type": "string"
+}Input schema / properties / windowAdded value: +{
+ "default": "",
+ "title": "Window",
+ "type": "string"
+}Output schema / properties / result / anyOfAdded value: +[
+ {
+ "items": {},
+ "type": "array"
+ },
+ {
+ "type": "string"
+ }
+]Output schema / properties / result / typeRemoved value: -"string"pointer3 fields changedInput schema / properties / thenAdded value: +{
+ "default": "none",
+ "title": "Then",
+ "type": "string"
+}Output schema / properties / result / anyOfAdded value: +[
+ {
+ "items": {},
+ "type": "array"
+ },
+ {
+ "type": "string"
+ }
+]Output schema / properties / result / typeRemoved value: -"string"screenshot1 field changedInput schema / properties / losslessAdded value: +{
+ "default": false,
+ "title": "Lossless",
+ "type": "boolean"
+}use_bind3 fields changedInput schema / properties / thenAdded value: +{
+ "default": "none",
+ "title": "Then",
+ "type": "string"
+}Output schema / properties / result / anyOfAdded value: +[
+ {
+ "items": {},
+ "type": "array"
+ },
+ {
+ "type": "string"
+ }
+]Output schema / properties / result / typeRemoved value: -"string"zoom1 field changedInput schema / properties / losslessAdded value: +{
+ "default": false,
+ "title": "Lossless",
+ "type": "boolean"
+}v0.4.1screenshot1 field changedInput schema / properties / stableAdded value: +{
+ "default": false,
+ "title": "Stable",
+ "type": "boolean"
+}zoomv0.2.1bindsuse_bindwait_forv0.1.1desktophyprkeyboardlaunchpointerscreenshotScored across 14 tools
Each tool has a crisply distinct role, and the descriptions actively delineate boundaries (pointer vs click_ui, keyboard vs use_bind vs hypr, screenshot vs zoom vs marks vs ui). Overlapping-seeming tools like screenshot/zoom/marks and ui/click_ui are separated by explicit coarse-to-fine vs accessibility vs annotated-capture semantics. An agent can reliably pick the right tool.
All names are lowercase single tokens with underscore used consistently for multi-word tools (wait_for, click_ui, use_bind), so formatting is uniform. The only wobble is semantic: some names are nouns (desktop, marks, binds, pointer) and others verbs (launch, zoom), which is a minor deviation from a single predictable pattern.
14 tools is well within the ideal 3-15 range and each earns its place: perception (desktop, screenshot, zoom, ui, marks), input (pointer, keyboard, click_ui), orchestration (wait_for, sequence), and system control (hypr, launch, binds, use_bind) with no filler.
The desktop-automation surface is essentially full: snapshot, capture at two fidelities, accessibility read/annotate, mouse, keyboard, background-window targeting, event waiting, window/workspace IPC, app launch, keybind discovery and replay, plus a batch runner. It even documents fallback paths (vision when no a11y tree) and hard limits (locked session, auth dialogs), leaving no obvious dead ends.
Official MCP server for Agentwork — delegate tasks to AI agents with human-in-the-loop
MCP server for AI agents to plan, verify, and deploy Cloudflare-native apps.
MCP server for building and testing AI agents with multi-model experimentation and insights.
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…
