It gives AI agents hands-on control of a Hyprland Wayland desktop — semantic state, screenshots, accessibility-tree targeting, and native input — through MCP tools or shell verbs.
Inspect the desktop: monitors, workspaces, windows, layers, cursor, and active window in one call.
Capture screenshots of monitors, windows, or regions; zoom into points; return coordinate-mapping metadata.
Read a window's accessibility tree (AT-SPI) and click controls by name or numbered mark, no pixel guessing.
Control the mouse: move, click, drag, and scroll in global coordinates through the real pointer.
Type text or send app-level key combos, optionally focusing a target window first.
Manage windows/workspaces via Hyprland IPC: switch, focus, move, close, fullscreen, or float.
Launch applications on chosen workspaces and get the new window's address.
List and run the user's own Hyprland keybinds via binds/use_bind.
Chain multi-step actions with sequence, and wait for real compositor events like window open/close, title changes, or screen sharing.
Optionally read/write the clipboard (opt-in), with read-only mode, confinement, journals, dry runs, and replay for safe delegation.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@hypruse Take a screenshot of my focused window."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Computer use for Hyprland. An MCP server that gives AI agents native hands on your Wayland desktop: workspaces, windows, mouse, keyboard, screenshots.
No ydotool daemon. No root. No portals. No X11.
Computer use exists on macOS and Windows. On Linux there is effectively nothing: the Claude Desktop Linux beta explicitly ships without screen control, Anthropic's reference implementation is an X11 container, and the existing Wayland attempts lean on setuid uinput hacks or GNOME-only portals.
Meanwhile Hyprland already exposes everything an agent needs, better than any accessibility bridge: a complete IPC surface for state and window management, and first-class Wayland protocols for input. hypruse just wires them to MCP:
Semantic first. desktop returns the real window/workspace tree (addresses, classes, titles, geometry) in one call. The agent switches workspaces and focuses windows the way you do (instantly, over IPC), not by squinting at pixels.
Vision when it matters. Screenshots of a monitor, an exact window crop, or a zoomed region, with the geometry/scale metadata to map any pixel back to a clickable coordinate: a coarse-to-fine loop grounded in the GUI-agents research.
Native input. Clicks and scrolls are spoken directly over the Wayland wire (zwlr_virtual_pointer_v1); typing goes through wtype's virtual keyboard with a proper XKB keymap, unicode-safe on any layout.
Related MCP server: hyprland-mcp
agent (Claude Code, or any MCP client)
│ stdio
▼
hypruse
├── hyprctl -j ········▶ desktop state: monitors, workspaces, windows, layers
├── hyprctl dispatch ··▶ focus / move / close / launch / movecursor
├── grim ··············▶ screenshots: monitor, window crop, region
├── busctl (AT-SPI) ···▶ accessibility tree: named controls, current
│ values, exact coords (ui / marks / click_ui)
├── wtype ·············▶ keyboard (zwp_virtual_keyboard_v1, real XKB keymap)
└── raw Wayland wire ··▶ click & scroll (zwlr_virtual_pointer_v1)Optional binaries gate two more tools: imagemagick draws the numbered
overlay for marks, and wl-clipboard backs the opt-in clipboard tool.
Design decisions:
No ydotool / uinput. That path needs a daemon, udev rules or root, and types US scancodes that break on other layouts. hypruse is just another Wayland client of your compositor, same standing as wlrctl.
No portals. xdg-desktop-portal-hyprland does not implement the RemoteDesktop portal (InputCapture is capture, not injection), so anything built on libei/portals silently degrades on Hyprland. hypruse doesn't try.
Cursor positioning through the compositor's own cursor dispatcher (global logical coordinates, exact on any monitor layout), with only button/axis events on the virtual pointer, sidestepping the known multi-monitor bugs of absolute virtual-pointer motion (hyprwm/Hyprland#6749).
tool | what it does |
| One-call semantic snapshot: monitors, workspaces, windows (address/class/title/geometry), active window, cursor, and layer surfaces (launchers, bars, notification popups) with a best-effort kind and geometry. A listed layer is one the compositor tracks, not one you can see: transparent and dormant surfaces are reported too, so screenshot when visibility matters |
| Focused monitor, exact window crop by address, or |
| Native-resolution re-capture around an estimated point (optionally clamped to a window): the precision step before clicking small controls, same metadata contract |
| Read a window's accessibility tree (AT-SPI, GTK/Qt apps that expose one) and return clickable elements by name with exact global coordinates, no screenshot; reports current values too (typed text, slider position, checkbox state); falls back to vision when an app exposes nothing |
| Set-of-Marks capture: the window screenshot with every accessible control drawn as a numbered mark, plus a JSON legend (role, name, current value, exact click point per number); needs ImageMagick for the drawing, degrades to the legend alone without it |
| Click a control by accessible NAME or by a |
| move / click / drag / scroll (discrete wheel notches) in global coordinates |
| Type literal text (unicode-safe) or press app-level combos ( |
| Switch workspace, focus/move/close windows, fullscreen, floating (pure IPC, milliseconds; |
| Start an app (optionally silent on another workspace), block on its actual |
| The user's own keybinds, decoded ( |
| Execute a keybind by combo ( |
| Run an ordered list of actions (pointer/keyboard/click_ui/hypr/wait_for) in one call; stops the moment the desktop changes in a way the current step did not expect, so a click/type/enter micro-sequence costs one round-trip instead of several |
| Block on real compositor events (window open/close, workspace change, title change, layer surfaces appearing/closing, urgency, screen sharing on/off) with a match filter and timeout; a filtered wait whose condition already holds (window already gone, workspace already active, layer already mapped) answers instantly with |
| Read or write the text clipboard via |
The acting tools (pointer, keyboard, click_ui, hypr, use_bind, sequence) take an optional then argument that appends the result to the same call, so the agent sees the effect without a second round-trip: then='desktop' adds a fresh semantic snapshot (~20 ms, cheap, best for window/focus changes), then='screenshot' a stable capture (best for visual changes), then='ui' the acted-on window's controls with their current values (a few hundred exact tokens, best after typing or toggling; click_ui reads the window it clicked even if the click handed focus to a dialog, the others read the focused window), then='none' nothing (the default everywhere except sequence, which defaults to 'desktop').
Every tool is also a shell verb (hypruse desktop, hypruse click_ui Save --window 0x...) for agents that run commands instead of MCP, with a skill that teaches them: see Shell verbs and the agent skill.
The tools group into five capabilities, ordered most-reliable-and-cheapest first. An agent that reaches for them in this order is both faster and more accurate, and many tasks never need a screenshot at all.
desktop returns the entire window and workspace tree in one call: every window's address, class, title, and geometry, the active window, the cursor, and any layer surfaces the compositor is tracking (launchers, bars, notification popups). hypr and launch then act on it over IPC in milliseconds: switch workspace, focus/move/close/fullscreen/float a window by address, or start an app.
Use it well: never take a screenshot to find or arrange windows. Read desktop, act on the address you want. launch blocks on the real openwindow event and hands back the new window's address, so there is nothing to poll or guess; it also relocates single-instance apps (browsers) that ignore workspace rules.
When an app exposes an accessibility tree (most GTK and Qt apps), you can target controls by name instead of by pixel. ui lists every control with its exact global coordinate, and reports the current value of the controls that carry one: the text in a field, a slider's percentage, a checkbox's state. click_ui resolves a name and clicks it in one call, through the real cursor (so the beacon and every safety guarantee still apply). marks draws numbered marks over a screenshot with a legend, for when you want to see the options first and then click_ui(mark=N).
Use it well: reach for click_ui name="Save" before estimating any pixel, since it is exact and spends no image. Read a form's state with ui (did the box actually tick?) instead of screenshotting it. An ambiguous name returns the candidates rather than guessing. When an app exposes no tree (terminals, canvas apps, Electron/Chrome without --force-renderer-accessibility) the tool says so, and you fall back to vision.
For everything the accessibility tree cannot name, screenshot (monitor, window crop, or region) and zoom (a native-resolution re-capture around a point) carry a strict coordinate contract, global = geometry + pixel / scale, that stays exact on every monitor and fractional scale.
Use it well: don't guess a small control from a full-screen image. Work coarse-to-fine: screenshot the window, estimate the target, zoom there, re-estimate on the sharp crop, then click. This two-step loop is the research-backed way to hit small targets.
For an agent the model calls dominate task latency, not the desktop, so the real speedups are structural. sequence runs an ordered micro-plan (click, type, press enter, wait) in a single call, stopping the moment the desktop changes structurally in a way a step did not intend (a window opening, closing, or moving, an unexpected workspace switch, or a seat-taking launcher or on-screen keyboard; it deliberately ignores bare focus changes and notification popups). then='desktop' | 'screenshot' | 'ui' fuses a fresh view of the result into the acting call itself. wait_for blocks on real compositor events (a window or launcher opening, a title changing, a workspace switch, an urgency hint, screen-sharing starting) instead of sleeping and hoping.
Use it well: collapse a known click/type/enter flow into one sequence. After typing into a form, add then='ui' to read the effect back in a few hundred exact tokens instead of a screenshot. After a launch or a shortcut that opens something, wait_for the event rather than sleeping.
hypruse hands an agent your real seat, so it ships the controls to bound what that agent can do. Beyond the always-on approval prompts and the Waybar activity beacon, opt-in env flags narrow what an agent can touch: HYPRUSE_READONLY exposes only the observation tools; HYPRUSE_CONFINE restricts input to the windows the agent launched, or a class/workspace allowlist; HYPRUSE_AUTH_GUARD (on by default) refuses to drive authentication dialogs; HYPRUSE_STRICT refuses to act if you took the seat back; HYPRUSE_MARK tags agent-owned windows and announces when the agent opens a window or captures the screen. Two more record rather than restrict: HYPRUSE_JOURNAL writes an auditable NDJSON line per tool call, refusals included, and HYPRUSE_DRYRUN runs every check and delivers nothing, so you can watch an agent plan the work before it touches your desktop.
Use it well: run read-only for the first week. When you trust a workflow, allowlist its tools and, if you want to walk away, confine the agent to a scope so your password manager on another workspace stays untouchable. Keep a panic bind handy (hypruse stop, or pkill -f hypruse). The Security model has the full story.
Not every agent speaks MCP, and the ones that do increasingly prefer a command line: a tool list costs a few thousand tokens per session before the first call, a shell verb costs nothing until it runs. So the same fifteen tools are verbs, going through the same functions the MCP server registers, with the same trust guards, journal and activity beacon:
hypruse desktop # one line per monitor, workspace, window, layer
hypruse launch --workspace 2 -- firefox --new-window https://example.org
hypruse ui --window 0x55a4479a1200 --name Save # [0] push button "Save" @1204,88
hypruse click_ui Save --window 0x55a4479a1200 --then ui
hypruse screenshot --window 0x55a4479a1200 # a file path, then {"geometry":[x,y,w,h],"scale":1.0,...}
hypruse zoom 1180 840 --window 0x55a4479a1200
hypruse keyboard type "hello" --window 0x55a4479a1200
hypruse wait_for title_change --match Inbox --timeout 10
hypruse sequence '[{"op":"click_ui","name":"Search","window":"0x55a4479a1200"},{"op":"keyboard","action":"type","text":"btop"},{"op":"keyboard","action":"key","keys":"enter"}]'The contract is built for a program reading the output: the verbs are the tool names, a tool's action is a positional sub-verb (pointer click 800 60, hypr workspace 3, clipboard read), output is compact plain text (one line per fact, --json for the raw result as one line), a capture prints its file path and coordinate metadata rather than bytes, and errors are one line on stderr with a meaningful exit code: 0 delivered, 1 error, 2 usage, 3 refused by a trust layer (or by read-only mode), 4 ran but found nothing to act on (a wait_for timeout, no accessibility tree, an ambiguous name). hypruse --help lists every verb and hypruse <verb> --help its flags; --dry-run rehearses an acting verb. A verb starts in a few hundred milliseconds: the MCP stack is never imported on that path.
Between one-shot processes hypruse keeps the little state a verb would otherwise lose in $XDG_RUNTIME_DIR/hypruse/cli-state.json, keyed by the compositor instance: the marks numbering (so click_ui --mark N works), the launched confinement set, and the HYPRUSE_STRICT seat baseline, which is the one that would otherwise fail open.
The skill that teaches an agent the verbs, the desktop-first workflow and the safety rules ships inside the package. Install it into the skill directories of the agents on your machine (Claude Code, Codex, Pi, Hermes, OpenClaw, OpenCode, Gemini, Antigravity, Cursor, Copilot), or via the skills CLI:
hypruse skill install # ~/.agents/skills/hypruse, linked into each agent that is installed
hypruse skill install --agent codex # one agent, created if absent
npx skills add IlyasKhallouki/hypruse -ghypruse init offers the same install. The skill pre-approves only the observation verbs, doctor, --help and reading the capture files; acting verbs stay behind your agent's own approval prompt, which is the boundary the Security model leans on.
Requirements: Hyprland (both config managers: hyprland.conf and the Lua hyprland.lua that 0.56 introduced), grim, wtype (most Hyprland setups already have both), and uv. The accessibility tools (ui/marks/click_ui) use busctl, which ships with systemd. Optional: wl-clipboard for the opt-in clipboard tool, imagemagick for numbered marks captures.
Arch Linux, from the AUR:
yay -S hypruse # or hypruse-git for mainThen let it set itself up and verify the environment:
hypruse init # detects your MCP clients, registers (asks first), offers the agent skill, runs doctor
hypruse doctor # just the diagnosticsManual registration, Claude Code:
claude mcp add -s user hypruse -- uvx hypruseFrom a source checkout:
claude mcp add -s user hypruse -- uv run --directory /path/to/hypruse hypruseAny other MCP client: run uvx hypruse as a stdio server. hypruse is also in
the official MCP registry as
io.github.IlyasKhallouki/hypruse, so clients that browse the registry can
install it from there.
Read-only mode: set HYPRUSE_READONLY=1 in the server config to expose only the observation tools (desktop, screenshot, zoom, ui, marks, binds, wait_for). The agent can see and narrate but cannot click, type, or launch. A good first week.
The Linux beta ships without Anthropic's first-party computer use, but stdio MCP servers work in chat, which makes hypruse the workaround. In ~/.config/Claude/claude_desktop_config.json:
{
"mcpServers": {
"hypruse": {
"command": "uvx",
"args": ["hypruse"],
"env": { "HYPRUSE_SCREENSHOT_MODE": "image" }
}
}
}Two Desktop-specific notes: use image mode (Desktop renders inline MCP images and has no file-read tool), and the app must run natively inside your Hyprland session so the server inherits WAYLAND_DISPLAY/HYPRLAND_INSTANCE_SIGNATURE; from a VM or container it cannot reach your compositor. If your Desktop install bypasses tool-approval prompts, treat the Waybar indicator + panic keybind as mandatory, not optional.
Read this section before installing. hypruse hands an agent your mouse, your keyboard, your screen contents, and an app launcher. The layers that keep that sane:
Approval: MCP clients gate tool calls. In Claude Code, allowlist the read-only tools (desktop, screenshot) and leave pointer/keyboard/hypr/launch on ask-first until you trust a workflow.
Visibility: the server maintains an activity beacon ($XDG_RUNTIME_DIR/hypruse/state.json); the shipped Waybar module is invisible when idle and shows a robot indicator while an agent has hands on your desktop.
Interruption: click the indicator, or bind a panic key. The portable form works for every install: bind = SUPER SHIFT, BackSpace, exec, pkill -f hypruse. If hypruse is on your PATH (the AUR or a pipx install), bind = SUPER SHIFT, BackSpace, exec, hypruse stop is nicer: it signals the server to shut down gracefully, releasing any held pointer button and clearing the beacon. For a uvx install use exec, uvx hypruse stop; for a source checkout, exec, uv run --directory /path/to/hypruse hypruse stop. Killing it mid-action is safe either way: button press/release pairs never span tool calls, and even a long drag's held button is released on the way out.
The seat is shared. There is one cursor and one keyboard focus, and Hyprland's focus-follows-mouse means a cursor move alone can retarget keystrokes. Don't type while an agent is driving; watch the indicator.
Scope: stdio only (no network listener), nothing persisted except the beacon and the capped screenshot cache in $XDG_RUNTIME_DIR (tmpfs, newest 20) and, if you turn it on, the action journal on disk under $XDG_STATE_HOME (rotated, one generation, and text-redacted by default). No clipboard access unless you opt in: HYPRUSE_CLIPBOARD=1 registers a clipboard tool (never in read-only mode); clipboards hold passwords, so leave it off unless a workflow needs it. A screenshot sees everything visible: treat an agent session like screen sharing.
What the agent reads is untrusted. Window titles, accessibility names and values, and clipboard text flow verbatim into the agent's context, and any web page, filename, or document can put instructions there (prompt injection). hypruse cannot sanitize meaning, so the approval layer is the backstop: keep consequential tools (launch, keyboard, clipboard) on ask-first when the agent will look at untrusted windows, and treat "the screen told me to" as attacker input when reviewing an approval prompt.
Input never lands where it silently would not work. Three always-on checks (no env flag) refuse or annotate rather than report a phantom success: a click aimed under a launcher or on-screen keyboard layer surface (which sits above windows and would swallow it), typing while a launcher holds the keyboard grab, and any input while the session is locked (a live hyprlock/swaylock process, which is an ext-session-lock client invisible to the window and layer lists). While locked, keyboard/click_ui/pointer refuse unless allow_auth=true says a human wants the agent driving the unlock prompt. These are truthfulness aids, not a sandbox: they fail open on an unreadable system state, so they harden the common case without being a boundary you can lean on.
Six opt-in env flags. The first four narrow what an agent can touch, and each fails toward less action; the last two record and rehearse rather than restrict. All compose with the layers above:
HYPRUSE_CONFINE restricts input to a scope of windows: launched (only windows hypruse itself opened this session), class:firefox,kitty, or workspace:3,special:notes. Keyboard, click_ui, and hypr window ops are refused outside the scope; a pointer click is refused when any window under the point is out of scope (Hyprland's window list is not z-ordered, so hypruse fails closed rather than guess which window is on top). This is what lets you leave an agent working while your password manager sits on another workspace, untouchable. use_bind is refused outright while confinement is set, because a keybind runs an arbitrary compositor action that cannot be scoped to a window.
HYPRUSE_AUTH_GUARD (default on) refuses to click or type into a system authentication dialog (polkit agents, the GNOME keyring prompt), so a manipulated agent cannot approve a privilege escalation. Set HYPRUSE_AUTH_GUARD=strict to also refuse typing into a password field inside an ordinary window (a browser login), detected via the accessibility tree. A per-call allow_auth=true on pointer/keyboard/click_ui overrides it, and because it changes the tool's arguments the override surfaces distinctly in the approval prompt. HYPRUSE_AUTH_GUARD=0 disables it.
HYPRUSE_STRICT refuses to act when the cursor or focused window moved since hypruse's last action (the human, or a popup, took the seat): the agent must re-read desktop/screenshot and retry, so it never types into a window you just switched to.
HYPRUSE_MARK makes the agent's presence legible on the desktop: it tags every window the agent opens hypruse-owned and flashes an on-screen notice when the agent opens a window or captures the screen. It also installs a border_color window rule on that tag so owned windows get a colored outline, but whether a runtime rule renders depends on your Hyprland version and config precedence (on some setups it does not take effect); when it cannot be installed at all, hypruse says so on stderr rather than leaving you with a marking layer that is quietly not running. For a guaranteed outline, add the rule to your own config, which hypruse's tagging then matches: windowrule = border_color rgb(ff5555), tag hypruse-owned in hyprland.conf (older Hyprland: tag:hypruse-owned), or hl.window_rule({ match = { tag = "hypruse-owned" }, border_color = "rgb(ff5555)" }) in hyprland.lua.
HYPRUSE_JOURNAL records what the agent did: one NDJSON line per tool call in $XDG_STATE_HOME/hypruse/journal.ndjson (HYPRUSE_JOURNAL=1), or a path of your own. Read it with hypruse journal, re-run it with hypruse replay. See The record below.
HYPRUSE_DRYRUN is a rehearsal: every argument check and every guard above runs, then the call reports what it would have done and delivers nothing.
The flags above decide what an agent may do in the moment and then forget it happened. HYPRUSE_JOURNAL is the memory. One JSON object per line, so tail -f, grep, and jq all work on a live file:
{"v":1,"seq":7,"ts":"2026-08-02T09:12:13.456Z","kind":"act","tool":"pointer",
"args":{"action":"click","x":800,"y":60},"outcome":"ok","ms":37,"result":"click ok"}kind splits the two questions people actually ask: act is input delivered to your desktop, observe is the agent looking, which is what a privacy audit wants (when the screen was captured, when the clipboard was read). Observation results are never recorded, only that they happened, so the journal never becomes a second copy of everything the agent saw. Refusals are recorded too, with the guard's own message: it is the only place the history of your trust layers doing their job exists.
Typed and copied text is recorded as a length plus a short digest, not as text, because keystrokes are passwords. HYPRUSE_JOURNAL_TEXT=1 keeps it verbatim, which you need only to replay typing. Note what a digest is and is not: it proves two entries typed the same thing and it will not hand a reader your password, but it is an unsalted SHA-256 prefix next to an exact character count, so a four-digit PIN is trivially recovered from it. Treat the journal as sensitive either way. It is written 0600 in a 0700 directory, and rotated at HYPRUSE_JOURNAL_MAX_BYTES (8 MiB, one previous generation kept as .1; set 0 to never rotate).
The journal is a recorder, not a guard: if it cannot be written the action still happens and hypruse warns once on stderr, because failing your desktop over a log line is the wrong trade. What it does not record is a then= observation as its own entry: the acting call that carried it is recorded, then=screenshot and all, but the capture does not get a second line of its own.
Read it back with hypruse journal (--acts for actions only, --refused for what the guards stopped, -n N to tail, -v to include each result):
1 09:12:10 session start pid 4211 0.10.0 confine=launched auth_guard=1 strict=True
14 09:12:13 act pointer action=click x=800 y=60
16 09:12:15 act keyboard action=type text=<11 chars>
19 09:12:19 act hypr action=close_window target=0x5f2a10
REFUSED TrustError: 0x5f2a10 (Signal) is outside the agent's confinement scope
312 actions (0 dry), 604 observations, 1 refused by a trust layer, 0 errorsHYPRUSE_DRYRUN=1 turns the same session into a rehearsal. Every acting tool validates its arguments and runs every trust guard, then reports the plan instead of executing it, so a dry run refuses exactly what a real run would:
DRY RUN, nothing was delivered: would click push button 'Send' at (1204, 880) in signalNothing reaches the desktop: not the click, not the keystroke, not even the window focus that normally precedes typing. Enforced twice, once at each tool and once at the input path itself, so a code path nobody thought of fails loudly rather than quietly acting during a simulation. The agent is told dry run is on, so it reports a plan instead of retrying an action that "did not work". The scope is the agent's actions, not the server's own startup: HYPRUSE_MARK, if you set it, still installs its window rule when the server starts.
hypruse replay <journal> re-issues a journal's actions through the same tool functions, so the same guards apply to the replay. It prints the plan and stops there unless you pass --execute. Before it takes the seat it refuses outright, rather than failing halfway and leaving your desktop part-way through someone else's plan, when: an action was recorded by a newer hypruse, a recorded window no longer exists (--skip-missing runs the rest), typed text was recorded as a digest, the entry is a click_ui(mark=N) whose numbering died with the session that drew it, the entry is a clipboard write and HYPRUSE_CLIPBOARD is not set, or HYPRUSE_READONLY or HYPRUSE_DRYRUN is set. It paces itself from the recorded timing, capped by --max-gap and scaled by --speed, and its own actions are recorded and marked, so replaying the same file again runs the original plan rather than the plan plus the replay of it.
The honest limit is window addresses: they are heap pointers, so yesterday's journal mostly names windows that are gone, and an address can even be reused by a different window later, which no pre-flight can catch. Replay is for re-running a flow on a desktop that still looks like the one recorded.
Dry run and replay compose in the obvious direction: let the agent work with HYPRUSE_DRYRUN=1, read the journal, and replay it with --execute once the plan is one you like.
Measured on a live session (Hyprland 0.55, 1080p, 20 windows): desktop
~20 ms (one batched hyprctl call), workspace/window dispatch ~10-20 ms, full-monitor screenshot ~65 ms
(fast JPEG default; ~800 ms if you ask for lossless PNG, grim's zlib
path dominates), region/zoom captures well under that. If tool
calls feel slow, it is almost certainly the MCP approval prompt in
front of each call, not the server. Allowlist the tools you trust and the
latency disappears. Claude Code (.claude/settings.json):
{
"permissions": {
"allow": [
"mcp__hypruse__desktop",
"mcp__hypruse__screenshot",
"mcp__hypruse__hypr"
// add pointer/keyboard/launch once you trust your workflows
]
}
}Everything speaks Hyprland's global logical coordinates, the space hyprctl cursorpos and window at use. Screenshots are pixel-space; each capture returns geometry and scale so global = origin + pixel / scale. On scale 1.0 monitors (most setups) image pixels are global coordinates.
The zoom tool does the precision arithmetic for the agent: give it an estimated global point and it captures a native-resolution box around it, clamped to the screen (or to a window), with the same metadata contract. That two-step loop, estimate on the full view then re-estimate on the zoom, is the research-backed way to hit small controls.
Captures default to JPEG q90: on a 1080p frame that is roughly 12x faster to encode than PNG (grim's zlib path dominates capture time, measured ~65 ms vs ~800 ms) and 3-4x smaller, while full-res q90 reads UI text well. Pass lossless=true for exact pixels (PNG). In image mode, captures also fit the host's result-size limit (Claude Desktop caps tool results at 1 MB) by degrading quality before resolution, since grim's downscale filter is slower than a full-res capture, and cap the long edge at HYPRUSE_MAX_IMAGE_EDGE pixels (default 1568) so the host never downscales the image under the model. The applied scale is folded into the returned metadata, so coordinate mapping stays exact; tune with HYPRUSE_MAX_IMAGE_BYTES, or pass scale for a deliberate zoom-out.
By default the screenshot tool writes the image under $XDG_RUNTIME_DIR/hypruse/ and returns its path; MCP hosts with a file reader (Claude Code's Read) render it natively. This default exists because some hosts (including Claude Code 2.1.x) serialize inline MCP image blocks to base64 text the model cannot see. HYPRUSE_SCREENSHOT_MODE=image switches to inline image content blocks for hosts that render them correctly.
uv sync --group dev
uv run pytest # unit tests, no compositor needed
uv run pytest -m e2e --override-ini addopts= # live seat-safe checks
uv run python scripts/e2e_input.py # supervised: takes the seat ~10sThe input e2e is deliberately manual: it borrows your cursor and keyboard, counts down, proves click/scroll/type delivery by reading the target terminal's screen back over kitty remote control, and restores your focus.
Grounded in measured hot-path latencies and the finding that LLM calls are 76 to 96% of computer-use task latency (OSWorld-Human), so cutting round-trips beats shaving milliseconds. The round-trip work that framing motivated has largely shipped: sequence, act-and-observe then= (including then='ui'), and the accessibility-tree tools (ui, marks, click_ui) that target controls by name with no screenshot. What remains:
Faster
In-process wlr-screencopy over the raw wire (as input already works): drop grim's fork floor from small captures and add damage-tracked wait-for-stable that returns the instant the screen settles.
Fewer round-trips
Semantic screen diff: after an action, return only what changed (window topology from the event stream, or a bounded changed-region crop) instead of a full frame. The socket2 event expansion behind wait_for already tracks most of the topology; the missing piece is folding it into a post-action delta.
Deeper reading
Wider accessibility coverage: close the gaps the ui and marks tools hit today, chiefly GTK's newer combo boxes that publish neither their text nor selection (so a rendered dropdown value still needs a screenshot), plus AT-SPI value-change events so then='ui' can report a control settling without a poll.
Notifications: read recent desktop-notification content and history, not just wait for the popup to appear (wait_for already matches layer_open on the notification namespace).
Trust
record tool: a scoped GIF or mp4 of the agent driving the desktop, via wf-recorder (a wlroots-family binary like grim), a visual companion to the journal.
Platform
sway / niri support: the wire client already speaks the wlr protocols; what remains is an IPC layer alongside hyprctl.py (contributions welcome).
Headless end-to-end tests in CI: needs a QEMU virtio-gpu VM, since Hyprland's aquamarine backend requires a real GPU render node that hosted runners lack.
Measurement
Zoom-loop precision benchmark: measure click accuracy of the coarse-to-fine loop against known targets.
End-to-end task-success benchmark on Hyprland (OSWorld-style): the deferred bigger sibling of the zoom-loop microbenchmark, scoring full multi-step tasks through the real MCP surface so the a11y-versus-vision and round-trip work is judged on task completion, not latency alone.
project | approach | on Hyprland |
computer-use-linux | AT-SPI + portals, ydotool fallback | GNOME-first; the RemoteDesktop portal it prefers is not implemented by xdg-desktop-portal-hyprland |
hyprmcp | hyprctl wrapper | window management only; no screenshots or input |
wayland-mcp | evemu input, VLM analysis | requires elevated setup for input; no Hyprland semantics |
Anthropic computer-use-demo | X11 + xdotool in Docker | a sandboxed reference environment rather than a live desktop |
hypruse ships no OCR engine; its universal precision mechanism is the coarse-to-fine zoom loop: screenshot a window, re-capture the target region at native resolution, click through the exact coordinate mapping. That choice follows what the GUI-agents field converged on. Anthropic's computer use grounds clicks from raw pixels and ships a zoom action as the documented fix for small text; its troubleshooting guidance for near-miss clicks prescribes zooming and region cropping, never OCR [1]. OpenAI's CUA is likewise pure pixel grounding under resolution discipline, with no OCR layer at all [2]. Zoom is also the measured lever: training-free iterative zooming roughly doubles high-resolution grounding accuracy (OS-Atlas-7B, 18.9 → 49.7 on ScreenSpot-Pro) [3], and the benchmark's official harness implements a dozen grounding-model adapters plus four zoom/crop strategies, but zero OCR baselines [4]. Vision-only agents match or beat agents that additionally consume HTML or accessibility trees [5], substrates Wayland doesn't guarantee anyway, and state-of-the-art native agents run from screenshots alone [7]. OCR was rejected because it is blind to icons, the element class every grounding model handles worst (SeeClick: 30-52% on icons vs 56-78% on text) [6]; where OCR survives in modern stacks it is a text-disambiguation sidecar, not the targeting mechanism [8].
Where an app exposes an accessibility tree, hypruse also reads it (the ui tool, AT-SPI over D-Bus via busctl) to target controls by name with exact coordinates and no screenshot. This follows the strongest Linux precedent: OSWorld, the standard computer-use benchmark, exposes the desktop accessibility tree (obtained on Ubuntu through AT-SPI) as a first-class observation alongside screenshots, and reports the accessibility-tree-plus-screenshot combination as its best configuration [9]. The tree gives exact element identity and coordinates that current models cannot reliably infer from pixels: Agent-S tags each element with an id because MLLMs "lack an internal coordinate system," lifting OSWorld success from 11.2 to 20.6 percent [10]; Microsoft's UFO drives Windows through the UI Automation tree and fuses it with vision [11]; browser agents read the accessibility tree, which Playwright serializes to compact YAML, rather than pixels for the same reason [12]. hypruse's Wayland-specific trick is coordinate mapping: AT-SPI screen coordinates are unreliable on Wayland because an app does not know its global position, so hypruse uses window-relative extents plus the window position it already has from hyprctl. Coverage is uneven by nature: canvas, games, terminals, and Electron/Chrome without a flag expose little or nothing [13]. Testing this implementation against real GTK and Qt apps sharpened that in both directions: a typed entry reads back its exact contents, and sliders and toggles report their position and state, but GTK's newer combo boxes publish neither their text nor a selection, so reading a rendered dropdown value still needs a screenshot. Toolkits also describe widgets they never laid out (zero-height scroll arrows, unrendered tab pages reporting origins in the millions), which hypruse rejects against the window rect hyprctl knows authoritatively. A strong visual grounder can substitute for the tree entirely [5], so the accessibility tree complements the zoom loop rather than replacing it, and vision stays the guaranteed fallback.
Anthropic, Computer use tool: platform docs (computer_20251124, enable_zoom, resolution guidance)
OpenAI, Computer-Using Agent and the computer use guide
DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning, EMNLP 2025. arXiv:2507.00008
Li et al., ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. arXiv:2504.07981; official harness
Gou et al., Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (UGround), ICLR 2025 Oral. arXiv:2410.05243
Cheng et al., SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents, ACL 2024. arXiv:2401.10935
Qin et al., UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv:2501.12326
Agyeya et al., Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents (Tesseract as a textual-grounding sidecar). arXiv:2504.00906
Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv:2404.07972; the a11y tree is obtained on Ubuntu through AT-SPI (code)
Agashe et al., Agent S: An Open Agentic Framework that Uses Computers Like a Human. arXiv:2410.08164
Zhang et al., UFO: A UI-Focused Agent for Windows OS Interaction (UI Automation + vision). arXiv:2402.07939
Playwright, ARIA snapshots: the accessibility tree as compact structured text
Wang et al., GUI Agents: A Survey (accessibility-API coverage gaps; a11y complements vision). arXiv:2412.13501
The user's own Hyprland keybinds: combo, action, arg, and a
description when the config provides one. This is how the desktop's
owner drives it: to perform one of these workflows, call use_bind
with the combo (it runs the bound action). An action of lua means the
bind is a closure in a Lua Hyprland config, which nothing can run from
outside: read its description and do the same thing with hypr or
launch. NOTE: the keyboard tool canNOT trigger these compositor
binds (synthetic keys reach apps, not Hyprland's bind matcher), so do
not try to press them.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It explains that this is a read-only listing of binds, that `lua` actions are closures that cannot be run externally, and that synthetic keys from the `keyboard` tool will not trigger compositor binds. It does not explicitly state that the tool returns a list, but the output schema exists and the description's framing as 'the user's own keybinds' implies a listing. The behavioral caveats about `lua` and `keyboard` are valuable beyond what any schema would show.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single dense paragraph that front-loads the core purpose and then adds routing guidance. Every sentence earns its place: the first defines the resource, the second explains how to use it, the third handles the `lua` edge case, and the fourth warns against the `keyboard` tool. It is slightly long but not bloated, and the structure is logical. A small formatting improvement (separating the warning) would make it a 5.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that this is a zero-parameter listing tool with an output schema, the description covers the essential context: what the data is, how to act on it, and what not to do. It does not describe the exact return format, but the output schema exists and the description's mention of fields (combo, action, arg, description) aligns with that. The edge case of `lua` actions is handled. The only minor gap is not explicitly stating that the tool returns a list of all binds, but that is inferable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the schema is trivially complete (100% coverage). The description adds context about what the returned data means (combo, action, arg, description) and how to use it, which is more than a baseline 4 would require. Since there are no parameters to document, the description's job is to explain the data model and usage, which it does.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that this tool exposes the user's own Hyprland keybinds (combo, action, arg, description) and explicitly frames it as a read-only listing of the desktop owner's workflows. It distinguishes itself from siblings by explaining that these are compositor-level binds that the `keyboard` tool cannot trigger, and it names `use_bind` as the way to execute them. The verb 'binds' is a noun here, but the description makes the resource and its role unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use guidance: to perform one of these workflows, call `use_bind` with the combo. It also provides a when-not-to-use exclusion: the `keyboard` tool cannot trigger these binds, so do not try to press them. It even handles the `lua` action edge case, telling the agent to read the description and use `hypr` or `launch` instead. This is exemplary routing behavior.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Click a control by its accessible NAME, or by a mark number from
the last marks capture, in ONE call: the exact coordinate comes from
the accessibility tree, the window is focused first, and the click goes
through the real pointer (visible cursor, same panic-kill guarantees),
so no screenshot and no pixel estimation is spent. Pass exactly one of
name (matched against window's controls, exact accessible name
preferred, substring otherwise) or mark. An ambiguous name returns
the candidates instead of guessing: disambiguate with index (0-based
into that list) or a more specific name. Falls back with a note when
the app exposes no tree (use screenshot + zoom + pointer then).
then ('desktop'|'screenshot'|'ui'|'none') appends the result;
'ui' shows the click's effect on the controls in the same call.
allow_auth=true overrides the refusal to click a system
authentication dialog.
| Name | Required | Description | Default |
|---|---|---|---|
| mark | No | ||
| name | No | ||
| then | No | none | |
| index | No | ||
| button | No | left | |
| double | No | ||
| window | No | ||
| allow_auth | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and delivers extensively: the click goes through the real pointer with visible cursor and 'same panic-kill guarantees', the window is focused first (a side effect), ambiguous names return candidates instead of guessing, the tool auto-falls back when no tree exists, and `allow_auth=true` overrides a refusal to click system authentication dialogs — a major safety-relevant behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single dense paragraph where every clause earns its place — mechanism, disambiguation, fallback, `then`, and auth override are all covered without filler, and the core action is front-loaded in the first clause. It is slightly run-on and would benefit from visual separation, but nothing is redundant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 8-parameter tool with zero schema parameter descriptions, this is quite complete: core action, matching semantics, ambiguity resolution, no-tree fallback routing, `then` behavior, and auth-dialog safety are all present. An output schema exists, so return values are covered by structured data. The only material gaps are `button` and `double` semantics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It adds real semantics for six parameters: `name` matching (exact preferred, substring otherwise), `mark` sourced from the last `marks` capture, 0-based `index` for disambiguation, all four `then` values with the 'ui' effect explained, and `allow_auth` overriding the auth-dialog refusal. However, `button` and `double` are left undocumented — their defaults hint at meaning but the description adds nothing.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Click a control by its accessible NAME, or by a mark number', and explains the mechanism (coordinate from the accessibility tree, window focused first, real pointer). The phrase 'no screenshot and no pixel estimation is spent' explicitly distinguishes it from the screenshot/zoom/pointer workflow among its siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use (click by accessible name or mark in one call) and explicit when-not: 'Falls back with a note when the app exposes no tree (use screenshot + zoom + pointer then)', naming the alternatives. It also instructs to pass exactly one of `name` or `mark`, and explains when `index` should be supplied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Semantic desktop snapshot: monitors, workspaces, windows (address,
class, title, at + size in global coords), active window, cursor,
and layers: launchers (wofi/rofi), bars, notification popups, and
on-screen keyboards are NOT windows and appear only there, with a
best-effort kind and global geometry you can screenshot by region or
click into. A listed layer is one the compositor tracks, not one you
can see: it may be transparent or dormant, so screenshot when
visibility matters. Call first; act on the addresses it returns.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
| Name | Required | Description |
|---|---|---|
No output parameters | ||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Since no annotations are provided, the description carries the full burden. It reveals important behavioral nuances: layers are not windows and may be invisible, and the snapshot returns geometry for screenshotting or clicking. It also notes that 'a listed layer is one the compositor tracks, not one you can see,' which is critical for correct interpretation. This is rich, non-obvious context beyond what a schema could convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise but information-dense. It front-loads the core purpose ('Semantic desktop snapshot') and then provides necessary details about layers and action points without fluff. Every sentence adds value: the layer clarification, the visibility caveat, and the directive to call first. The structure is logical, progressing from what, to nuances, to usage.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (snapshot of multiple desktop elements) and the absence of annotations, the description provides sufficient detail for correct invocation and interpretation. It explains the distinction between windows and layers, mentions best-effort 'kind' for layers, and advises on screenshot when visibility matters. The output schema likely details the return format, so the description complements it well.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the description adds value by explaining what the output contains (the snapshot details). Even though there's no parameter to document, the description clarifies the structure of the returned data, which is essential for the agent. Since there are no parameters, the description complements the lack of input schema with domain context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that this tool takes a semantic snapshot of the desktop environment, enumerating specific entities (monitors, workspaces, windows, active window, cursor) and their properties (address, class, title, geometry). It distinguishes itself from siblings like 'screenshot' and 'click_ui' by emphasizing the returned addresses and the semantic layer, making its purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit instructions are given on when to use this tool: 'Call first; act on the addresses it returns.' It also provides guidance on handling layers, noting that they may be transparent or dormant, and advises to screenshot when visibility matters, which helps the agent decide between using this snapshot tool versus a screenshot tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Window/workspace ops over IPC (instant, no vision).
action='workspace' (workspace: number/name/'special:name') |
'focus_window' (target: address) | 'move_window' (target + workspace,
silent) | 'close_window' (target) | 'fullscreen' (target?) |
'toggle_floating' (target?). then ('desktop'|'screenshot'|'ui'|'none')
appends the result to this call.
| Name | Required | Description | Default |
|---|---|---|---|
| then | No | none | |
| action | Yes | ||
| target | No | ||
| workspace | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavioral disclosure. It does add useful traits: 'instant, no vision', move_window is 'silent', and `then` 'appends the result to this call'. However, it does not disclose failure behavior, permissions, or irreversible effects of close_window/fullscreen/toggle_floating beyond the action names themselves.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and dense with no filler. Main domain is front-loaded, action alternatives are presented in an unambiguous pipe notation, and the trailing `then` sentence covers chaining semantics in one line.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a multi-action IPC tool with no schema descriptions and no annotations, the description covers actions, parameter formats, and chaining. It doesn't elaborate on address format or failure/error cases, but the presence of an output schema means return-value details are not strictly necessary.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema properties have no descriptions (0% coverage), and the description fully compensates by mapping each action to its parameters: workspace accepts number/name/'special:name', target is an address, and then has valid values 'desktop'|'screenshot'|'ui'|'none'. This makes the bare schema actionable.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening line 'Window/workspace ops over IPC (instant, no vision)' names the resource domain and adds a distinguishing mechanism, and the action list gives concrete verbs: workspace, focus_window, move_window, close_window, fullscreen, toggle_floating. This separates it from siblings like ui/click_ui/screenshot, which rely on vision.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'over IPC (instant, no vision)' clearly sets context for direct window manager operations without visual processing, which implies using this rather than vision-based siblings. It does not explicitly name alternatives or say when not to use it, so it stops short of full when/when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Keyboard to the focused app. action='type' (text, unicode-safe) |
'key' (keys combo: 'ctrl+shift+t', 'esc', 'F5'; aliases
enter/esc/tab/backspace/pgup/pgdn/arrows, else XKB keysyms). Pass
window (an address from desktop) to focus that window first, so
keystrokes land in the intended app rather than whatever currently
holds focus. This drives shortcuts the focused application handles
(ctrl+t, ctrl+l). It does NOT trigger Hyprland's own keybinds
(super+...): those go through use_bind, and workspace/window actions
through hypr. then ('desktop'|'screenshot'|'ui'|'none') appends the
result to this call. allow_auth=true overrides the default refusal to
type into a password field or a system authentication dialog (only when
a human intends that credential entry).
| Name | Required | Description | Default |
|---|---|---|---|
| keys | No | ||
| text | No | ||
| then | No | none | |
| action | Yes | ||
| window | No | ||
| allow_auth | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite no annotations, the description discloses key behaviors: unicode-safe typing, key aliases, window focusing for reliability, that it does not trigger Hyprland binds, and the allow_auth override. This covers safety and operational traits comprehensively.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Despite length, every sentence is essential and packed with information. The description is well-structured, front-loaded with core purpose, uses clear formatting (vertical bars, examples). No redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers all parameters and usage guidelines thoroughly. Does not describe the output schema or error behavior, but given the tool's nature and existence of output schema, this is a minor gap. Overall, it provides sufficient context for effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema coverage, the description compensates fully by explaining each parameter: action values ('type'/'key'), keys format and aliases, text, window, then options, and allow_auth boolean with behavioral implications. Provides examples and valid values.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool sends keystrokes to the focused app, with actions 'type' or 'key', and can target a specific window. It differentiates from sibling tools (use_bind, hypr) for Hyprland-specific actions, establishing a distinct purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly explains when to use this tool (for application shortcuts) and when not to (Hyprland binds, workspace actions), directing to siblings use_bind and hypr. Also notes special case of allow_auth for password fields.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run command via Hyprland exec. Optional workspace placement
(silent, works even for single-instance apps like browsers, whose
window gets moved after it appears) and wait_s (1-30, default 8;
raise for slow apps). Returns the new window's
address/class/title/workspace, or a timeout note.
| Name | Required | Description | Default |
|---|---|---|---|
| wait_s | No | ||
| command | Yes | ||
| workspace | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses behavioral traits like workspace placement behavior for single-instance apps and timeout, but could mention error cases or permissions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no fluff. First sentence states purpose, second explains optional parameters and return value. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given low complexity and presence of an output schema, the description is complete. It summarizes return fields and timeout note, sufficient for the agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but the description adds complete meaning for each parameter: command, workspace placement, wait_s range and default. Fully compensates for missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it runs a command via Hyprland exec, with specific verb and resource. It also distinguishes itself from sibling tools (desktop, hypr, etc.) by focusing on launching commands.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context on when to use, including optional workspace placement and wait_s parameter. However, it does not explicitly state when not to use or mention alternatives, though siblings are distinct.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Set-of-Marks capture: a screenshot of the window WITH its accessible
controls drawn as numbered red marks, plus a JSON legend mapping each
number to the control's role, name, current value, and exact global
click point. One glance replaces the estimate-zoom-estimate loop for
every control the accessibility tree knows: read the number off the
image and call click_ui(mark=N) (or pointer at the legend's x,y).
window is an address from desktop (default: focused); name filters
the marked controls. Falls back to the plain legend when ImageMagick is
not installed, and to a fall-back-to-vision note when the app exposes
no tree (then use screenshot + zoom).
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | ||
| window | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It discloses the capture behavior, reliance on accessibility tree, fallback to plain legend when ImageMagick is absent, and a note when no tree exists. It does not mention destructive actions, rate limits, or authentication, but these are not critical for a read-like capture tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is moderately long but all sentences serve a purpose: core functionality, usage advice, fallback conditions. It is front-loaded with the primary action. Minor redundancy (e.g., 'Falls back...') could be more concise, but overall well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of fallbacks (ImageMagick, no tree) and the presence of an output schema (return values not needing further explanation), the description covers all necessary context: parameters, output (JSON legend), and use case. It references sibling tools appropriately.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description adds meaning: `window` is 'an address from desktop (default: focused)' and `name` 'filters the marked controls'. It does not specify the exact format of the window address, but provides enough context for the agent to use parameters correctly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool captures a screenshot with numbered red marks and a JSON legend mapping each mark to control details. It uses the specific verb 'capture' and resource 'Set-of-Marks', and distinguishes from siblings like 'screenshot' and 'ui' by highlighting marks and legend.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises when to use the tool: to replace the 'estimate-zoom-estimate loop' for controls in the accessibility tree. It provides follow-up actions (using `click_ui(mark=N)` or `pointer`), and specifies fallbacks for missing ImageMagick or no accessibility tree, with an alternative (screenshot + zoom).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Mouse in global coordinates. action='move' (x,y) | 'click' (optional
x,y first; button left/right/middle; double=true) | 'drag' (x,y →
to_x,to_y holding button) | 'scroll' (scroll_dy notches, positive =
content down; optional x,y first). then appends the result to this
call so you skip a round-trip: 'desktop' a fresh snapshot, 'screenshot'
a stable capture, 'ui' the focused window's elements with current
values, 'none' (default) nothing. allow_auth=true overrides the
refusal to click over a system authentication dialog.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | ||
| y | No | ||
| then | No | none | |
| to_x | No | ||
| to_y | No | ||
| action | Yes | ||
| button | No | left | |
| double | No | ||
| scroll_dx | No | ||
| scroll_dy | No | ||
| allow_auth | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It discloses coordinate system, action behaviors, 'then' chaining, and auth override. However, it lacks explicit information on error handling or side effects like destructive actions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is a single dense paragraph that front-loads the core concept. It covers actions and special parameters efficiently. Could be more structured (e.g., list actions), but remains clear and concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has 11 parameters and an output schema (not shown). Description covers key actions and special parameters well. However, it does not fully explain parameter combinations or error handling, leaving some gaps for complex use cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but description explains parameters for each action (e.g., x,y for move, button for click). It adds meaning beyond schema titles, though scroll_dx is omitted. Overall, it compensates well for missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool is for mouse operations in global coordinates, lists all actions (move, click, drag, scroll), and explains special parameters like 'then' and 'allow_auth'. It distinguishes itself from sibling tools like keyboard or desktop.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for mouse input and provides context like coordinate system and action options. It mentions 'allow_auth' for authentication dialogs but does not explicitly contrast with alternative tools or specify when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Capture the focused monitor, a window (window: "active" or an
address from desktop, cheapest for reading one app), or a region
"x,y,WxH". Returns the image (or a file path to read) + JSON metadata
with geometry/scale for pixel→global mapping. scale 0.1-1.0:
optional deliberate downscale, usually leave unset. stable=true
waits (up to 2s) until two consecutive frames match, so a capture
right after an action is not taken mid-animation; metadata gains
stable. Captures are fast JPEG by default; lossless=true returns
PNG for pixel-exact work. Before clicking a small control, follow with
zoom at the estimated point.
| Name | Required | Description | Default |
|---|---|---|---|
| scale | No | ||
| region | No | ||
| stable | No | ||
| window | No | ||
| lossless | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, description fully discloses behavior: default JPEG, lossless PNG option, stable wait up to 2s, scale downscale, and metadata details. Contradicts no annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single paragraph, front-loads main verb and resource. Slightly dense with multiple details but no wasted words. Could be split for clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers input parameters comprehensively, mentions output format (image/path + JSON metadata), and provides usage advice. Output schema exists, so return details are not needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but description explains all 5 parameters: window, region, scale, stable, lossless. Each parameter's purpose and effect are clearly described beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool captures a monitor, window, or region, and specifies output format. It distinguishes from siblings like 'zoom' by mentioning its purpose after screenshot.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides context on when to use 'window' (cheapest for reading one app) and advises to follow with 'zoom' for clicking small controls. Lacks explicit when-not-to-use scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run an ordered list of actions in ONE call, so a click/type/enter
micro-sequence costs one round-trip instead of several. Each step is
{"op": "pointer"|"keyboard"|"click_ui"|"hypr"|"wait_for", ...that tool's
args}, e.g. [{"op":"pointer","action":"click","x":800,"y":60},
{"op":"keyboard","action":"type","text":"hello","window":"0x.."},
{"op":"keyboard","action":"key","keys":"enter"}]. With stop_on_change
(default) the run stops, best-effort, when it notices a STRUCTURAL
change between steps that the step did not intend: a window opening
(e.g. a dialog), closing, or moving, a switch to an unexpected
workspace, or a seat-taking layer surface (a launcher or on-screen
keyboard) appearing, so later steps do not act on stale state.
Notification popups and bars are not treated as changes. It does NOT catch
a bare focus change, so to type into a specific window reliably give
that keyboard step a window= address (it focuses first). Bounded to 20
steps and ~30s total. then observes the final state ('desktop'
default, 'screenshot', 'ui', 'none').
| Name | Required | Description | Default |
|---|---|---|---|
| then | No | desktop | |
| steps | Yes | ||
| stop_on_change | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Since no annotations are provided, the description bears full burden. It thoroughly explains the stop_on_change behavior, detailing what structural changes are detected and what are not (e.g., notification popups, bare focus changes). It also specifies limits (20 steps, 30 seconds) and the `then` parameter's role. This provides excellent transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise yet comprehensive. It front-loads the core purpose, then logically explains behavior, parameters, and constraints. Every sentence adds value without redundancy, making it easy to digest.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity and the presence of an output schema (not shown), the description is largely complete. It covers step structure, stop logic, and the then parameter. Minor omissions like return value details are likely covered by the output schema, keeping completeness high.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so description must compensate. It explains the structure of the steps array (objects with op and args) and the effects of stop_on_change and then. While it does not list every allowed op or argument detail, it references sibling tool args, which adds significant value beyond the raw schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states the tool's purpose: 'Run an ordered list of actions in ONE call' with a concrete example of a micro-sequence. It clearly distinguishes from sibling tools like pointer, keyboard, etc. by combining multiple actions into a single round-trip.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description indicates when to use (multi-step sequences) and contrasts with individual tool calls. However, it does not explicitly state when NOT to use or provide direct comparisons to alternatives, though the context implies that single actions should use the respective individual tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Read a window's accessibility tree (AT-SPI) and return its elements
with GLOBAL click points, so you can target a control by NAME with no
screenshot and no pixel guessing. window is an address from desktop
(default: the focused window). name filters to elements whose
accessible name contains it (case-insensitive); actionable (default)
keeps only interactive roles (buttons, entries, menu items, ...).
Returns [{role, name, x, y, clickable}] where x,y is the click point:
focus the window, then click it with pointer (the window must be
visible to receive the click), or do both in one call with click_ui.
Controls that carry a CURRENT VALUE also
report it: value (text typed into an entry, or a slider/spinner
number), percent for a slider's position, checked for a box or
toggle. Password fields never report contents, and many dropdowns
expose no value at all, so read the screen with screenshot when a
rendered value matters. Not every app exposes a tree (terminals, and
Electron/Chrome without --force-renderer-accessibility, expose little
or nothing); when it does not, fall back to screenshot + zoom.
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | ||
| window | No | ||
| actionable | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses important behaviors: the tree may not be available for certain apps, password fields never report contents, dropdowns may lack value, and click points are global requiring window focus. It also mentions that the tool returns elements with roles, names, and click points. The description does not contradict annotations (none provided).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is detailed and well-structured, front-loading the main purpose. Each sentence adds value, though it is slightly long. There is no redundancy, and the technical details are clearly presented. It could be slightly more concise, but overall it is effective.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (accessibility tree with varying behavior), the description is remarkably complete. It explains limitations, fallback strategies, return format, and exceptions. It covers all aspects needed for correct invocation, including implicit output schema description. No output schema is provided, but the description compensates with clear return structure.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so description must compensate. It thoroughly explains all three parameters: 'window' (address from desktop, default focused), 'name' (filters by accessible name, case-insensitive), and 'actionable' (default true, keeps interactive roles). It also describes the return structure, adding significant value beyond the input schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it reads the accessibility tree and returns elements with click points, distinguishing it from sibling tools like screenshot (visual fallback) and pointer (clicking). It specifies the action 'Read a window's accessibility tree' and the resource 'elements...with GLOBAL click points'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear guidance on when to use the tool (to target a control by name without screenshot or pixel guessing), and when to fall back (when the tree is not exposed, like in terminals or Electron/Chrome). It also mentions limitations (password fields, dropdowns) and suggests alternatives (click_ui, screenshot). However, it does not explicitly state when NOT to use this tool versus specific siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run one of the user's own Hyprland keybinds by its combo (from the
binds tool), e.g. 'SUPER+F'. This executes the bound action directly
(the only reliable way: synthetic keypresses do not trigger compositor
binds). Use it to drive the owner's configured workflows: launchers,
layout shortcuts, scratchpads. then ('desktop'|'screenshot'|'ui'|'none')
appends the result to this call (handy after a launcher bind). Refused
while HYPRUSE_CONFINE is set: a bind runs an arbitrary compositor action
that cannot be scoped to a window.
| Name | Required | Description | Default |
|---|---|---|---|
| then | No | none | |
| combo | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses that the tool executes an arbitrary compositor action, that it is refused under HYPRUSE_CONFINE, and that synthetic keypresses do not trigger compositor binds. It does not detail side effects or reversibility, but for a tool that runs user-defined binds, the key behavioral constraints are clearly stated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the core action and example. Every sentence adds value: the reliability rationale, use cases, 'then' semantics, and the confinement refusal. It is slightly dense, but not bloated.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (running arbitrary user keybinds) and the absence of annotations, the description covers the essential context: what it does, why it's needed, when it's refused, and how 'then' works. It doesn't describe the output schema, but an output schema exists, so that burden is partially lifted. Minor gaps remain around error behavior if the combo is invalid.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It explains 'combo' with an example and clarifies that it comes from the 'binds' tool. It also explains the 'then' parameter's purpose and its allowed values ('desktop'|'screenshot'|'ui'|'none'). This is strong compensation, though it could be slightly more explicit about the exact format of 'then' values.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Run'), a specific resource ('one of the user's own Hyprland keybinds by its combo'), and gives a concrete example ('SUPER+F'). It also distinguishes itself from synthetic keypresses and from the sibling 'binds' tool, making its purpose unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says when to use it ('to drive the owner's configured workflows: launchers, layout shortcuts, scratchpads') and when not to use it ('Refused while HYPRUSE_CONFINE is set'). It also explains why it is the only reliable way, which helps an agent choose it over alternatives like synthetic keypresses or the 'binds' tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Block until a desktop event happens (real compositor events, not
polling). event: 'window_open' | 'window_close' | 'workspace' |
'title_change' | 'layer_open' | 'layer_close' (layer-shell surfaces:
launchers, notification popups; match on the namespace, e.g. 'wofi') |
'urgent' (a window demands attention) | 'screencast' (screen sharing
started/stopped). match: optional case-insensitive substring filter
over the event's fields (class/title/workspace name/address/namespace).
timeout_s 1-60, default 10. Returns the event payload, or a timeout
note; a filtered wait whose condition ALREADY holds (the window is
already gone, the workspace already active, the layer already mapped)
returns instantly with already: true instead of missing an event
that fired before it could subscribe. Use it after actions with
delayed effects: app startups, page loads that change a window title,
a launcher bind that pops a layer.
| Name | Required | Description | Default |
|---|---|---|---|
| event | Yes | ||
| match | No | ||
| timeout_s | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description fully carries the burden of behavioral disclosure. It details that it blocks on real compositor events (not polling), returns the event payload or a timeout note, and that a filtered wait whose condition already holds returns instantly with `already: true`. It also explains that matching is case-insensitive substring filtering. This is thorough behavioral transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is lengthy but every sentence adds value. It is front-loaded with the core purpose ('Block until a desktop event happens, not polling'), then details event types, matching, timeout, return behavior, and usage examples. There is no fluff; each clause conveys essential information. The structure is logical and efficient for the complexity of the tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of the tool (multiple event types, matching logic, timeout semantics, edge case for conditions already holding), the description covers all aspects necessary for an agent to use it correctly. It includes examples of when to use it and explains the behavior for every scenario. The presence of an output schema means return-value details are not required in the description, but it still mentions what the tool returns. This is fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description must compensate. It does so comprehensively: the `event` parameter is fully enumerated with descriptions for each event type (e.g., 'layer-shell surfaces: launchers, notification popups; match on the namespace, e.g. wofi'), `match` is explained as an optional case-insensitive substring filter over the event's fields, and `timeout_s` is given a range (1-60) and default (10). This fully compensates for the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the primary purpose: 'Block until a desktop event happens', and then enumerates the specific event types supported. It distinguishes this tool from its siblings (e.g., launch, use_bind, screenshot) by focusing on waiting for events rather than performing actions. The agent can immediately understand what the tool does and how it differs from the other tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit guidance on when to use the tool: 'Use it after actions with delayed effects: app startups, page loads that change a window title, a launcher bind that pops a layer.' It also explains the behavior when the condition already holds, which helps the agent anticipate edge cases. This is clear and actionable usage guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Native-resolution re-capture around a point: the precision step of
the coarse-to-fine loop. Screenshot first, estimate the target's global
x,y, zoom there, re-estimate on the zoomed image (scale ~1.0, so
global = geometry[:2] + image_pixel), then click. size "WxH" in
logical pixels (default 480x360) is clamped to the screen; window
(an address from desktop) clamps to that window instead. The metadata
echoes the requested point back as point; stable=true waits for
the frame to settle first; lossless=true returns PNG.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| size | No | ||
| stable | No | ||
| window | No | ||
| lossless | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
In the absence of annotations, the description does a good job disclosing key behaviors: size clamping, window constraint, stable waiting, lossless PNG, and coordinate transformation. However, it omits potential side effects or requirements like permissions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is somewhat dense and technical, mixing procedural steps with parameter details. It is front-loaded with purpose but could be more structured (e.g., bullet points) for clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the availability of an output schema, the description covers workflow and parameter behavior well. It lacks error handling or prerequisites but is adequate for a visual interaction tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema coverage, the description adds significant meaning to all parameters: explains 'size' format and clamping, 'window' as address, 'stable' for frame settling, and 'lossless' for format. x/y are implied but not explicitly described, leaving minor ambiguity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's role as 'Native-resolution re-capture around a point: the precision step of the coarse-to-fine loop.' It explains the workflow (screenshot, estimate, zoom, re-estimate, click) and distinguishes from sibling tools like 'screenshot' by focusing on refinement.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage within a coarse-to-fine loop but does not explicitly state when to use this tool versus alternatives like 'screenshot' or 'desktop'. No 'when-not-to-use' guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.11.0click_uihyprlaunchuse_bindwait_forv0.9.4desktophyprkeyboard1 field changedInput schema / properties / allow_authAdded value: +{
+ "default": false,
+ "title": "Allow Auth",
+ "type": "boolean"
+}launchmarkspointerscreenshotuiuse_bindwait_forzoomv0.6.0pointerscreenshotsequencezoomv0.5.0desktophypr3 fields changedInput schema / properties / thenAdded value: +{
+ "default": "none",
+ "title": "Then",
+ "type": "string"
+}Output schema / properties / result / anyOfAdded value: +[
+ {
+ "items": {},
+ "type": "array"
+ },
+ {
+ "type": "string"
+ }
+]Output schema / properties / result / typeRemoved value: -"string"keyboard4 fields changedInput schema / properties / thenAdded value: +{
+ "default": "none",
+ "title": "Then",
+ "type": "string"
+}Input schema / properties / windowAdded value: +{
+ "default": "",
+ "title": "Window",
+ "type": "string"
+}Output schema / properties / result / anyOfAdded value: +[
+ {
+ "items": {},
+ "type": "array"
+ },
+ {
+ "type": "string"
+ }
+]Output schema / properties / result / typeRemoved value: -"string"pointer3 fields changedInput schema / properties / thenAdded value: +{
+ "default": "none",
+ "title": "Then",
+ "type": "string"
+}Output schema / properties / result / anyOfAdded value: +[
+ {
+ "items": {},
+ "type": "array"
+ },
+ {
+ "type": "string"
+ }
+]Output schema / properties / result / typeRemoved value: -"string"screenshot1 field changedInput schema / properties / losslessAdded value: +{
+ "default": false,
+ "title": "Lossless",
+ "type": "boolean"
+}use_bind3 fields changedInput schema / properties / thenAdded value: +{
+ "default": "none",
+ "title": "Then",
+ "type": "string"
+}Output schema / properties / result / anyOfAdded value: +[
+ {
+ "items": {},
+ "type": "array"
+ },
+ {
+ "type": "string"
+ }
+]Output schema / properties / result / typeRemoved value: -"string"zoom1 field changedInput schema / properties / losslessAdded value: +{
+ "default": false,
+ "title": "Lossless",
+ "type": "boolean"
+}v0.4.1screenshot1 field changedInput schema / properties / stableAdded value: +{
+ "default": false,
+ "title": "Stable",
+ "type": "boolean"
+}zoomv0.2.1bindsuse_bindwait_forv0.1.1desktophyprkeyboardlaunchpointerscreenshotScored across 14 tools
Each tool serves a clearly distinct function: desktop provides semantic state, screenshot/zoom capture visual data, marks/ui expose accessibility info, keyboard/pointer/click_ui handle input, hypr/launch/use_bind execute actions, wait_for listens for events, and sequence orchestrates multi-step workflows. No two tools have overlapping purposes that could cause misselection; even similar tools like marks and ui differ in output format and use case.
All names are lowercase and descriptive, with compound names consistently using underscores (wait_for, click_ui, use_bind). However, the set mixes nouns (desktop, marks, pointer, screenshot) with verbs (launch, wait_for, click_ui), so there's no strict verb_noun pattern, but the inconsistency is minor and doesn't hinder understanding.
14 tools is well-scoped for a desktop automation server, covering state inspection, input, action execution, event waiting, and sequencing. Each tool is necessary and non-redundant, and the count is within the ideal 3-15 range.
The tool surface is comprehensive for desktop automation: it provides state discovery (desktop, ui, binds), input (keyboard, pointer, click_ui), action execution (hypr, launch, use_bind), observation (screenshot, zoom, marks), event handling (wait_for), and multi-step orchestration (sequence). No obvious gaps exist for typical workflows; even edge cases like authentication dialogs are addressed via allow_auth flags.
Official MCP server for Agentwork — delegate tasks to AI agents with human-in-the-loop
MCP server for AI agents to plan, verify, and deploy Cloudflare-native apps.
MCP server for building and testing AI agents with multi-model experimentation and insights.
A comprehensive Model Context Protocol (MCP) server that enables AI assistants to interact with yo…
