hypruse
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| HYPRUSE_MAX_IMAGE_BYTES | No | Maximum image size in bytes (e.g., 1048576 for 1 MB). Screenshots will degrade resolution to fit. | 1048576 |
| HYPRUSE_SCREENSHOT_MODE | No | Screenshot output mode: 'path' (default) writes PNG file and returns path, 'image' returns inline image data. | path |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
hypruse controls a live Hyprland desktop. Workflow: call desktop first
and prefer hypr/launch (IPC, instant and exact) for anything window-
or workspace-shaped; use screenshot + pointer/keyboard only to see
and operate inside application windows. Launchers, bars, notification
popups, and on-screen keyboards are not windows: desktop reports them
under layers. A listed layer is one the compositor tracks, not one you
can see: it may be transparent or dormant, so screenshot when visibility
matters. To CLICK a named control inside an app, call click_ui FIRST:
for GTK/Qt apps it resolves the control in the accessibility tree and
clicks it in ONE call, no image and no pixel estimation (ui lists the
controls and their current values without clicking; marks returns a
numbered screenshot plus legend when you want to SEE what is clickable,
then click_ui(mark=N)). Use screenshot + zoom when the tree exposes
nothing (terminals, canvas UIs, Electron/Chrome without a flag). To
verify an effect without a second round-trip, pass then='desktop' (a
fresh snapshot), then='ui' (controls with current values, cheapest
after typing or toggling: the focused window, or for click_ui the
window it just clicked), or then='screenshot' to
the acting call itself instead of calling desktop again. binds lists
the owner's own keybinds; to run one, call
use_bind with its combo (synthetic keypresses do NOT trigger compositor
binds, so keyboard is only for shortcuts the focused app handles). After
actions with delayed effects, block on wait_for (window_open,
title_change, layer_open) instead of sleeping. To run several actions at
once (click, type, press enter, wait), pass them to sequence as one
call: it stops if
the desktop changes structurally between steps, so you spend one
round-trip instead of one per step. It does not catch a bare focus change,
so give a keyboard step a window= address when typing matters.
Coordinates: one space everywhere, Hyprland global logical pixels (window
at, cursor, clicks). Screenshots are pixel space: map back with
global = geometry[:2] + image_pixel / scale, using the metadata's scale
and image [w,h] (they already account for any downscaling). When
screenshot returns a file path instead of an image, read that file.
Clicking precisely is hard: estimating a target's pixel from a full-screen
image is only accurate to within tens of pixels, which misses small
controls. For anything small, work coarse-to-fine: screenshot the window,
estimate the target's global point, then call zoom at that point and
re-estimate on the zoomed image before clicking. Zoomed captures come back
near 1:1 (scale ~1.0) with the target large and their origin in geometry,
so global = geometry[:2] + image_pixel lands cleanly. Estimate by
proportion (e.g. "60% across a 300px-wide crop → x≈180"), not absolute
guessing, and after a click that should change something, screenshot again
to confirm before continuing (stable=true waits out animations).
The cursor and keyboard focus are SHARED with the human at the desk:
finish what you start, and expect every action on their screen to be
visible. Before typing, pass keyboard(window=<address>) so keystrokes
never land in the wrong window.
Background mode is automatic. An action on the window the human is in
runs on their screen, as above. An action on any other window runs in the
background, and the human's cursor, keyboard focus and workspace stay
where they are: click_ui presses the control through its accessibility
action, keyboard(window=...) types through the accessibility interface
or sends the keys to that window alone, and launch with no workspace
opens the app on one of the agent's own workspaces (workspaces 99 down to 90), out of
the human's sight. Each result says which method it used and whether the
human's focus stayed put; a check box, slider or field is read back, and
a WARNING means the human's focus moved during it, so stop and say so. Check
a button press or keys with ui. The pointer cannot work in the background,
so it is refused over any window but the human's: press controls there
with click_ui. A hidden window's screenshot can be stale (many apps stop
drawing while hidden), so read its state with ui. When something cannot
be done in the background, the refusal says why. background=false does
it on the human's screen instead and takes it over, so pass that only
when the human expects it.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| desktopA | Semantic desktop snapshot: monitors, workspaces, windows (address,
class, title, |
| screenshotA | Capture the focused monitor, a window ( |
| zoomA | Native-resolution re-capture around a point: the precision step of
the coarse-to-fine loop. Screenshot first, estimate the target's global
x,y, zoom there, re-estimate on the zoomed image (scale ~1.0, so
global = geometry[:2] + image_pixel), then click. |
| uiA | Read a window's accessibility tree (AT-SPI) and return its elements
with GLOBAL click points, so you can target a control by NAME with no
screenshot and no pixel guessing. |
| marksA | Set-of-Marks capture: a screenshot of the window WITH its accessible
controls drawn as numbered red marks, plus a JSON legend mapping each
number to the control's role, name, current value, and exact global
click point. One glance replaces the estimate-zoom-estimate loop for
every control the accessibility tree knows: read the number off the
image and call |
| bindsA | The user's own Hyprland keybinds: combo, action, arg, and a
description when the config provides one. This is how the desktop's
owner drives it: to perform one of these workflows, call |
| wait_forA | Block until a desktop event happens (real compositor events, not
polling). event: 'window_open' | 'window_close' | 'workspace' |
'title_change' | 'layer_open' | 'layer_close' (layer-shell surfaces:
launchers, notification popups; match on the namespace, e.g. 'wofi') |
'urgent' (a window demands attention) | 'screencast' (screen sharing
started/stopped). match: optional case-insensitive substring filter
over the event's fields (class/title/workspace name/address/namespace).
timeout_s 1-60, default 10. Returns the event payload, or a timeout
note; a filtered wait whose condition ALREADY holds (the window is
already gone, the workspace already active, the layer already mapped)
returns instantly with |
| pointerA | Mouse in global coordinates. action='move' (x,y) | 'click' (optional
x,y first; button left/right/middle; double=true) | 'drag' (x,y →
to_x,to_y holding button) | 'scroll' (scroll_dy notches, positive =
content down; optional x,y first). |
| keyboardA | Keyboard to an app. action='type' (text, unicode-safe) |
'key' (keys combo: 'ctrl+shift+t', 'esc', 'F5'; aliases
enter/esc/tab/backspace/pgup/pgdn/arrows, else XKB keysyms). Pass
|
| click_uiA | Click a control by its accessible NAME, or by a |
| hyprA | Window/workspace ops over IPC (instant, no vision).
action='workspace' (workspace: number/name/'special:name') |
'focus_window' (target: address) | 'move_window' (target + workspace,
silent) | 'close_window' (target) | 'fullscreen' (target?) |
'toggle_floating' (target?). |
| launchA | Open an app: run |
| use_bindA | Run one of the user's own Hyprland keybinds by its combo (from the
|
| sequenceA | Run an ordered list of actions in ONE call, so a click/type/enter
micro-sequence costs one round-trip instead of several. Each step is
{"op": "pointer"|"keyboard"|"click_ui"|"hypr"|"wait_for", ...that tool's
args}, e.g. [{"op":"pointer","action":"click","x":800,"y":60},
{"op":"keyboard","action":"type","text":"hello","window":"0x.."},
{"op":"keyboard","action":"key","keys":"enter"}]. With stop_on_change
(default) the run stops, best-effort, when it notices a STRUCTURAL
change between steps that the step did not intend: a window opening
(e.g. a dialog), closing, or moving, a switch to an unexpected
workspace, or a seat-taking layer surface (a launcher or on-screen
keyboard) appearing, so later steps do not act on stale state.
Notification popups and bars are not treated as changes. It does NOT catch
a bare focus change, so to type into a specific window reliably give
that keyboard step a window= address (the keys go to that window).
Bounded to 20
steps and ~30s total. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 14 tools
Each tool has a crisply distinct role, and the descriptions actively delineate boundaries (pointer vs click_ui, keyboard vs use_bind vs hypr, screenshot vs zoom vs marks vs ui). Overlapping-seeming tools like screenshot/zoom/marks and ui/click_ui are separated by explicit coarse-to-fine vs accessibility vs annotated-capture semantics. An agent can reliably pick the right tool.
All names are lowercase single tokens with underscore used consistently for multi-word tools (wait_for, click_ui, use_bind), so formatting is uniform. The only wobble is semantic: some names are nouns (desktop, marks, binds, pointer) and others verbs (launch, zoom), which is a minor deviation from a single predictable pattern.
14 tools is well within the ideal 3-15 range and each earns its place: perception (desktop, screenshot, zoom, ui, marks), input (pointer, keyboard, click_ui), orchestration (wait_for, sequence), and system control (hypr, launch, binds, use_bind) with no filler.
The desktop-automation surface is essentially full: snapshot, capture at two fidelities, accessibility read/annotate, mouse, keyboard, background-window targeting, event waiting, window/workspace IPC, app launch, keybind discovery and replay, plus a batch runner. It even documents fallback paths (vision when no a11y tree) and hard limits (locked session, auth dialogs), leaving no obvious dead ends.