hypruse

Computer use for Hyprland. An MCP server that gives AI agents native hands on your Wayland desktop: workspaces, windows, mouse, keyboard, screenshots.

No ydotool daemon. No root. No portals. No X11.

Why

Computer use exists on macOS and Windows. On Linux there is effectively nothing: the Claude Desktop Linux beta explicitly ships without screen control, Anthropic's reference implementation is an X11 container, and the existing Wayland attempts lean on setuid uinput hacks or GNOME-only portals.

Meanwhile Hyprland already exposes everything an agent needs, better than any accessibility bridge: a complete IPC surface for state and window management, and first-class Wayland protocols for input. hypruse just wires them to MCP:

• Semantic first. desktop returns the real window/workspace tree (addresses, classes, titles, geometry) in one call. The agent switches workspaces and focuses windows the way you do (instantly, over IPC), not by squinting at pixels.

• Vision when it matters. Screenshots of a monitor, an exact window crop, or a zoomed region, with the geometry/scale metadata to map any pixel back to a clickable coordinate: a coarse-to-fine loop grounded in the GUI-agents research.

• Native input. Clicks and scrolls are spoken directly over the Wayland wire (zwlr_virtual_pointer_v1); typing goes through wtype's virtual keyboard with a proper XKB keymap, unicode-safe on any layout.

Related MCP server: hyprland-mcp

How it works

agent (Claude Code, or any MCP client)
   │ stdio
   ▼
hypruse
   ├── hyprctl -j ········▶ desktop state: monitors, workspaces, windows, layers
   ├── hyprctl dispatch ··▶ focus / move / close / launch / movecursor
   ├── grim ··············▶ screenshots: monitor, window crop, region
   ├── busctl (AT-SPI) ···▶ accessibility tree: named controls, current
   │                        values, exact coords (ui / marks / click_ui)
   ├── wtype ·············▶ keyboard (zwp_virtual_keyboard_v1, real XKB keymap)
   └── raw Wayland wire ··▶ click & scroll (zwlr_virtual_pointer_v1)

Optional binaries gate two more tools: imagemagick draws the numbered overlay for marks, and wl-clipboard backs the opt-in clipboard tool.

Design decisions:

• No ydotool / uinput. That path needs a daemon, udev rules or root, and types US scancodes that break on other layouts. hypruse is just another Wayland client of your compositor, same standing as wlrctl.

• No portals. xdg-desktop-portal-hyprland does not implement the RemoteDesktop portal (InputCapture is capture, not injection), so anything built on libei/portals silently degrades on Hyprland. hypruse doesn't try.

• Cursor positioning through the compositor's own cursor dispatcher (global logical coordinates, exact on any monitor layout), with only button/axis events on the virtual pointer, sidestepping the known multi-monitor bugs of absolute virtual-pointer motion (hyprwm/Hyprland#6749).

Tools

The acting tools (pointer, keyboard, click_ui, hypr, use_bind, sequence) take an optional then argument that appends the result to the same call, so the agent sees the effect without a second round-trip: then='desktop' adds a fresh semantic snapshot (~20 ms, cheap, best for window/focus changes), then='screenshot' a stable capture (best for visual changes), then='ui' the acted-on window's controls with their current values (a few hundred exact tokens, best after typing or toggling; click_ui reads the window it clicked even if the click handed focus to a dialog, the others read the focused window), then='none' nothing (the default everywhere except sequence, which defaults to 'desktop').

Features

The tools group into five capabilities, ordered most-reliable-and-cheapest first. An agent that reaches for them in this order is both faster and more accurate, and many tasks never need a screenshot at all.

1. Semantic desktop control (start here)

desktop returns the entire window and workspace tree in one call: every window's address, class, title, and geometry, the active window, the cursor, and any layer surfaces the compositor is tracking (launchers, bars, notification popups). hypr and launch then act on it over IPC in milliseconds: switch workspace, focus/move/close/fullscreen/float a window by address, or start an app.

Use it well: never take a screenshot to find or arrange windows. Read desktop, act on the address you want. launch blocks on the real openwindow event and hands back the new window's address, so there is nothing to poll or guess; it also relocates single-instance apps (browsers) that ignore workspace rules.

2. Click controls by name, no pixels

When an app exposes an accessibility tree (most GTK and Qt apps), you can target controls by name instead of by pixel. ui lists every control with its exact global coordinate, and reports the current value of the controls that carry one: the text in a field, a slider's percentage, a checkbox's state. click_ui resolves a name and clicks it in one call, through the real cursor (so the beacon and every safety guarantee still apply). marks draws numbered marks over a screenshot with a legend, for when you want to see the options first and then click_ui(mark=N).

Use it well: reach for click_ui name="Save" before estimating any pixel, since it is exact and spends no image. Read a form's state with ui (did the box actually tick?) instead of screenshotting it. An ambiguous name returns the candidates rather than guessing. When an app exposes no tree (terminals, canvas apps, Electron/Chrome without --force-renderer-accessibility) the tool says so, and you fall back to vision.

3. Vision when it matters: the zoom loop

For everything the accessibility tree cannot name, screenshot (monitor, window crop, or region) and zoom (a native-resolution re-capture around a point) carry a strict coordinate contract, global = geometry + pixel / scale, that stays exact on every monitor and fractional scale.

Use it well: don't guess a small control from a full-screen image. Work coarse-to-fine: screenshot the window, estimate the target, zoom there, re-estimate on the sharp crop, then click. This two-step loop is the research-backed way to hit small targets.

4. Fewer round-trips: the latency lever

For an agent the model calls dominate task latency, not the desktop, so the real speedups are structural. sequence runs an ordered micro-plan (click, type, press enter, wait) in a single call, stopping the moment the desktop changes structurally in a way a step did not intend (a window opening, closing, or moving, an unexpected workspace switch, or a seat-taking launcher or on-screen keyboard; it deliberately ignores bare focus changes and notification popups). then='desktop' | 'screenshot' | 'ui' fuses a fresh view of the result into the acting call itself. wait_for blocks on real compositor events (a window or launcher opening, a title changing, a workspace switch, an urgency hint, screen-sharing starting) instead of sleeping and hoping.

Use it well: collapse a known click/type/enter flow into one sequence. After typing into a form, add then='ui' to read the effect back in a few hundred exact tokens instead of a screenshot. After a launch or a shortcut that opens something, wait_for the event rather than sleeping.

5. Safe delegation: trust layers

hypruse hands an agent your real seat, so it ships the controls to bound what that agent can do. Beyond the always-on approval prompts and the Waybar activity beacon, opt-in env flags narrow what an agent can touch: HYPRUSE_READONLY exposes only the observation tools; HYPRUSE_CONFINE restricts input to the windows the agent launched, or a class/workspace allowlist; HYPRUSE_AUTH_GUARD (on by default) refuses to drive authentication dialogs; HYPRUSE_STRICT refuses to act if you took the seat back; HYPRUSE_MARK tags agent-owned windows and announces when the agent opens a window or captures the screen. Two more record rather than restrict: HYPRUSE_JOURNAL writes an auditable NDJSON line per tool call, refusals included, and HYPRUSE_DRYRUN runs every check and delivers nothing, so you can watch an agent plan the work before it touches your desktop.

Use it well: run read-only for the first week. When you trust a workflow, allowlist its tools and, if you want to walk away, confine the agent to a scope so your password manager on another workspace stays untouchable. Keep a panic bind handy (hypruse stop, or pkill -f hypruse). The Security model has the full story.

Install

Requirements: Hyprland (both config managers: hyprland.conf and the Lua hyprland.lua that 0.56 introduced), grim, wtype (most Hyprland setups already have both), and uv. The accessibility tools (ui/marks/click_ui) use busctl, which ships with systemd. Optional: wl-clipboard for the opt-in clipboard tool, imagemagick for numbered marks captures.

Arch Linux, from the AUR:

yay -S hypruse        # or hypruse-git for main

Then let it set itself up and verify the environment:

hypruse init     # detects your MCP clients, registers (asks first), runs doctor
hypruse doctor   # just the diagnostics

Manual registration, Claude Code:

claude mcp add -s user hypruse -- uvx hypruse

From a source checkout:

claude mcp add -s user hypruse -- uv run --directory /path/to/hypruse hypruse

Any other MCP client: run uvx hypruse as a stdio server. hypruse is also in the official MCP registry as io.github.IlyasKhallouki/hypruse, so clients that browse the registry can install it from there.

Read-only mode: set HYPRUSE_READONLY=1 in the server config to expose only the observation tools (desktop, screenshot, zoom, ui, marks, binds, wait_for). The agent can see and narrate but cannot click, type, or launch. A good first week.

Claude Desktop (Linux beta)

The Linux beta ships without Anthropic's first-party computer use, but stdio MCP servers work in chat, which makes hypruse the workaround. In ~/.config/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "hypruse": {
      "command": "uvx",
      "args": ["hypruse"],
      "env": { "HYPRUSE_SCREENSHOT_MODE": "image" }
    }
  }
}

Two Desktop-specific notes: use image mode (Desktop renders inline MCP images and has no file-read tool), and the app must run natively inside your Hyprland session so the server inherits WAYLAND_DISPLAY/HYPRLAND_INSTANCE_SIGNATURE; from a VM or container it cannot reach your compositor. If your Desktop install bypasses tool-approval prompts, treat the Waybar indicator + panic keybind as mandatory, not optional.

Security model

Read this section before installing. hypruse hands an agent your mouse, your keyboard, your screen contents, and an app launcher. The layers that keep that sane:

1. Approval: MCP clients gate tool calls. In Claude Code, allowlist the read-only tools (desktop, screenshot) and leave pointer/keyboard/hypr/launch on ask-first until you trust a workflow.

2. Visibility: the server maintains an activity beacon ($XDG_RUNTIME_DIR/hypruse/state.json); the shipped Waybar module is invisible when idle and shows a robot indicator while an agent has hands on your desktop.

3. Interruption: click the indicator, or bind a panic key. The portable form works for every install: bind = SUPER SHIFT, BackSpace, exec, pkill -f hypruse. If hypruse is on your PATH (the AUR or a pipx install), bind = SUPER SHIFT, BackSpace, exec, hypruse stop is nicer: it signals the server to shut down gracefully, releasing any held pointer button and clearing the beacon. For a uvx install use exec, uvx hypruse stop; for a source checkout, exec, uv run --directory /path/to/hypruse hypruse stop. Killing it mid-action is safe either way: button press/release pairs never span tool calls, and even a long drag's held button is released on the way out.

4. The seat is shared. There is one cursor and one keyboard focus, and Hyprland's focus-follows-mouse means a cursor move alone can retarget keystrokes. Don't type while an agent is driving; watch the indicator.

5. Scope: stdio only (no network listener), nothing persisted except the beacon and the capped screenshot cache in $XDG_RUNTIME_DIR (tmpfs, newest 20) and, if you turn it on, the action journal on disk under $XDG_STATE_HOME (rotated, one generation, and text-redacted by default). No clipboard access unless you opt in: HYPRUSE_CLIPBOARD=1 registers a clipboard tool (never in read-only mode); clipboards hold passwords, so leave it off unless a workflow needs it. A screenshot sees everything visible: treat an agent session like screen sharing.

6. What the agent reads is untrusted. Window titles, accessibility names and values, and clipboard text flow verbatim into the agent's context, and any web page, filename, or document can put instructions there (prompt injection). hypruse cannot sanitize meaning, so the approval layer is the backstop: keep consequential tools (launch, keyboard, clipboard) on ask-first when the agent will look at untrusted windows, and treat "the screen told me to" as attacker input when reviewing an approval prompt.

7. Input never lands where it silently would not work. Three always-on checks (no env flag) refuse or annotate rather than report a phantom success: a click aimed under a launcher or on-screen keyboard layer surface (which sits above windows and would swallow it), typing while a launcher holds the keyboard grab, and any input while the session is locked (a live hyprlock/swaylock process, which is an ext-session-lock client invisible to the window and layer lists). While locked, keyboard/click_ui/pointer refuse unless allow_auth=true says a human wants the agent driving the unlock prompt. These are truthfulness aids, not a sandbox: they fail open on an unreadable system state, so they harden the common case without being a boundary you can lean on.

Optional confinement

Six opt-in env flags. The first four narrow what an agent can touch, and each fails toward less action; the last two record and rehearse rather than restrict. All compose with the layers above:

• HYPRUSE_CONFINE restricts input to a scope of windows: launched (only windows hypruse itself opened this session), class:firefox,kitty, or workspace:3,special:notes. Keyboard, click_ui, and hypr window ops are refused outside the scope; a pointer click is refused when any window under the point is out of scope (Hyprland's window list is not z-ordered, so hypruse fails closed rather than guess which window is on top). This is what lets you leave an agent working while your password manager sits on another workspace, untouchable. use_bind is refused outright while confinement is set, because a keybind runs an arbitrary compositor action that cannot be scoped to a window.

• HYPRUSE_AUTH_GUARD (default on) refuses to click or type into a system authentication dialog (polkit agents, the GNOME keyring prompt), so a manipulated agent cannot approve a privilege escalation. Set HYPRUSE_AUTH_GUARD=strict to also refuse typing into a password field inside an ordinary window (a browser login), detected via the accessibility tree. A per-call allow_auth=true on pointer/keyboard/click_ui overrides it, and because it changes the tool's arguments the override surfaces distinctly in the approval prompt. HYPRUSE_AUTH_GUARD=0 disables it.

• HYPRUSE_STRICT refuses to act when the cursor or focused window moved since hypruse's last action (the human, or a popup, took the seat): the agent must re-read desktop/screenshot and retry, so it never types into a window you just switched to.

• HYPRUSE_MARK makes the agent's presence legible on the desktop: it tags every window the agent opens hypruse-owned and flashes an on-screen notice when the agent opens a window or captures the screen. It also installs a border_color window rule on that tag so owned windows get a colored outline, but whether a runtime rule renders depends on your Hyprland version and config precedence (on some setups it does not take effect); when it cannot be installed at all, hypruse says so on stderr rather than leaving you with a marking layer that is quietly not running. For a guaranteed outline, add the rule to your own config, which hypruse's tagging then matches: windowrule = border_color rgb(ff5555), tag hypruse-owned in hyprland.conf (older Hyprland: tag:hypruse-owned), or hl.window_rule({ match = { tag = "hypruse-owned" }, border_color = "rgb(ff5555)" }) in hyprland.lua.

• HYPRUSE_JOURNAL records what the agent did: one NDJSON line per tool call in $XDG_STATE_HOME/hypruse/journal.ndjson (HYPRUSE_JOURNAL=1), or a path of your own. Read it with hypruse journal, re-run it with hypruse replay. See The record below.

• HYPRUSE_DRYRUN is a rehearsal: every argument check and every guard above runs, then the call reports what it would have done and delivers nothing.

The record: journal, dry run, replay

The flags above decide what an agent may do in the moment and then forget it happened. HYPRUSE_JOURNAL is the memory. One JSON object per line, so tail -f, grep, and jq all work on a live file:

{"v":1,"seq":7,"ts":"2026-08-02T09:12:13.456Z","kind":"act","tool":"pointer",
 "args":{"action":"click","x":800,"y":60},"outcome":"ok","ms":37,"result":"click ok"}

kind splits the two questions people actually ask: act is input delivered to your desktop, observe is the agent looking, which is what a privacy audit wants (when the screen was captured, when the clipboard was read). Observation results are never recorded, only that they happened, so the journal never becomes a second copy of everything the agent saw. Refusals are recorded too, with the guard's own message: it is the only place the history of your trust layers doing their job exists.

Typed and copied text is recorded as a length plus a short digest, not as text, because keystrokes are passwords. HYPRUSE_JOURNAL_TEXT=1 keeps it verbatim, which you need only to replay typing. Note what a digest is and is not: it proves two entries typed the same thing and it will not hand a reader your password, but it is an unsalted SHA-256 prefix next to an exact character count, so a four-digit PIN is trivially recovered from it. Treat the journal as sensitive either way. It is written 0600 in a 0700 directory, and rotated at HYPRUSE_JOURNAL_MAX_BYTES (8 MiB, one previous generation kept as .1; set 0 to never rotate).

The journal is a recorder, not a guard: if it cannot be written the action still happens and hypruse warns once on stderr, because failing your desktop over a log line is the wrong trade. What it does not record is a then= observation as its own entry: the acting call that carried it is recorded, then=screenshot and all, but the capture does not get a second line of its own.

Read it back with hypruse journal (--acts for actions only, --refused for what the guards stopped, -n N to tail, -v to include each result):

    1  09:12:10  session start pid 4211 0.10.0 confine=launched auth_guard=1 strict=True
   14  09:12:13  act  pointer    action=click x=800 y=60
   16  09:12:15  act  keyboard   action=type text=<11 chars>
   19  09:12:19  act  hypr       action=close_window target=0x5f2a10
                  REFUSED  TrustError: 0x5f2a10 (Signal) is outside the agent's confinement scope

312 actions (0 dry), 604 observations, 1 refused by a trust layer, 0 errors

HYPRUSE_DRYRUN=1 turns the same session into a rehearsal. Every acting tool validates its arguments and runs every trust guard, then reports the plan instead of executing it, so a dry run refuses exactly what a real run would:

DRY RUN, nothing was delivered: would click push button 'Send' at (1204, 880) in signal

Nothing reaches the desktop: not the click, not the keystroke, not even the window focus that normally precedes typing. Enforced twice, once at each tool and once at the input path itself, so a code path nobody thought of fails loudly rather than quietly acting during a simulation. The agent is told dry run is on, so it reports a plan instead of retrying an action that "did not work". The scope is the agent's actions, not the server's own startup: HYPRUSE_MARK, if you set it, still installs its window rule when the server starts.

hypruse replay <journal> re-issues a journal's actions through the same tool functions, so the same guards apply to the replay. It prints the plan and stops there unless you pass --execute. Before it takes the seat it refuses outright, rather than failing halfway and leaving your desktop part-way through someone else's plan, when: an action was recorded by a newer hypruse, a recorded window no longer exists (--skip-missing runs the rest), typed text was recorded as a digest, the entry is a click_ui(mark=N) whose numbering died with the session that drew it, the entry is a clipboard write and HYPRUSE_CLIPBOARD is not set, or HYPRUSE_READONLY or HYPRUSE_DRYRUN is set. It paces itself from the recorded timing, capped by --max-gap and scaled by --speed, and its own actions are recorded and marked, so replaying the same file again runs the original plan rather than the plan plus the replay of it.

The honest limit is window addresses: they are heap pointers, so yesterday's journal mostly names windows that are gone, and an address can even be reused by a different window later, which no pre-flight can catch. Replay is for re-running a flow on a desktop that still looks like the one recorded.

Dry run and replay compose in the obvious direction: let the agent work with HYPRUSE_DRYRUN=1, read the journal, and replay it with --execute once the plan is one you like.

Performance

Measured on a live session (Hyprland 0.55, 1080p, 20 windows): desktop ~20 ms (one batched hyprctl call), workspace/window dispatch ~10-20 ms, full-monitor screenshot ~65 ms (fast JPEG default; ~800 ms if you ask for lossless PNG, grim's zlib path dominates), region/zoom captures well under that. If tool calls feel slow, it is almost certainly the MCP approval prompt in front of each call, not the server. Allowlist the tools you trust and the latency disappears. Claude Code (.claude/settings.json):

{
  "permissions": {
    "allow": [
      "mcp__hypruse__desktop",
      "mcp__hypruse__screenshot",
      "mcp__hypruse__hypr"
      // add pointer/keyboard/launch once you trust your workflows
    ]
  }
}

Coordinates

Everything speaks Hyprland's global logical coordinates, the space hyprctl cursorpos and window at use. Screenshots are pixel-space; each capture returns geometry and scale so global = origin + pixel / scale. On scale 1.0 monitors (most setups) image pixels are global coordinates.

The zoom tool does the precision arithmetic for the agent: give it an estimated global point and it captures a native-resolution box around it, clamped to the screen (or to a window), with the same metadata contract. That two-step loop, estimate on the full view then re-estimate on the zoom, is the research-backed way to hit small controls.

Captures default to JPEG q90: on a 1080p frame that is roughly 12x faster to encode than PNG (grim's zlib path dominates capture time, measured ~65 ms vs ~800 ms) and 3-4x smaller, while full-res q90 reads UI text well. Pass lossless=true for exact pixels (PNG). In image mode, captures also fit the host's result-size limit (Claude Desktop caps tool results at 1 MB) by degrading quality before resolution, since grim's downscale filter is slower than a full-res capture, and cap the long edge at HYPRUSE_MAX_IMAGE_EDGE pixels (default 1568) so the host never downscales the image under the model. The applied scale is folded into the returned metadata, so coordinate mapping stays exact; tune with HYPRUSE_MAX_IMAGE_BYTES, or pass scale for a deliberate zoom-out.

By default the screenshot tool writes the image under $XDG_RUNTIME_DIR/hypruse/ and returns its path; MCP hosts with a file reader (Claude Code's Read) render it natively. This default exists because some hosts (including Claude Code 2.1.x) serialize inline MCP image blocks to base64 text the model cannot see. HYPRUSE_SCREENSHOT_MODE=image switches to inline image content blocks for hosts that render them correctly.

Development

uv sync --group dev
uv run pytest            # unit tests, no compositor needed
uv run pytest -m e2e --override-ini addopts=   # live seat-safe checks
uv run python scripts/e2e_input.py             # supervised: takes the seat ~10s

The input e2e is deliberately manual: it borrows your cursor and keyboard, counts down, proves click/scroll/type delivery by reading the target terminal's screen back over kitty remote control, and restores your focus.

Roadmap

Grounded in measured hot-path latencies and the finding that LLM calls are 76 to 96% of computer-use task latency (OSWorld-Human), so cutting round-trips beats shaving milliseconds. The round-trip work that framing motivated has largely shipped: sequence, act-and-observe then= (including then='ui'), and the accessibility-tree tools (ui, marks, click_ui) that target controls by name with no screenshot. What remains:

Faster

• In-process wlr-screencopy over the raw wire (as input already works): drop grim's fork floor from small captures and add damage-tracked wait-for-stable that returns the instant the screen settles.

Fewer round-trips

• Semantic screen diff: after an action, return only what changed (window topology from the event stream, or a bounded changed-region crop) instead of a full frame. The socket2 event expansion behind wait_for already tracks most of the topology; the missing piece is folding it into a post-action delta.

Deeper reading

• Wider accessibility coverage: close the gaps the ui and marks tools hit today, chiefly GTK's newer combo boxes that publish neither their text nor selection (so a rendered dropdown value still needs a screenshot), plus AT-SPI value-change events so then='ui' can report a control settling without a poll.

• Notifications: read recent desktop-notification content and history, not just wait for the popup to appear (wait_for already matches layer_open on the notification namespace).

Trust

• record tool: a scoped GIF or mp4 of the agent driving the desktop, via wf-recorder (a wlroots-family binary like grim), a visual companion to the journal.

Platform

• sway / niri support: the wire client already speaks the wlr protocols; what remains is an IPC layer alongside hyprctl.py (contributions welcome).

• Headless end-to-end tests in CI: needs a QEMU virtio-gpu VM, since Hyprland's aquamarine backend requires a real GPU render node that hosted runners lack.

Measurement

• Zoom-loop precision benchmark: measure click accuracy of the coarse-to-fine loop against known targets.

• End-to-end task-success benchmark on Hyprland (OSWorld-style): the deferred bigger sibling of the zoom-loop microbenchmark, scoring full multi-step tasks through the real MCP surface so the a11y-versus-vision and round-trip work is judged on task completion, not latency alone.

Related projects

Research

hypruse ships no OCR engine; its universal precision mechanism is the coarse-to-fine zoom loop: screenshot a window, re-capture the target region at native resolution, click through the exact coordinate mapping. That choice follows what the GUI-agents field converged on. Anthropic's computer use grounds clicks from raw pixels and ships a zoom action as the documented fix for small text; its troubleshooting guidance for near-miss clicks prescribes zooming and region cropping, never OCR [1]. OpenAI's CUA is likewise pure pixel grounding under resolution discipline, with no OCR layer at all [2]. Zoom is also the measured lever: training-free iterative zooming roughly doubles high-resolution grounding accuracy (OS-Atlas-7B, 18.9 → 49.7 on ScreenSpot-Pro) [3], and the benchmark's official harness implements a dozen grounding-model adapters plus four zoom/crop strategies, but zero OCR baselines [4]. Vision-only agents match or beat agents that additionally consume HTML or accessibility trees [5], substrates Wayland doesn't guarantee anyway, and state-of-the-art native agents run from screenshots alone [7]. OCR was rejected because it is blind to icons, the element class every grounding model handles worst (SeeClick: 30-52% on icons vs 56-78% on text) [6]; where OCR survives in modern stacks it is a text-disambiguation sidecar, not the targeting mechanism [8].

Where an app exposes an accessibility tree, hypruse also reads it (the ui tool, AT-SPI over D-Bus via busctl) to target controls by name with exact coordinates and no screenshot. This follows the strongest Linux precedent: OSWorld, the standard computer-use benchmark, exposes the desktop accessibility tree (obtained on Ubuntu through AT-SPI) as a first-class observation alongside screenshots, and reports the accessibility-tree-plus-screenshot combination as its best configuration [9]. The tree gives exact element identity and coordinates that current models cannot reliably infer from pixels: Agent-S tags each element with an id because MLLMs "lack an internal coordinate system," lifting OSWorld success from 11.2 to 20.6 percent [10]; Microsoft's UFO drives Windows through the UI Automation tree and fuses it with vision [11]; browser agents read the accessibility tree, which Playwright serializes to compact YAML, rather than pixels for the same reason [12]. hypruse's Wayland-specific trick is coordinate mapping: AT-SPI screen coordinates are unreliable on Wayland because an app does not know its global position, so hypruse uses window-relative extents plus the window position it already has from hyprctl. Coverage is uneven by nature: canvas, games, terminals, and Electron/Chrome without a flag expose little or nothing [13]. Testing this implementation against real GTK and Qt apps sharpened that in both directions: a typed entry reads back its exact contents, and sliders and toggles report their position and state, but GTK's newer combo boxes publish neither their text nor a selection, so reading a rendered dropdown value still needs a screenshot. Toolkits also describe widgets they never laid out (zero-height scroll arrows, unrendered tab pages reporting origins in the millions), which hypruse rejects against the window rect hyprctl knows authoritatively. A strong visual grounder can substitute for the tree entirely [5], so the accessibility tree complements the zoom loop rather than replacing it, and vision stays the guaranteed fallback.

1. Anthropic, Computer use tool: platform docs (computer_20251124, enable_zoom, resolution guidance)

2. OpenAI, Computer-Using Agent and the computer use guide

3. DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning, EMNLP 2025. arXiv:2507.00008

4. Li et al., ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. arXiv:2504.07981; official harness

5. Gou et al., Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents (UGround), ICLR 2025 Oral. arXiv:2410.05243

6. Cheng et al., SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents, ACL 2024. arXiv:2401.10935

7. Qin et al., UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv:2501.12326

8. Agyeya et al., Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents (Tesseract as a textual-grounding sidecar). arXiv:2504.00906

9. Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv:2404.07972; the a11y tree is obtained on Ubuntu through AT-SPI (code)

10. Agashe et al., Agent S: An Open Agentic Framework that Uses Computers Like a Human. arXiv:2410.08164

11. Zhang et al., UFO: A UI-Focused Agent for Windows OS Interaction (UI Automation + vision). arXiv:2402.07939

12. Playwright, ARIA snapshots: the accessibility tree as compact structured text

13. Wang et al., GUI Agents: A Survey (accessibility-API coverage gaps; a11y complements vision). arXiv:2412.13501

License

MIT