Eyes, ears and hands on a real graphical program, for an AI agent, a script or CI.
An agent working on a program with a window can't see it or hear it.
shotbox runs the program in a sealed session of its own, X11 or Wayland,
and lets whatever is driving it look, listen and act: read the accessible
tree, take screenshots, record what it plays, click by a button's name,
type, hold keys and gamepad buttons, and wait until the program is
actually ready. Then measure what it finds: names a screen reader can't
say, contrast, the Tab order, pixels changed against a reference, where
it flickers, how fast it follows the pointer, where its sound drops out,
and what it used.
Nothing it runs reaches your desktop or speakers by accident, and with
--net loopback, not your network either.
Made with shotbox: each scene is a play script in a sealed
session, recorded by shotbox record. With sound.
claude mcp add shotbox -- shotbox mcp --allow ~/src/app/shotbox-programsWithout an agent, it's the same session from the command line:
shotbox shoot editor.png --window 'Text Editor' -- gnome-text-editor notes.mdWorked examples, from checking a change to a game's smoke test: docs/recipes.md. How it differs from other tools that drive a GUI: docs/comparison.md.
shotbox mcp --allow PROGRAMS gives an agent sealed sessions over the
Model Context Protocol (stdio). It can start a session, spawn programs
from the list in PROGRAMS (NAME = COMMAND a line, and nothing else),
and then:
read the accessible tree, and take screenshots, which come back as images;
click by a node's name, press keys, type and work a gamepad;
wait for windows, stillness and announcements;
lint, measure contrast and walk the Tab order;
record what a session plays, and hear where it dropped out.
That's enough for an agent to check its own change in a GUI, test a game by playing it, audit what a screen reader hears, or reproduce a bug report in a session that's the same each time. The accessible tree is the cheap way to look, a thousand or two tokens of text; a screenshot is for what only pixels say. An agent can't run anything off the list, and its sessions stop when the server does.
What it's for, the tools, and how it's built: docs/mcp.md.
An agent that runs a program itself, from its shell, runs it on your
desktop: a window opens on your screen, a sound plays through your
speakers, a browser run under xvfb-run finds your Wayland display all
the same. To keep it in a session, put this in the project's CLAUDE.md
or AGENTS.md, or your own:
## Graphical programs, browsers and sound
Never run a graphical program, a browser (headless too) or anything that
makes sound straight on this machine: it reaches my desktop and speakers.
Run it in a shotbox session:
- a picture: `shotbox shoot OUT.png --window RE -- COMMAND`
- a script or a test suite with windows: `shotbox run [--audio] -- COMMAND`
- with none, a headless browser say: `shotbox exec [--audio] -- COMMAND`;
the shotbox package's `sealed()` seals a browser's environment, but
not the speakers
- a session to come back to: `shotbox start --name NAME`, then
`shotbox spawn --session NAME -- COMMAND`, and `shotbox stop NAME`
Not `xvfb-run`: it lets `WAYLAND_DISPLAY` and the session bus through. A
server started for a test listens on 127.0.0.1, not 0.0.0.0.Related MCP server: AI Web Tester
An agent checking its change, a screenshot for a README, and a test in CI all want the same things, and every project was growing its own copy:
Sealed. Nothing of yours leaks in by accident, and nothing the program does reaches your desktop or speakers, where bubblewrap can make a sandbox (it says when it can't): What's sealed.
Ready, not slept. Waits are for things: a window mapped, a port open, a
file written, text on a terminal's screen, or the picture holding still.
No sleep 6 and hope.
Deterministic. X11 on Xvfb, GTK's software renderer, LANG=C.UTF-8,
TZ=UTC, a cursor that doesn't blink (GTK's text caret included, and GTK's
animations off), GTK 4's text drawn at its color, as its manual font
rendering does, not fainter by however much a GTK release's automatic
one draws it (more), and PNGs with no
timestamps in them.
Cleaned up. Every program runs in its own process group, and the group is killed on the way out: signalling a wrapper script alone leaves its children running.
On Debian, with what every command and option can use:
sudo apt install xvfb xauth dbus-daemon imagemagick x11-utils python3-gi gir1.2-vte-2.91 \
at-spi2-core gir1.2-atspi-2.0 libglib2.0-bin pipewire wireplumber ffmpeg bubblewrap \
libfaketime
pipx install --system-site-packages shotbox--system-site-packages lets it see the system's PyGObject, which
shotbox term and --a11y use; pip install shotbox works too. Or run
it from a checkout: ln -s "$PWD/bin/shotbox" ~/.local/bin/shotbox.
shotbox doctor says what's here and what's missing, a line each, and
how to get it. What each command and option needs:
Needs.
No Python packages: it's the standard library and those tools. Wayland sessions need more; see Wayland.
In a GitHub Actions job on Ubuntu, the Action installs it and what it runs, as of the release (or branch) it's used at, with what each feature needs asked for by name (all off unless set), and lets bubblewrap make its sandbox, which Ubuntu's AppArmor holds back:
- uses: mishan/shotbox@v0.6.0
with:
a11y: true # --a11y and `shotbox a11y`
term: true # `shotbox term`
# wayland, audio, record, gamepads and clock too
- run: shotbox shoot app.png --window 'My App' -- ./my-appshotbox shoot app.png --window 'My App' -- ./my-app # a picture, once it's up
shotbox shoot app.png --twice --window 'My App' -- ./my-app # the same every time?
shotbox compare ref.png app.png --diff diff.png # what changed, and where
shotbox run -- ./take-pictures.sh # a script that waits, clicks and captures inside
shotbox exec --audio -- npx playwright test # sealed, with no display
shotbox start --name ed --a11y # a session to come back to
shotbox spawn --session ed -- gnome-text-editor
shotbox --session ed a11y click Open
shotbox --session ed capture open.png
shotbox stop edCommand | What it does |
runs a program, waits, takes the picture; | |
runs a command in a sealed session and exits with its status | |
the same, with no display: a headless browser, a test suite | |
a session that lasts, the programs in it, stopping one, listing them | |
a terminal to take pictures of | |
waits for a window, a port, a file, stillness, a picture, a change, a log line | |
takes a picture, or a burst of them | |
records a video, the sound, or both | |
presses keys, types, works the pointer; held too | |
moves the pointer out of the way; gives a window the keyboard | |
plays a script of timed input, on one clock | |
how fast what's drawn follows the pointer | |
finds a part of the screen that flickers or flashes | |
works a virtual gamepad | |
the accessible tree: read, lint, Tab order, click by name, contrast, announcements | |
counts the pixels that differ, picture by picture or directory by directory | |
a picture as colorblind eyes see it | |
a recording's level, clipping and dropouts | |
sessions for an agent, over MCP | |
what's here of what each command needs |
The options sessions take, --wayland, --a11y, --audio,
--gamepads, --net loopback, --identity, --clock, --dark and
the rest: Session options.
A session keeps a program from reaching your desktop by accident, not one that sets out to:
Its environment is rebuilt from an allowlist; HOME and every XDG
directory are a scratch dir.
Its display and session bus, and with --a11y and --audio its
accessibility bus and sound server, are its own; the sound server plays
to nowhere.
Where bubblewrap can make a sandbox (it says when it can't), nothing reaches the machine's sound devices or your desktop's Wayland, PulseAudio, PipeWire and bus sockets.
Your network is reachable unless --net loopback, your user and host
name show unless --identity, and the real time unless --clock.
Not sealed: the files outside the scratch home, your X server, other processes, and the machine's system bus.
All of it, and what each option covers and doesn't: docs/sealed.md.
--a11y starts an accessibility (AT-SPI) bus of the session's own, so
what a screen reader would hear can be read and checked. Without it, a
session has none: GTK is told not to look. Your desktop's bus is never
touched either way.
shotbox a11y prints the tree, lints it for what has no name to say,
walks the Tab order, clicks by name and measures contrast, and shotbox wait announce waits for what's announced; for GTK, Qt, AccessKit's
toolkits and Chromium. The commands, how it's put together, and what
each check can and can't tell: docs/accessibility.md.
--wayland makes a session a Wayland one: the program runs under a
headless sway, drawing in software, instead of on Xvfb. --xwayland adds
an Xwayland to it, and DISPLAY, so an X client runs the way it does on a
Wayland desktop, GTK programs made to with --env GDK_BACKEND=x11.
Needs sway and grim, Xwayland for --xwayland, and wf-recorder
for record: sudo apt install sway grim xwayland wf-recorder. Why
sway, how it fits, and what differs from X11:
docs/wayland.md.
Commands: every command and session option, what each needs, resource use and the clock
Recipes: worked examples
How it compares with other tools that drive a GUI
MCP: the tools an agent gets, and how they're built
Python: shotbox.Session and here(), for a script
that drives the display itself
Node: helpers for browser checks with Playwright
Contributing: the tests
MIT. See LICENSE.
What a screen reader would find: every visible node, a line each, indented, with its role, name, description and the states a screen reader says (focusable, focused, checked...). The cheapest way to see what's on screen and what can be clicked by name. A node with an empty name ("") is one a screen reader can say nothing for. Use screenshot for how it looks.
| Name | Required | Description | Default |
|---|---|---|---|
| bounds | No | Add @x,y WIDTHxHEIGHT to each line | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, lowering the bar. The description adds real value beyond them: the line-by-line indented format, the enumerated state set, and the meaning of an empty name ('a screen reader can say nothing for'). It does not mention output size/truncation limits, but the added disclosure is substantive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core concept and no filler sentences; each sentence (format, cheapest path, empty-name semantics, screenshot alternative) earns its place. Slightly run-on, but nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the return-value burden and does so by describing the indented line format and included fields. Safety is covered by annotations, so the remaining gaps are only minor operational details like output size. Adequate for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so all three parameters are documented in the schema itself, making 3 the baseline. The description alludes to line content and name-based clickability but adds no format or regex guidance for bounds or window beyond what the schema already supplies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states precisely what the tool returns: an indented line-per-node rendering of role, name, description and screen-reader states. It explicitly distinguishes itself from the sibling screenshot tool ('Use screenshot for how it looks'), so an agent can route correctly without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear context ('the cheapest way to see what's on screen and what can be clicked by name') and names one alternative with its selecting condition (screenshot for appearance). It stops short of addressing other plausible siblings like tab_walk or click, so it is clear context rather than exhaustive when/when-not routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Everything programs have announced to a screen reader in the session, a line each, "polite: Saved": GTK's announcements, a browser's live regions.
| Name | Required | Description | Default |
|---|---|---|---|
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, covering the safety profile. The description adds useful output context: one line per announcement, example format ("polite: Saved"), and source systems (GTK, browser live regions). It does not describe ordering, time range, or whether announcements accumulate, leaving some behavioral gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that is appropriately sized for the tool's simplicity. The core idea ('Everything programs have announced...') is front-loaded, though the embedded example and colon make it slightly clunky.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only, single-parameter tool with full schema coverage and no output schema, the description adequately explains what is returned, its line-based format, and example content. Minor gaps around time scope and ordering are acceptable given the simplicity and annotation coverage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with the session parameter documented as 'Its name, from start_session'. The description adds no additional meaning for the parameter, so the baseline of 3 applies when the schema fully handles parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the resource precisely: screen-reader announcements from all programs in the session. It distinguishes itself from siblings like program_output or accessible_tree through the specific medium (screen reader) and example formats such as "polite: Saved". No explicit verb is used, but the meaning is clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains what the tool returns but gives no guidance on when to use it versus alternatives. There is no mention of prerequisites, when-not-to-use conditions, or sibling tools that might be more appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Click a node by its name, or a point by x and y, on the screen or inside a window. By name is the one to use: it waits up to timeout seconds for the node, raises its window and clicks its middle. When two nodes match it lists them, with where they are; when none does, the nearest names. A click gives a window the keyboard, which nothing else does on X11. hold and state, by x and y, hold the button as press_keys holds keys.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | Pixels from the left | |
| y | No | Pixels from the top | |
| hold | No | Seconds to keep it down before letting go | |
| name | No | A node's name, as accessible_tree gives it: exactly, or a regex matched in full | |
| role | No | Narrows name: exactly, or a regex | |
| state | No | Only press, leaving it down until a call with up; or only let go | |
| button | No | 1 left, 2 middle, 3 right | |
| double | No | Click twice, as a double click | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session | |
| timeout | No | By name, seconds to wait for the node (10) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the annotations (readOnlyHint=false, destructiveHint=false) by disclosing wait/timeout behavior, window raising, ambiguity handling (lists matching nodes, or nearest names when none match), the hold/state press-and-release semantics, and the key X11 side effect that clicking grants a window the keyboard.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the primary verb and the two targeting modes, and every sentence carries information. Some clauses are dense and run-on ('hold and state, by x and y, hold the button as press_keys holds keys'), which slightly hurts readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 11-parameter tool with no output schema but full schema coverage, the description covers mode selection, timing, failure modes, and side effects. It stops short of describing what an ambiguous-match response actually returns, though it gestures at it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds real meaning beyond the schema: the name-vs-x/y mode split, the interactive behavior of timeout in name mode, and the hold/state pairing ('hold the button as press_keys holds keys').
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (click) and two concrete targeting modes (a node by name, or a point by x/y) on the screen or inside a window. It is immediately distinguishable from siblings like press_keys, scroll, or type_text by the action it performs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly recommends the mode to prefer ('By name is the one to use') and explains what that mode gives you (waiting, window raising, middle-click). It does not name alternative sibling tools or state when not to use click, so it stops short of full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Contrast measured on the screen, in each node's bounds: text, buttons and links against their background, images, and with edges, a button's or field's edge against what's round it; large text needs 3. A line each, the node as accessible_tree has it with its bounds, then the colors; then what wasn't measured, which doesn't fail (too small, or GTK 4's, whose positions are wrong), and why. Switches, sliders, check boxes and what's disabled are left out, and so is what's scrolled out of view.
| Name | Required | Description | Default |
|---|---|---|---|
| edges | No | Least for edges (unchecked) | |
| ignore | No | Regexes for lines that can't be fixed yet | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session | |
| graphics | No | Least for images (3) | |
| text_ratio | No | Least for text (4.5) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the annotations, disclosing what is deliberately excluded (switches, sliders, check boxes, disabled controls, off-screen content), what is skipped without failing (too small, GTK 4 wrong positions), and the default thresholds. One tension worth noting: the passive 'measured' framing implies a read, yet readOnlyHint is false; this is mild and plausibly explained by internal screen capture, so it is not a hard contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The core statements (what it measures, thresholds) are front-loaded, but the middle sentence is a dense run-on mixing output format, exclusions, and rationale. It is serviceable yet not tight; some clauses could be split or trimmed.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description compensates by sketching the return shape ('a line each, the node as accessible_tree has it with its bounds, then the colors... then what wasn't measured, and why'). Combined with 100% schema coverage and explicit exclusions, an agent has nearly everything needed to call and interpret it, though the safety/mutation profile is left to annotations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already carries the parameter meaning and the baseline is 3. The description only tangentially reinforces parameter intent via threshold defaults ('large text needs 3'), adding no syntax or format detail beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
It states a specific operation (contrast measurement on the screen) and enumerates exactly which node types are checked (text, buttons, links, images, edges), which lets an agent separate it from siblings like screenshot or accessible_tree. The framing is a bit passive ('Contrast measured...') rather than a clean verb+resource, but the resource and scope are unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: the thresholds ('large text needs 3') and the reference to accessible_tree make clear this is an accessibility-audit step, but there is no explicit 'use this when / use X instead' guidance. It never contrasts itself with lint or accessible_tree as the right tool for a given task.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Work a virtual gamepad, in a session started with gamepads: press buttons together, tapped, held for hold seconds, or with state down left down until a call with state up; push a stick to x, y; pull a trigger to value; or neutral, letting go of everything. A stick or trigger stays where it's put, unless hold says for how long. Programs in the session see a real Xbox 360 controller, and none of the machine's.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | The stick, left to right | |
| y | No | The stick, up to down | |
| pad | No | Which of the session's pads (1) | |
| hold | No | Seconds to keep it down before letting go | |
| state | No | Only press, leaving it down until a call with up; or only let go | |
| value | No | The trigger, let go to pulled all the way | |
| which | No | Which stick or trigger | |
| action | Yes | Press buttons, push a stick, pull a trigger, or let go of everything | |
| buttons | No | a b x y lb rb back start guide ls rs, and the d-pad's up down left right | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare this as a non-read-only, non-destructive action, so the description carries most of the behavioral burden and does well: it explains persistence ('a stick or trigger stays where it's put, unless hold'), the down/up state semantics, and isolation ('programs see a real Xbox 360 controller, and none of the machine's'). It does not cover error conditions or limits on pads/buttons.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, both information-dense with no filler, and the action modes are front-loaded ahead of the state/persistence caveat. The first sentence is long and semicolon-heavy, which slightly taxes readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 10-parameter action tool with no output schema and minimal annotations, the description covers the action model, state persistence, hold duration, session prerequisite, and emulation isolation. Remaining details (pad indexing, button names) live in the schema, so nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3; each parameter already has a descriptive field. The prose clarifies cross-parameter behavior (hold overrides the persistent state, state down persists until an up call) but adds little per-parameter meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names the specific resource (a virtual gamepad) and enumerates the four action modes (press, stick, trigger, neutral) matching the action enum, so the agent knows exactly what it does. It does not explicitly contrast itself with siblings like press_keys or click, so it falls short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states the precondition 'in a session started with gamepads' and the session param points back to start_session, so the implied workflow is that a session must exist first. There is no explicit guidance on when to prefer gamepad over press_keys or other input tools, and no exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Nodes a screen reader has no name to say for: controls, images (not inside a named control) and anything else that takes the focus. A line each: the node, as accessible_tree has it, then what's wrong.
| Name | Required | Description | Default |
|---|---|---|---|
| ignore | No | Regexes for lines that can't be fixed yet | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, so the safety profile is covered. The description adds genuinely useful behavior beyond that: the output is one line per offending node, each line shows the node as accessible_tree renders it followed by the problem, and it clarifies a scoping rule (images are only flagged when not inside a named control).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tightly packed sentences with no filler, and the core purpose is front-loaded. The phrasing ('has no name to say for') is slightly cryptic, costing it a point, but nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the burden of explaining returns and does so (one line per node, node rendering plus the defect). Combined with 100% schema coverage for the three parameters and annotations covering the read-only nature, this is close to complete, though it omits anything about result volume or ordering.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents ignore, window, and session. The description adds no parameter-level detail (e.g., how ignore regexes match, or window vs app=RE syntax), so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific function: reporting focusable nodes (controls, images, other focus targets) that lack an accessible name for a screen reader to announce. It is clearly an accessibility audit and references the accessible_tree sibling as the format reference, though it does not explicitly position itself as the lint counterpart to that tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no when-to-use or when-not-to-use guidance. accessible_tree is mentioned only as a format reference, not as an alternative the agent should choose between, so the agent gets no routing help for deciding when to run lint versus inspecting the tree directly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Press keys in turn, each a chord: Tab, Return, Escape, ctrl+s, alt+shift+Tab, F6. They go to whatever has the keyboard; click a window first if nothing does. hold presses them all together and keeps them down that long (walking while running); state down leaves them down, across calls, until a call with state up (move or click while a key is held).
| Name | Required | Description | Default |
|---|---|---|---|
| hold | No | Seconds to keep it down before letting go | |
| keys | Yes | Chords, pressed one after another | |
| state | No | Only press, leaving it down until a call with up; or only let go | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare it is not read-only and not destructive. The description adds real behavioral context beyond that: the focus-targeting requirement (input goes to whatever has the keyboard), and the cross-call persistence of state=down until a state=up call — a stateful side effect an agent must know. Does not cover error behavior or what happens with invalid key names.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action before conditions and parameter nuance, and it is tightly packed without filler. The parenthetical chord list (Tab, Return, Escape, ctrl+s, alt+shift+Tab, F6) is illustrative padding that could be trimmed, but each remaining sentence carries meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 4 parameters, no output schema and minimal annotations, the description covers the essential unknowns: focus targeting, chord sequencing, hold duration, and cross-call key state. It does not address return values or failure modes, but for a 60-second-bounded input tool this is close to complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description earns more by explaining semantics the schema cannot: the 'walking while running' analogy for hold, and that state down persists across calls while state up releases. Chord examples (ctrl+s, alt+shift+Tab) also clarify the keys array format beyond 'Chords, pressed one after another'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Press keys in turn, each a chord') with concrete examples of what a chord is. The scope is unambiguous. However, it never names or contrasts with the adjacent sibling type_text, which is the most likely confusion point for an agent choosing between keyboard tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear operational context: keys go to whatever holds focus, and 'click a window first if nothing does' handles the precondition. It also distinguishes when to use hold (simultaneous press with a duration) vs state down/up (persistent press across calls). No explicit exclusion naming type_text as the alternative for free text.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Whether a spawned program is still running, and the end of what it printed: for a window that never came up, or a crash.
| Name | Required | Description | Default |
|---|---|---|---|
| lines | No | How many lines from the end (40) | |
| program | Yes | Its name in the allow list | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, so the safety profile is covered. The description adds that it reports both liveness and trailing output, which is genuinely useful context, but it omits the tail-limit behavior (default 40, max 2000) and what is returned for an already-exited program.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single compact sentence with no filler, front-loading the primary return (running status). Slightly dense and grammatically choppy, but no wasted text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the return-value burden. It gestures at the two return components but does not specify the tail size, truncation, or behavior on non-running programs, leaving gaps for a diagnostic tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and each parameter (session, program, lines) already carries a description, so the baseline is 3. The description adds nothing about parameter semantics beyond what the schema documents.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states two outcomes (running status and tail of printed output) but the phrasing is awkward ('the end of what it printed') and it never cleanly says 'returns the last N lines of a spawned program's output.' It is distinguishable from spawn/stop_session, but the scope is muddled.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It implies a diagnostic use case ('for a window that never came up, or a crash') but does not say when to prefer it over alternatives such as wait_for, accessible_tree, or screenshot when diagnosing a program that failed to appear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Record a clip of the screen, or of one window, and save it as an .mp4: in the session's scratch directory, gone when the session stops, or at path. The answer is the saved path, how long the clip is, and up to 8 frames from it RETURNED AS ONE IMAGE in rows of four, with the times they show: you can't play the clip, but a person or another tool can. A clip stops at 100 MB. sound also says what it heard, as the sound tool does. until_line ends it at an event instead; the answer gives the line. For a few moments of motion, screenshot's burst is cheaper.
| Name | Required | Description | Default |
|---|---|---|---|
| crop | No | WIDTHxHEIGHT+X+Y, inside what's taken | |
| file | No | until_line's, by its path from the allow file's directory | |
| path | No | Where to save it, a new .mp4, by its path from the allow file's directory | |
| tail | No | Seconds to go on after the line (0) | |
| sound | No | What plays too, in a session started with audio | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| seconds | No | How long; with until_line, the most (60) | |
| session | Yes | Its name, from start_session | |
| until_line | No | A regex: stop tail seconds after a line matching it is written to file |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Adds substantial context beyond the annotations: default save location in the session scratch directory that is deleted when the session stops, a 100 MB clip cap, what the response contains (path, duration, up to 8 frames composed into one image), that the clip itself cannot be played back, and that sound reports what was heard. These are exactly the operational facts annotations cannot convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose is front-loaded and nearly every sentence carries information, but the prose is dense and run-on, with mid-sentence shifts (the 'RETURNED AS ONE IMAGE' aside, the compressed 'until_line's, by its path' fragments) that slow parsing. Information-dense but not well structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description does the work of describing the return value (path, duration, frames image, matched line) and covers the scratch-directory lifecycle and size cap. Only peripheral details like required session state or failure behavior are absent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is already 100%, so the baseline is 3, but the description goes further by explaining cross-parameter relationships the schema only hints at — e.g. 'seconds' acts as a maximum when 'until_line' is set, 'tail' extends past the matched line, and 'path' must be a new .mp4 relative to the allow file's directory.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource up front ('Record a clip of the screen, or of one window, and save it as an .mp4') and explicitly distinguishes itself from a sibling by noting screenshot's burst is cheaper for brief motion. An agent can tell this apart from screenshot/sound without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear context plus an explicit alternative ('For a few moments of motion, screenshot's burst is cheaper') and notes the sound tool equivalence. It lacks a general when-not-to-record rule or prerequisites (e.g. session must be running), so it stops short of full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
A picture of the screen, or of one window, RETURNED AS AN IMAGE for you to look at: the only way to see colors, layout and what's drawn. Pictures cost far more than the accessible tree; take the window rather than the screen, and crop when you need a part. vision checks nothing is told apart by color alone; burst, as one image in rows of four, is for motion, an animation or a transition.
| Name | Required | Description | Default |
|---|---|---|---|
| crop | No | WIDTHxHEIGHT+X+Y, inside what's taken | |
| park | No | Move the pointer away first, so nothing shows its hover | |
| burst | No | Pictures to take | |
| every | No | With burst, seconds between them | |
| vision | No | protanopia (no red cones), deuteranopia (no green), tritanopia (no blue), or gray (no color) | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already carry the safety profile, and the description adds genuinely useful behavioral detail: the return is an image, burst returns 'one image in rows of four', and pictures are costlier than the accessible tree. It does not, however, explain the surprising readOnlyHint=false (e.g. that park moves the pointer) or any side effects, so it adds moderate context rather than rich disclosure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded and free of filler, but the compressed, fragmentary phrasing hurts clarity ('vision checks nothing is told apart by color alone' is grammatically broken and ambiguous). It is dense rather than wasteful, but some sentences do not parse cleanly on first read.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter tool with no output schema, the description covers purpose, selection, cost, and most parameter intent adequately. Remaining gaps (park semantics, the readOnlyHint=false rationale) are minor and partly covered by the schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description goes beyond the schema strings by explaining the intent behind window-vs-screen, cropping, vision modes, and burst layout. Only 'park' and 'session' are left entirely to the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('A picture of the screen, or of one window, RETURNED AS AN IMAGE') and immediately differentiates it from the sibling accessible_tree by naming the alternative. An agent can tell this is the visual-capture tool without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use ('the only way to see colors, layout and what's drawn'), names the competing option (accessible_tree) and the cost tradeoff, then prescribes preferences ('take the window rather than the screen, and crop when you need a part') plus scenario routing (vision for color-alone checks, burst for motion/animation/transition). This is close to a complete selection guide.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Turn the mouse wheel over a node by its name, found as click finds it, or at a point by x and y. For what's past a list's, a page's or a window's edge; then wait_for stable and look again.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | Pixels from the left | |
| y | No | Pixels from the top | |
| dx | No | Clicks right; negative, left | |
| dy | No | Clicks down; negative, up | |
| name | No | A node's name, as accessible_tree gives it: exactly, or a regex matched in full | |
| role | No | Narrows name: exactly, or a regex | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=false and destructiveHint=false, so the safety profile is covered. The description adds real behavioral context — scrolling reveals content past an edge and the viewport should be re-read only after wait_for reports stable — but it says nothing about scroll magnitude limits or how the view state changes. Adequate but not rich against an already-covered safety profile.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no padding, and the primary targeting mechanism (node name, else x/y point) is front-loaded. The second sentence is slightly cryptic and grammatically compressed ('For what's past a list's ... edge'), but it is not wasteful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 8-param, no-output-schema interaction tool with safety annotations already present, the description covers targeting, the scenario that motivates scrolling, and the required post-scroll workflow. The remaining detail (dx/dy direction, bounds) lives in the schema, so little is missing for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with all 8 params documented, including dx/dy direction and bounds. The description only restates name and x/y targeting, adding no meaning beyond the schema and omitting the wheel direction parameters entirely. Baseline 3 is appropriate when the schema carries the load.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Turn the mouse wheel over a node by its name ... or at a point by x and y.' It also ties its targeting semantics to the sibling click ('found as click finds it'), which helps an agent distinguish it. It stops short of a crisp differentiation of when scrolling beats other interactions, so not a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly scopes the use case: 'For what's past a list's, a page's or a window's edge,' and prescribes the follow-up workflow 'then wait_for stable and look again.' That is clear when-to-use guidance, though it names no when-not or alternative tool beyond wait_for.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Record what plays in a session started with audio: action start, then do what makes the sound, then action stop, which says what it heard: its length, peak and RMS (dBFS), samples clipped, and each gap of exact silence between the first sound and the last, as a program that falls behind leaves; then what's wrong, if anything (gaps, clipping, silence). Sounds meant to be apart, two beeps say, need a gap longer than the silence between them.
| Name | Required | Description | Default |
|---|---|---|---|
| gap | No | Silence, in milliseconds, said as a gap (1) | |
| action | Yes | start, then stop | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With annotations only declaring readOnly=false, destructive=false and openWorld=false, the description carries the behavioral load and does so well by disclosing the stop-action report: length, peak and RMS in dBFS, clipped samples, exact-silence gaps, and any detected problems. It omits latency, resource cost, and what happens if stop is never called, which keeps it out of the top band.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The content is dense and largely earns its place, but it is delivered as one long semicolon-chained sentence that is hard to parse on first read ('then what's wrong, if anything... as a program that falls behind leaves'). The start/make-sound/stop workflow is front-loaded, yet the structure obscures rather than supports the content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description correctly carries the return-value burden, describing both the measurement report and the failure conditions (gaps, clipping, silence). The main missing piece is the relationship to start_session/stop_session and whether the session lifecycle is managed elsewhere.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema baseline is 3, but the description adds real meaning: 'session' is tied to start_session, and 'gap' is explained as the silence threshold that separates intended distinct sounds rather than just a millisecond range. That is genuine semantic value beyond the schema's numeric bounds.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: it records audio during a session, driven by start/stop actions. It is understandable that this is an audio-capture tool, but it never distinguishes itself from the similarly-named siblings 'record' or 'program_output', leaving the agent to guess which capture tool applies.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It spells out the required sequence (action start, make the sound, action stop) and gives concrete usage advice for the gap parameter ('sounds meant to be apart need a gap longer than the silence between them'). It does not name alternatives or state when not to use it, so it falls short of full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Start one of the programs the person allowed this server to run, by its name, in a session; nothing else can run. Then wait_for its window before looking at it. If it doesn't come up, program_output says why.
| Name | Required | Description | Default |
|---|---|---|---|
| program | Yes | Its name in the allow list | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only say readOnlyHint=false and destructiveHint=false; the description adds meaningful context beyond that: execution is restricted to an allow list ("nothing else can run"), the window must be awaited, and failure is diagnosable via program_output. It does not say whether the process persists after the session ends or what happens on repeated spawns of the same program.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the action and its constraint, followed by the required next step and the failure path. No filler, and each sentence carries a distinct instruction.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description covers the outcome sufficiently: either the window comes up (then wait_for it) or program_output explains the failure. For a 2-parameter, fully documented launch tool this is complete enough to call correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both parameters (program, session) are already documented, including the enum constraint on program. The description's "by its name, in a session" adds only light reinforcement of what the schema states, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ("Start") and resource ("one of the programs the person allowed this server to run") plus the scoping constraint that nothing else can run, which distinguishes it from the input/interaction siblings. It also names the downstream tools (wait_for, program_output), so an agent can place it correctly in a sequence without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly prescribes the follow-up flow: wait_for the window before looking at it, and use program_output if the program doesn't come up. There is no sibling alternative to spawn, so no exclusion is needed, but it stops short of stating prerequisites (e.g., that a session from start_session must already exist, which is only implied by the session parameter).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Start a sealed session: a display of its own (X11 under Xvfb, or Wayland under a headless sway), its own D-Bus and accessibility bus, and a scratch home. Nothing in it reaches the desktop of the person you're working for, and nothing of theirs reaches it. Start one, spawn a program in it, then look with accessible_tree (cheap, text) before screenshot (pixels). It lasts until stop_session, or until this server stops.
| Name | Required | Description | Default |
|---|---|---|---|
| tz | No | Its time zone, Europe/Berlin (UTC) | |
| net | No | A network its programs share and nothing else is on: nothing reaches in or out | |
| a11y | No | An accessibility bus (default true) | |
| dark | No | GTK and libadwaita's dark style | |
| name | No | What to call it (s1, s2... by default) | |
| audio | No | A sound server whose sink plays to nowhere | |
| clock | No | ISO 8601, 2026-09-26T18:40:00, in tz or UTC: where its wall clock starts and runs on from, so dates in pictures don't change (needs libfaketime) | |
| usage | No | Say, when it's stopped, what its programs used: peak memory and CPU time | |
| screen | No | WIDTHxHEIGHT (1280x800) | |
| wayland | No | Wayland, not X11 | |
| gamepads | No | Virtual Xbox 360 pads, for the gamepad tool | |
| identity | No | USER@HOST: who and where its programs are, for whoami, hostname, ls -l and $USER | |
| xwayland | No | Wayland, with Xwayland for X programs | |
| text_scale | No | Their text size, 1.25 for 125% | |
| high_contrast | No | Their high contrast style |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare the generic profile (readOnlyHint=false, openWorldHint=false, destructiveHint=false); the description adds the genuinely non-obvious traits: the isolation guarantee in both directions, and the session's lifetime and termination condition. That is meaningful context the structured fields cannot convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with the core capability, then the workflow, then the lifetime. The rhetorical pair ('nothing in it reaches the desktop ... nothing of theirs reaches it') earns its space by making the isolation guarantee unambiguous.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 15-parameter tool with no output schema, the description supplies the mental model, workflow, and lifetime, which is most of what is needed. The one gap is that it never says what starting a session returns (presumably the name/handle that spawn and stop_session take), which an agent must infer.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with 15 well-annotated optional properties (each has a concrete description, enum, or bounds), so the schema carries the parameter burden entirely. The description adds no parameter-level syntax or defaults beyond what is already there, making the baseline 3 correct.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource ('Start a sealed session') and immediately enumerates what that session contains: its own display (X11/Xvfb or Wayland/sway), D-Bus and a11y bus, and a scratch home. That level of specification lets an agent distinguish it from siblings like stop_session or spawn without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit ordered workflow ('Start one, spawn a program in it, then look with accessible_tree (cheap, text) before screenshot (pixels)') and names the terminating alternative ('lasts until stop_session, or until this server stops'). Both when-to-use and the choice between inspection siblings are covered.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Stop a session: every program in it ends, unsaved work and all, and its scratch directory goes. With usage, it says their peak memory and CPU time.
| Name | Required | Description | Default |
|---|---|---|---|
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already flag destructiveHint=true and idempotentHint=true, but the description adds the concrete blast radius: all programs terminate, unsaved work is destroyed, and the scratch directory is deleted. That is exactly the beyond-annotation detail an agent needs before calling a destructive tool, plus it hints at the returned metrics.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, no filler, with the destructive scope front-loaded. The trailing clause 'With usage, it says their peak memory and CPU time' is ambiguous in phrasing but still short and informative.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter destructive tool with no output schema, the description covers the side effects and gestures at return values, with annotations handling safety semantics. The vague 'with usage' phrasing is the only real gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single parameter is documented as 'Its name, from start_session'. The description adds nothing about the parameter, so the baseline of 3 applies when the schema does all the work.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Stop a session') and immediately spells out the scope and consequence: every program ends, unsaved work is lost, scratch directory removed. An agent can distinguish this from start_session purely from the text.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no when-to-use or when-not-to-use guidance and no mention of the sibling start_session as the counterpart operation. The only extra sentence describes a return value rather than a usage condition, so an agent gets no routing help.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Press a key until the keyboard focus comes back round, and say where it went each time: the Tab order. It ends in a cycle (fine), a trap (a press that leaves the focus where it was), or after most presses. A text view that takes Tab is left with ctrl+Tab, shift+Tab, Escape or F6, Tab not pressed there; the controls Tab never reached are said. Leaves the focus where it ends.
| Name | Required | Description | Default |
|---|---|---|---|
| key | No | The chord to press (Tab) | |
| most | No | Presses at most (50) | |
| ignore | No | Regexes for unreached controls to leave out | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session | |
| visible | No | Also check each place it lands shows it has the focus |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare it is non-read-only and non-destructive; the description goes further by disclosing the side effect ('leaves the focus where it ends'), the three termination modes (cycle, trap, max presses), and that unreached controls are reported. That is meaningful behavioral context beyond the annotations, though permission or failure behavior is unstated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four relatively tight sentences with no filler and the core purpose front-loaded, but the telegraphic, run-on phrasing makes the middle sentences harder to parse than they need to be. Compact but not clean.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a six-parameter, no-output-schema tool it covers the essentials: what it does, where focus ends, when it stops, and what it reports. It leaves the window/session/visible parameters to the schema, which is acceptable given full schema coverage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all six parameters. The description only loosely ties to a few of them (the key chord, 'most presses', unreached controls), adding little syntax or default detail the schema does not already carry — the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The first sentence names a concrete action and resource: press a key repeatedly until focus cycles, reporting each landing — i.e., walk the Tab order. It is distinguishable from siblings like press_keys or click, though the clipped phrasing ('say where it went each time') costs some readability.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It hints at a usage caveat (a text view that captures Tab must be left with ctrl+Tab, shift+Tab, Escape or F6), but never states when to reach for tab_walk versus press_keys, accessible_tree, or click. No explicit alternatives or when-not conditions are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Type text into whatever has the keyboard focus, a key at a time. ASCII only: a character with no key fails the call, once what's before it is typed.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | The text to type | |
| session | Yes | Its name, from start_session |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare non-read-only and non-destructive, so safety is partly covered, and the description adds genuinely useful behavior: ASCII-only constraint and, critically, non-atomic failure semantics (a bad character fails the call only after everything preceding it has already been typed). That partial-effect disclosure is exactly the kind of trait an agent needs and cannot get from annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the action and scoped by the ASCII/partial-failure caveat. Dense and waste-free, though the trailing clause 'once what's before it is typed' is slightly garbled and could be phrased more cleanly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter, no-output-schema tool, the description covers the action, the targeting model (keyboard focus), an input constraint, and the failure mode. It omits how newlines or non-character keys are handled and that press_keys should be used for those, but nothing essential to a correct call is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with only two required parameters, so the schema fully documents 'text' and 'session'. The description adds no syntax or format guidance beyond the ASCII constraint already implied, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (type) and resource (text into the focused control), and the phrase 'into whatever has the keyboard focus' distinguishes it from siblings like press_keys, which sends key combinations rather than character streams. It stops short of naming an alternative explicitly, but the action is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: the description makes clear the tool requires something already focused, and the 'a key at a time' phrasing hints at when press_keys would be preferable. There is no explicit when-to-use/when-not or named alternative, so an agent must infer the routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Wait, instead of sleeping, for: a window (value: its title, exactly or a regex, or app=RE), stable (the screen, or the window, unchanged for value seconds, 0.5, up to 60: after a click, for the repaint), an announcement (value: a regex in what was announced to a screen reader; each announcement is waited for once), focus (value: the name of the node with the keyboard focus, as click takes one), match (value: a PNG, by its path from the allow file's directory, that the screen, or the window, cropped to crop, should look like), change (from how it looked when the wait started), or line (value: a regex found in a line of file, written after the wait starts, or with from_start, anywhere in it: for a program that logs its events, rather than a sleep; a file not there yet is waited for, and one rotated or rewritten is read from the top). match and change say how long they took; line answers with the line. Fails after timeout seconds, saying so.
| Name | Required | Description | Default |
|---|---|---|---|
| crop | No | WIDTHxHEIGHT+X+Y, inside what's taken | |
| file | No | line's, by its path from the allow file's directory | |
| fuzz | No | match, change: percent a pixel may differ by (0) | |
| most | No | match, change: pixels past fuzz allowed (0) | |
| what | Yes | The kind of wait | |
| value | No | What to wait for, as what says | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session | |
| timeout | No | Seconds before it fails (10) | |
| from_start | No | line: anywhere in file |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint/openWorldHint annotations, it discloses real behavior: failure after timeout, that each announcement is waited for once, that a not-yet-existing file is waited for and a rotated file is read from the top, and what match/change/line return. No annotation contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Content is front-loaded with the enumeration, but it is delivered as one sprawling, clause-heavy sentence that is hard to scan. Nearly every clause earns its place informationally, yet the structure hurts readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 10-parameter tool with no output schema, the description covers the failure mode and the return behavior of the output-producing modes, and schema coverage is complete for parameters. Enough to call the tool correctly, though mode-specific return details are only partially sketched.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3, but the description meaningfully expands the opaque 'value' parameter by defining what it means for each mode (window title/regex/app=RE, regex of announcement, node name, PNG path, regex in a file line). This adds semantic value the schema alone does not.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb (wait) and enumerates the exact conditions it can wait on (window, stable, announcement, focus, match, change, line), which clearly separates it from a sleep and from other tools. It never explicitly contrasts itself with siblings like screenshot or program_output, but the resource scope is concrete enough for an agent to identify the tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Each mode carries an implicit when-to-use cue: stable 'after a click, for the repaint', line 'for a program that logs its events, rather than a sleep', announcement for screen-reader output. It states the alternative to sleeping but gives no guidance on when to prefer a sibling tool instead of waiting.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.7.1accessible_treeannouncementsclickcontrastgamepadlintpress_keysprogram_outputrecordscreenshotscrollsoundspawnstart_sessionstop_sessiontab_walktype_textwait_forScored across 18 tools
Most tools have distinct purposes (session lifecycle vs. input vs. inspection vs. audio/gamepad). However, accessible_tree, lint, contrast, and tab_walk all inspect UI but overlap in outputting node information, and screenshot/record/sound overlap in capturing media. Descriptions help but boundaries are not always crisp.
Mixed conventions: snake_case verbs like type_text, start_session, wait_for, press_keys, tab_walk, accessible_tree sit alongside bare nouns like spawn, screenshot, announcements, sound, gamepad, click, scroll, lint, contrast, record. Readable but not a predictable pattern.
18 tools is borderline heavy for a UI-automation server and several specialized tools (gamepad, sound, record, contrast, lint, tab_walk) dilute the core surface. Each earns some place, but the count feels overgrown relative to essential capabilities.
Strong lifecycle coverage: session start/stop, program spawn/output, inspection (tree, screenshot, announcements), input (click, scroll, keys), and specialized capture/audit tools. Missing explicit window management (resize, move, close) and clipboard operations, but agents can work around most gaps.
Browser-based QA for AI-built software. Test pages with real browsers via agents.
Accessibility pre-checks (WCAG/BFSG) in a real browser + statement drafts. Pay per call.
Run multi-step tasks in a real Chrome browser: persistent environments, live view, human takeover.
Read-only web QA agent: scans a site in a real browser and returns evidence-backed findings
