shotbox
This MCP server lets an agent run and drive allowed GUI programs in isolated shotbox sessions, then inspect, interact with, record, and test what they show and play.
Start and stop sealed X11/Wayland sessions with configurable display size, network, audio, accessibility, dark/high-contrast style, timezone/clock, identity, text scale, and virtual gamepads.
Spawn only allow-listed programs in a session and read their recent stdout/stderr and exit status.
Inspect the accessible UI tree, take screenshots (including bursts, crops, and colorblind simulations), and list screen-reader announcements.
Interact with the GUI by clicking elements by accessible name or coordinates, scrolling, typing text, pressing key chords, and using virtual Xbox 360 gamepads.
Wait for windows, stable screens, announcements, focus, image matches or changes, and log lines instead of sleeping.
Run accessibility checks for missing names, contrast, and keyboard focus order.
Record video and audio, inspect saved frames, and analyze audio for length, levels, clipping, silence gaps, and dropouts.
Report session program resource usage (peak memory and CPU time) when enabled.
Provides tools for testing and interacting with GTK applications, including accessibility tree inspection, contrast measurement, deterministic rendering, and input simulation.
Allows running and interacting with graphical programs in sealed Wayland sessions, including starting a headless Wayland compositor and using accessibility trees, screenshots, and input simulation.
shotbox
Eyes, ears and hands on a real graphical program, for an AI agent, a script or CI.
An agent working on a program with a window can't see it or hear it.
shotbox runs the program in a sealed session of its own, X11 or Wayland,
and lets whatever is driving it look, listen and act: read the accessible
tree, take screenshots, record what it plays, click by a button's name,
type, hold keys and gamepad buttons, and wait until the program is
actually ready. Then measure what it finds: names a screen reader can't
say, contrast, the Tab order, pixels changed against a reference, where
it flickers, how fast it follows the pointer, where its sound drops out,
and what it used.
Nothing it runs reaches your desktop or speakers by accident, and with
--net loopback, not your network either.

Made with shotbox: each scene is a play script in a sealed
session, recorded by shotbox record. With sound.
claude mcp add shotbox -- shotbox mcp --allow ~/src/app/shotbox-programsWithout an agent, it's the same session from the command line:
shotbox shoot editor.png --window 'Text Editor' -- gnome-text-editor notes.mdWorked examples, from checking a change to a game's smoke test: docs/recipes.md. How it differs from other tools that drive a GUI: docs/comparison.md.
For agents
shotbox mcp --allow PROGRAMS gives an agent sealed sessions over the
Model Context Protocol (stdio). It can start a session, spawn programs
from the list in PROGRAMS (NAME = COMMAND a line, and nothing else),
and then:
read the accessible tree, and take screenshots, which come back as images;
click by a node's name, press keys, type and work a gamepad;
wait for windows, stillness and announcements;
lint, measure contrast and walk the Tab order;
record what a session plays, and hear where it dropped out.
That's enough for an agent to check its own change in a GUI, test a game by playing it, audit what a screen reader hears, or reproduce a bug report in a session that's the same each time. The accessible tree is the cheap way to look, a thousand or two tokens of text; a screenshot is for what only pixels say. An agent can't run anything off the list, and its sessions stop when the server does.
What it's for, the tools, and how it's built: docs/mcp.md.
An agent that runs a program itself, from its shell, runs it on your
desktop: a window opens on your screen, a sound plays through your
speakers, a browser run under xvfb-run finds your Wayland display all
the same. To keep it in a session, put this in the project's CLAUDE.md
or AGENTS.md, or your own:
## Graphical programs, browsers and sound
Never run a graphical program, a browser (headless too) or anything that
makes sound straight on this machine: it reaches my desktop and speakers.
Run it in a shotbox session:
- a picture: `shotbox shoot OUT.png --window RE -- COMMAND`
- a script or a test suite with windows: `shotbox run [--audio] -- COMMAND`
- with none, a headless browser say: `shotbox exec [--audio] -- COMMAND`;
the shotbox package's `sealed()` seals a browser's environment, but
not the speakers
- a session to come back to: `shotbox start --name NAME`, then
`shotbox spawn --session NAME -- COMMAND`, and `shotbox stop NAME`
Not `xvfb-run`: it lets `WAYLAND_DISPLAY` and the session bus through. A
server started for a test listens on 127.0.0.1, not 0.0.0.0.Related MCP server: AI Web Tester
Why
An agent checking its change, a screenshot for a README, and a test in CI all want the same things, and every project was growing its own copy:
Sealed. Nothing of yours leaks in by accident, and nothing the program does reaches your desktop or speakers, where bubblewrap can make a sandbox (it says when it can't): What's sealed.
Ready, not slept. Waits are for things: a window mapped, a port open, a file written, text on a terminal's screen, or the picture holding still. No
sleep 6and hope.Deterministic. X11 on Xvfb, GTK's software renderer,
LANG=C.UTF-8,TZ=UTC, text carets that don't blink (GTK's, Qt 6.5's, browsers'; and GTK's animations off), GTK 4's text drawn at its color, as its manual font rendering does, not fainter by however much a GTK release's automatic one draws it (more), and PNGs with no timestamps in them. It's the same every time, not the same as a desktop: differences from a desktop.Cleaned up. Every program runs in its own process group, and the group is killed on the way out: signalling a wrapper script alone leaves its children running.
Install
On Debian, with what every command and option can use:
sudo apt install xvfb xauth dbus-daemon imagemagick x11-utils python3-gi gir1.2-vte-2.91 \
at-spi2-core gir1.2-atspi-2.0 python3-gi-cairo gir1.2-pango-1.0 libglib2.0-bin pipewire \
wireplumber ffmpeg bubblewrap libfaketime
pipx install --system-site-packages shotbox--system-site-packages lets it see the system's PyGObject, which
shotbox term and --a11y use; pip install shotbox works too. Or run
it from a checkout: ln -s "$PWD/bin/shotbox" ~/.local/bin/shotbox.
shotbox doctor says what's here and what's missing, a line each, and
how to get it. What each command and option needs:
Needs.
No Python packages: it's the standard library and those tools. Wayland sessions need more; see Wayland.
In a GitHub Actions job on Ubuntu, the Action installs it and what it runs, as of the release (or branch) it's used at, with what each feature needs asked for by name (all off unless set), and lets bubblewrap make its sandbox, which Ubuntu's AppArmor holds back:
- uses: mishan/shotbox@v0.10.0
with:
a11y: true # --a11y and `shotbox a11y`
speak: true # `shotbox a11y speak`: Orca, and what a11y installs
term: true # `shotbox term`
# wayland, audio, record, gamepads and clock too
- run: shotbox shoot app.png --window 'My App' -- ./my-appA quick tour
shotbox shoot app.png --window 'My App' -- ./my-app # a picture, once it's up
shotbox shoot app.png --twice --window 'My App' -- ./my-app # the same every time?
shotbox compare ref.png app.png --diff diff.png # what changed, and where
shotbox run -- ./take-pictures.sh # a script that waits, clicks and captures inside
shotbox exec --audio -- npx playwright test # sealed, with no display
shotbox start --name ed --a11y # a session to come back to
shotbox spawn --session ed -- gnome-text-editor
shotbox --session ed a11y click Open
shotbox --session ed capture open.png
shotbox stop edCommands
Command | What it does |
runs a program, waits, takes the picture; | |
runs a command in a sealed session and exits with its status | |
the same, with no display: a headless browser, a test suite | |
a session that lasts, the programs in it, stopping one, listing them | |
a terminal to take pictures of | |
waits for a window, a port, a file, stillness, a picture, a change, a log line | |
takes a picture, or a burst of them | |
records a video, the sound, or both | |
presses keys, types, works the pointer; held too | |
moves the pointer out of the way; gives a window the keyboard | |
plays a script of timed input, on one clock | |
how fast what's drawn follows the pointer | |
finds a part of the screen that flickers or flashes | |
works a virtual gamepad | |
the accessible tree: read, lint, Tab order, click by name, contrast, announcements, a node to show | |
counts the pixels that differ, picture by picture or directory by directory | |
a picture as colorblind eyes see it | |
a recording's level, clipping and dropouts | |
sessions for an agent, over MCP | |
what's here of what each command needs |
The options sessions take, --wayland, --a11y, --audio,
--gamepads, --net loopback, --identity, --clock, --dark and
the rest: Session options.
What's sealed
A session keeps a program from reaching your desktop by accident, not one that sets out to:
Its environment is rebuilt from an allowlist;
HOMEand every XDG directory are a scratch dir.Its display, session bus and system bus, and with
--a11yand--audioits accessibility bus and sound server, are its own; the sound server plays to nowhere. It has no printers (but CUPS on localhost:631, unless--net loopback).--machine-busgives it the machine's system bus and printers.Where bubblewrap can make a sandbox (it says when it can't), nothing reaches the machine's sound devices or your desktop's Wayland, PulseAudio, PipeWire and bus sockets.
Your network is reachable unless
--net loopback, your user and host name show unless--identity, and the real time unless--clock.Not sealed: the files outside the scratch home, your X server, and other processes.
All of it, and what each option covers and doesn't: docs/sealed.md.
Accessibility
--a11y starts an accessibility (AT-SPI) bus of the session's own, so
what a screen reader would hear can be read and checked. Without it, a
session has none: GTK is told not to look. Your desktop's bus is never
touched either way.
shotbox a11y prints the tree, lints it for what has no name to say,
walks the Tab order, clicks by name and measures contrast, shotbox wait announce waits for what's announced, and shotbox wait node for a
node to show; for GTK, Qt, AccessKit's
toolkits and Chromium. The commands, how it's put together, and what
each check can and can't tell: docs/accessibility.md.
Wayland
--wayland makes a session a Wayland one: the program runs under a
headless sway, drawing in software, instead of on Xvfb. --xwayland adds
an Xwayland to it, and DISPLAY, so an X client runs the way it does on a
Wayland desktop, GTK programs made to with --env GDK_BACKEND=x11.
Needs sway and grim, Xwayland for --xwayland, and wf-recorder
for record: sudo apt install sway grim xwayland wf-recorder. Why
sway, how it fits, and what differs from X11:
docs/wayland.md.
Docs
Commands: every command and session option, what each needs, resource use and the clock
Recipes: worked examples
How it compares with other tools that drive a GUI
MCP: the tools an agent gets, and how they're built
Differences from a desktop: drawing, window manager, animations, language, scale, input, measurements
Python:
shotbox.Sessionandhere(), for a script that drives the display itselfNode: helpers for browser checks with Playwright
Contributing: the tests
License
MIT. See LICENSE.
Available Tools
18 toolscheck_accessible_namesARead-only
Lint the accessibility tree for nodes a screen reader has no name to say for: controls, images (not inside a named control) and anything else that takes the focus. The answer: a tree line each, with bounds, then what's wrong; or "No problems.". Use it to check labels after a change; for colors, use check_contrast.
| Name | Required | Description | Default |
|---|---|---|---|
| ignore | No | Regexes for lines that can't be fixed yet | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, so the safety profile is covered. The description goes further and discloses the exact output shape (one tree line per problem with bounds plus the defect, or "No problems."), which matters because there is no output schema. It does not discuss scoping or auth, but none is plausibly needed for a read-only lint.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the purpose, then the return format, then the routing hint — a sensible order. The colloquial phrasing ("no name to say for") costs a little precision, but every sentence carries information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description compensates by explaining the return value; combined with usage routing and a clear target set, an agent has enough to call it correctly. Only minor gaps (e.g. how many problems are returned, whether output is truncated) remain.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so session, window and ignore are already documented in the schema (including that window accepts exact titles, full-match regex, or app=RE). The description adds no parameter-level meaning beyond that, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (lint) on a specific resource (the accessibility tree) and enumerates what it targets: controls, images not inside a named control, and anything focusable that lacks an accessible name. This distinguishes it cleanly from siblings like check_contrast and check_focus_order without the agent opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit when-to-use ("check labels after a change") and names an alternative with its selecting condition ("for colors, use check_contrast"). The routing decision is fully specified rather than left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
check_contrastAIdempotent
Check that text, buttons, links and images stand out from their background: WCAG's 4.5:1 for text (3 for large text) and 3:1 for images unless text_ratio and graphics say otherwise, and with edges, a button's or field's edge too. It raises the window and measures a picture of the screen in each node's bounds. The answer: each node that fails, a tree line with bounds, its ratio and colors, or "No problems."; then what it couldn't measure, which doesn't fail (too small, or GTK 4's, whose positions are wrong), and why. A window where nothing can be measured, as GTK 4's, is an error, not a pass. Switches, sliders, check boxes, disabled nodes and what's scrolled out of view are skipped. For missing names, use check_accessible_names.
| Name | Required | Description | Default |
|---|---|---|---|
| edges | No | Least for edges (unchecked) | |
| ignore | No | Regexes for lines that can't be fixed yet | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session | |
| graphics | No | Least for images (3) | |
| text_ratio | No | Least for text (4.5) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without relying on annotations, it discloses the side effect of raising the window and capturing the screen in each node's bounds, the exact result shape (failed nodes with bounds, ratio, colors, or 'No problems.', plus unmeasurable items), and the critical behavioral rule that an unmeasurable window is an error rather than a pass. This explains why readOnlyHint=false despite being an analysis tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose is front-loaded and nearly every sentence carries unique information about thresholds, skips, output or errors. The prose is occasionally run-on (the 'The answer: ... ; then what it couldn't measure' sentence), which costs some clarity but not much length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully carries the burden: it describes the return values, the failure semantics, the error condition and the exclusion set for a 6-parameter tool. Nothing an agent needs to call and interpret this tool correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds meaning by tying text_ratio (4.5), graphics (3) and edges to WCAG rationale and explaining that the ratio parameters override the defaults. It does little extra for window, session and ignore, which the schema already covers.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: check that text, buttons, links and images stand out from their background, and grounds it in WCAG ratios. It also explicitly distinguishes itself from the sibling check_accessible_names for name problems, so an agent can route without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Names the alternative tool for missing names and enumerates what is skipped (switches, sliders, check boxes, disabled nodes, scrolled-out content), which is real usage context. It stops short of a general 'use this when / not when' framing beyond that one routing hint.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
check_focus_orderA
Check the keyboard focus order in one call: press Tab, or key, until the focus comes back round, and list where it went, a tree line each, after a "began in" line. Then the controls the key never reached, and how it ended: round again (fine), stuck (a trap, or a text view nothing leaves), not round after most presses, lost (no node has the focus), stopped (a text view closed as it was left), or nothing focused to begin with (click the window with click_element first). With visible, "focus not shown on N" counts stops that don't show it. A text view that takes Tab is left as a person would, with ctrl+Tab, shift+Tab, Escape or F6. The focus stays where it ended. To press a few keys yourself, use press_keys.
| Name | Required | Description | Default |
|---|---|---|---|
| key | No | The chord to press (Tab) | |
| most | No | Presses at most (50) | |
| ignore | No | Regexes for unreached controls to leave out | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session | |
| visible | No | Also check each place it lands shows it has the focus |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint=false and destructiveHint=false, which is ambiguous about side effects; the description resolves this by disclosing that keys are actually pressed and 'the focus stays where it ended'. It also enumerates the possible termination outcomes (round, stuck, lost, stopped, etc.), which is real behavioral context beyond the annotations. No contradiction: pressing keys that move focus is consistent with readOnlyHint=false.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The content largely earns its place given the tool's complexity, and the core action is front-loaded, but it is delivered as one sprawling run-on sentence with nested parentheticals and a long inline enum of ending states, making it hard to parse. It is dense rather than wasteful, but the structure is poor.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 6-parameter tool with no output schema and no nested objects, the description covers what it does, the shape of the output (a 'began in' line then one tree line per stop), the terminating states, and the visible behavior. Return format is partially carried by the description since no output schema exists; it is largely complete for an agent to call correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds semantics the schema does not: it explains what 'visible' does (the 'focus not shown on N' count), how the default key (Tab) drives the traversal, and how the run ends ('until the focus comes back round'). That goes beyond the terse schema strings.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource (check the keyboard focus order) and concretely describes the outcome: press Tab until focus cycles, list where it went as a tree, plus unreached controls and the ending state. It distinguishes itself from the sibling it names, press_keys, so an agent can tell the two apart without reading the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Routes the agent to press_keys for typing a few keys manually, and gives the prerequisite that if nothing is focused the agent must click the window with click_element first. It doesn't state explicit exclusions ('don't use this when...'), but the alternative and precondition cover the main decision points.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
click_elementA
Click an element by its accessible name, or a point by x and y, on the screen or inside a window. By name is the one to use: it waits up to timeout seconds for the node, raises its window and clicks its middle, and the answer gives the node and its bounds. When two nodes match it lists them, with where they are; when none does, the nearest names. A click gives a window the keyboard, which nothing else does on X11. hold and state, by x and y, hold the button as press_keys holds keys.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | Pixels from the left | |
| y | No | Pixels from the top | |
| hold | No | Seconds to keep it down before letting go | |
| name | No | A node's name, as get_accessible_tree gives it: exactly, or a regex matched in full | |
| role | No | Narrows name: exactly, or a regex | |
| state | No | Only press, leaving it down until a call with up; or only let go | |
| button | No | 1 left, 2 middle, 3 right | |
| double | No | Click twice, as a double click | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session | |
| timeout | No | By name, seconds to wait for the node (10) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only say readOnlyHint=false and destructiveHint=false; the description supplies the real behavioral picture: it waits up to timeout for the node, raises its window before clicking, and the click transfers keyboard focus on X11 — a side effect nothing else in the toolset performs. It also discloses ambiguity handling (lists multiple matches with locations, returns nearest names on failure) and the expected response payload.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the primary mode and the recommendation, and no sentence is padding. The telegraphic phrasing at the end ("hold and state, by x and y, hold the button as press_keys holds keys") is compact but slightly elliptical, costing a bit of readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 11 parameters, no output schema, and only sparse annotations, the description does the heavy lifting: it explains target resolution, waiting, window raising, focus side effect, ambiguity and failure behavior, and the return payload. It omits any permission/prerequisite notes or visibility requirements, but the essentials for correct invocation are present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the prose adds meaning the schema does not: hold and state are stated to apply with x and y (a mode coupling the schema never expresses), and hold is tied to press_keys semantics. That genuine cross-parameter clarification lifts it above baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (click) plus two concrete targeting modes (accessible name, or x/y point) with screen/window scope. It is trivially distinguishable from siblings like type_text, scroll_wheel, and press_keys, and even explains its relationship to press_keys.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"By name is the one to use" explicitly steers the agent to the preferred mode and implicitly relegates the x/y path (with hold/state) to a secondary case. It stops short of naming when to prefer sibling tools such as type_text for focusing a field, so alternative-tool guidance is partial rather than complete.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_accessible_treeARead-only
Read what a screen reader would find: every visible node, a line each, indented, with its role, name, description and the states a screen reader says (focusable, focused, checked...). The cheapest way to see what's on screen and what can be clicked by name. A node with an empty name ("") is one a screen reader can say nothing for. Use take_screenshot for how it looks.
| Name | Required | Description | Default |
|---|---|---|---|
| bounds | No | Add @x,y WIDTHxHEIGHT to each line | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already establish a safe read (readOnlyHint=true, openWorldHint=false), but the description adds real value beyond them: the exact output shape (indented lines, per-node role/name/description/state) and the semantic that an empty name string means the screen reader can say nothing. It does not mention cost limits or behavior on very large trees, which keeps it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with the capability and output format, then the cost/positioning claim, then the alternative route. Every sentence earns its place with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the burden of explaining return values and does so well (line-per-node, indented, field list, empty-name meaning). It is nearly complete; only truncation or large-tree behavior is unaddressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so bounds, window, and session are already documented in the schema and the description need not repeat them. The description adds no parameter-level detail (e.g. how bounds interacts with the line format), so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource — reading the accessibility tree — and enumerates exactly what each line contains (role, name, description, states). It is clearly distinguishable from sibling tools like take_screenshot and check_accessible_names, which operate on the same UI but answer different questions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Positions the tool as "the cheapest way to see what's on screen and what can be clicked by name" and names the alternative with its condition: "Use take_screenshot for how it looks." That is a clear routing rule, though it gives no explicit when-not (e.g. large trees, when a screenshot is required for verification).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_program_outputARead-only
Read the last lines (40, or lines) a spawned program wrote to stdout and stderr, after a line saying whether it's still running or exited, with its status. It works after the program has exited, until the session stops; for a program spawned more than once, it reads the latest. Use it when a window never comes up or a program seems to have crashed; to wait for a line in a log file, use wait_for line.
| Name | Required | Description | Default |
|---|---|---|---|
| lines | No | How many lines from the end (40) | |
| program | Yes | Its name in the allow list | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint and openWorldHint; the description adds substantive behavior beyond that: output is a status line plus the last 40 lines, stderr and stdout are combined, results persist after exit until session stop, and repeated spawns resolve to the latest instance. This is exactly the lifecycle/ordering context an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Information is front-loaded and dense, with no filler sentences; the alternative tool is mentioned last as routing guidance. Slightly awkward phrasing ('after a line saying whether it's still running or exited, with its status') costs a point on readability, but every clause carries content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description fully explains the return shape (status line plus trailing log lines), covers the lifecycle edge cases (post-exit reads, repeated spawns), and all parameters are schema-documented. Nothing an agent needs to invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all three parameters are already documented in the schema. The description only confirms the default line count (40) already noted in the schema's lines description, adding no real meaning beyond it. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (read) and resource (last lines a spawned program wrote to stdout/stderr), including the status header line. It explicitly distinguishes itself from the sibling wait_for, so an agent can select it without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit use case (a window never comes up, or a program seems to have crashed) and names the alternative tool plus the condition that selects it ('to wait for a line in a log file, use wait_for line'). It also notes the tool remains usable after the program exits, until the session stops.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_announcementsARead-only
List what programs in the session have announced to a screen reader since it started, oldest first, a line each with its politeness: "polite: Saved", "assertive: Error". That's GTK's announcements and a browser's live regions. Reading doesn't clear them; "Nothing announced yet." means none so far. Use it to check what a screen reader user heard after an action; to wait for one, use wait_for announcement.
| Name | Required | Description | Default |
|---|---|---|---|
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, so safety is covered. The description adds real behavioral context beyond that: reading does not clear the buffer, the empty-state string is 'Nothing announced yet.', and results are ordered oldest-first with politeness prefixes. It does not discuss growth/limits of the buffer, so not a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core purpose, then supporting detail on format, clearing behavior, and the alternative. Dense but each clause earns its place; the inline quoted examples slightly lengthen it without waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description must describe the return value, and it does: one line per announcement with politeness prefix, plus the empty-state message. For a single-parameter read tool this is complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the single 'session' parameter is documented in the schema ('Its name, from start_session'), so the schema carries the burden. The description adds nothing further about the parameter, making the baseline 3 correct.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('List what programs in the session have announced to a screen reader') plus scope ('since it started, oldest first') and output format. An agent can tell this apart from siblings like get_program_output or wait_for without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when to use it ('to check what a screen reader user heard after an action') and names the alternative for the adjacent need ('to wait for one, use wait_for announcement'). Nothing is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
press_keysA
Press keys in turn, each a chord: Tab, Return, Escape, ctrl+s, alt+shift+Tab, F6. They go to whatever has the keyboard; click a window first with click_element if nothing does. hold presses them all together and keeps them down that long (walking while running); state down leaves them down, across calls, until a call with state up (move or click while a key is held). The answer says what it pressed.
| Name | Required | Description | Default |
|---|---|---|---|
| hold | No | Seconds to keep it down before letting go | |
| keys | Yes | Chords, pressed one after another | |
| state | No | Only press, leaving it down until a call with up; or only let go | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only say readOnly=false / destructive=false, so the description carries the real behavioral load: keys go to whatever has focus, cross-call persistent state ('state down leaves them down, across calls, until a call with state up'), hold duration semantics, and what the response reports. These are stateful side effects an agent cannot infer from the annotations or schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose and chord examples are front-loaded, and nearly every clause carries information. It is dense rather than wasteful, though the state clause ('state down leaves them down, across calls, until a call with state up (move or click while a key is held)') is nested enough to require a second read.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a stateful input tool with no output schema, it covers focus targeting, the three modes (chord list, hold, state), and notes the answer reports what was pressed. It is close to complete, missing only edge behavior such as invalid key names or what happens if no focused window exists.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds interpretive value: it ties 'hold' to a concrete scenario (walking while running) and clarifies that 'state down' persists across calls until an 'up' call. That is meaning beyond the schema's one-line field descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Press keys in turn, each a chord') and backs it with concrete chord examples (Tab, ctrl+s, alt+shift+Tab, F6) that immediately separate it from the sibling type_text. An agent can distinguish keystroke injection from text entry without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear when-to-use context: chords vs. text, and the prerequisite 'click a window first with click_element if nothing does', naming a sibling tool. It also explains when to use hold vs. state (walking while running, move/click while a key is held). It never explicitly states when NOT to use it (e.g., for typing strings use type_text), so it falls short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
record_audioC
Record what a session started with audio plays, from a call with action start to one with action stop, across the calls between that make the sound. stop's answer: the sound's length, peak and RMS level (dBFS), samples clipped, and each gap of exact silence between the first sound and the last, as a program that falls behind leaves; then what's wrong, if anything (gaps, clipping, silence). Sounds meant to be apart, two beeps say, need a gap longer than the silence between them. For a video of the screen with its sound, up to a minute, use record_video with sound.
| Name | Required | Description | Default |
|---|---|---|---|
| gap | No | Silence, in milliseconds, said as a gap (1) | |
| action | Yes | start, then stop | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With annotations declaring readOnlyHint=false, destructiveHint=false and openWorldHint=false, the safety profile is already covered. The description adds useful behavioral detail about what stop returns (length, peak/RMS dBFS, clipped samples, silence gaps, and reported problems), which is valuable given there is no output schema. It still omits where the recording is written or how it is retrieved afterward.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The body is a run-on of awkward, hard-to-parse clauses with parenthetical and appended fragments rather than front-loaded, scannable sentences. The content is dense but the structure actively impedes reading.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 3-parameter tool with no output schema, the description covers the start/stop workflow and enumerates the stop result fields, which is the key gap it needs to fill. It stops short of explaining where the captured audio lives or how to access it after stop.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents session, action and gap, making 3 the baseline. The description adds only marginal meaning, mainly the idea that two intended-separate sounds require a gap longer than the natural silence between them.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description does convey that this records audio captured during a session, bounded by a start call and a stop call, and it explicitly names the sibling record_video for the screen-plus-audio case. However, the phrasing is convoluted ("Record what a session started with audio plays... across the calls between that make the sound"), so the core verb+resource is harder to extract than it should be for a clear 4.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives one routing rule ('For a video of the screen with its sound, up to a minute, use record_video with sound') and hints at the gap requirement for separating distinct sounds. It does not state prerequisites (e.g., audio device/permission needs) or an explicit when-not-to-use beyond the record_video pointer.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
record_videoA
Record a video of the screen, or of one window, and save it as an .mp4: in the session's scratch directory, gone when the session stops, or at path. The answer is the saved path, how long the clip is, and up to 8 frames from it RETURNED AS ONE IMAGE, with the times they show: you can't play the clip, but a person or another tool can. A clip stops at 100 MB. sound also says what it heard, as record_audio does. until_line ends it at an event instead; the answer gives the line. For a few moments of motion, take_screenshot's burst is cheaper.
| Name | Required | Description | Default |
|---|---|---|---|
| crop | No | WIDTHxHEIGHT+X+Y, inside what's taken | |
| file | No | until_line's, by its path from the allow file's directory | |
| path | No | Where to save it, a new .mp4, by its path from the allow file's directory | |
| tail | No | Seconds to go on after the line (0) | |
| sound | No | What plays too, in a session started with audio | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| seconds | No | How long; with until_line, the most (60) | |
| session | Yes | Its name, from start_session | |
| until_line | No | A regex: stop tail seconds after a line matching it is written to file |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnly=false, destructive=false, openWorld=false; the description adds substantial context beyond that: a 100 MB clip cap, that the scratch-directory file is deleted at session stop, that audio requires a session started with audio, and that the result is frames-as-one-image the agent cannot play. These are exactly the operational limits an agent needs and none are recoverable from structured fields.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose and output contract are front-loaded in the first two sentences, and each subsequent clause carries distinct information (size cap, audio requirement, stopping modes, cheaper alternative). The colon-chained, run-on phrasing makes it denser than ideal, but there is little filler to cut.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the return-value burden and does so fully: saved path, clip duration, up to 8 timestamped frames returned as one image, and the until_line case yielding the matching line. Combined with the size cap and file-lifetime note, an agent has everything needed to call and interpret this tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents all 9 parameters and the baseline is 3. The description still adds cross-parameter meaning the schema does not: seconds is a maximum when until_line is present, until_line's result reports the matching line, and sound only functions in an audio-enabled session.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource (record a video of the screen or one window) and the artifact produced (.mp4), naming both save destinations (scratch directory vs. path). It also differentiates itself from siblings by name, citing record_audio and take_screenshot's burst. An agent can distinguish it from every other capture tool without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly routes to alternatives: 'For a few moments of motion, take_screenshot's burst is cheaper' and 'sound also says what it heard, as record_audio does.' It also frames the until_line vs. seconds choice as two stopping modes. It stops short of stating when recording is inappropriate overall (e.g., headless/no-display sessions), so it is clear context rather than full when/when-not coverage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scroll_wheelA
Turn the mouse wheel over a node by its name, found as click_element finds it (waiting for it, its window raised), or at a point by x and y, leaving the pointer there. Use it for what's past a list's, a page's or a window's edge, then wait_for stable and look again; to move the keyboard focus, use press_keys.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | Pixels from the left | |
| y | No | Pixels from the top | |
| dx | No | Clicks right; negative, left | |
| dy | No | Clicks down; negative, up | |
| name | No | A node's name, as get_accessible_tree gives it: exactly, or a regex matched in full | |
| role | No | Narrows name: exactly, or a regex | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover the safety profile (readOnlyHint=false, destructiveHint=false) but say nothing about side effects. The description adds two non-obvious ones: the node's window is raised, and the pointer is left at the target position ('leaving the pointer there'). It doesn't say whether scrolling can itself change focus or trigger lazy loading failures, so it stops short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the operation and targeting modes, then the usage/recovery advice. Every clause carries information (node resolution semantics, pointer placement, follow-up wait, alternative tool) with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a side-effecting input tool with no output schema and fully documented parameters, the description supplies targeting semantics, side effects, and a post-scroll verification workflow. Only minor gaps remain, such as interaction with existing focus or modifier-key scrolling.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so all eight parameters are already documented, giving a baseline of 3. The description still adds the targeting model that the schema does not state: name/role/window behave exactly as click_element resolves them, and x/y are an alternative point-based target rather than a supplement — that is real meaning beyond the field descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a concrete verb+resource ('Turn the mouse wheel over a node by its name ... or at a point by x and y') and immediately distinguishes itself from click_element (the same node-finding semantics) and press_keys (the keyboard-focus alternative). An agent can place it precisely among its siblings without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit when-to-use ('for what's past a list's, a page's or a window's edge'), explicit follow-up ('then wait_for stable and look again'), and an explicit alternative with its selecting condition ('to move the keyboard focus, use press_keys'). Nothing about routing is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spawn_programA
Start one of the programs the person allowed this server to run, by its name, in a session; nothing else can run. The answer gives its pid. Then wait_for its window, by the program's name, before looking at it. If it doesn't come up, get_program_output says why.
| Name | Required | Description | Default |
|---|---|---|---|
| program | Yes | Its name in the allow list | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=false / destructiveHint=false; the description adds the allow-list restriction, the fact that a pid is returned, and a documented failure path. It does not cover lifecycle questions (can the same program be spawned twice, is the process tied to the session's lifetime), which keeps it short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with the action, then follow-up tooling, then the failure path; every sentence earns its place. Minor awkwardness in the phrasing 'in a session' and the compressed final clause keeps it from a 5.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description correctly supplies the return value (pid) and the diagnostic route for failure, so an agent can call and follow up correctly. Missing edge cases such as an invalid session name or repeated spawns of the same program.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and both parameters carry their own descriptions ('Its name in the allow list', 'Its name, from start_session'). The prose only echoes 'by its name' and 'in a session', so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (start/spawn) and resource (an allow-listed program) plus the hard scope constraint that nothing outside the allow list can run. It also distinguishes itself from siblings by naming wait_for and get_program_output in their respective roles.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit sequence: spawn the program, then wait_for its window by name before inspecting it, and fall back to get_program_output if it fails to appear. Alternatives and their triggering conditions are stated rather than left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
start_sessionA
Start a sealed session: a display of its own (X11 under Xvfb, or Wayland under a headless sway), its own D-Bus and accessibility bus, and a scratch home. Nothing in it reaches the desktop of the person you're working for, and nothing of theirs reaches it. The answer gives its name, display and size, the programs spawn_program can start in it, and anything it couldn't seal. It lasts until stop_session.
| Name | Required | Description | Default |
|---|---|---|---|
| tz | No | Its time zone, Europe/Berlin (UTC) | |
| net | No | A network its programs share and nothing else is on: nothing reaches in or out | |
| a11y | No | An accessibility bus (default true) | |
| dark | No | GTK and libadwaita's dark style | |
| name | No | What to call it (s1, s2... by default) | |
| audio | No | A sound server whose sink plays to nowhere | |
| clock | No | ISO 8601, 2026-09-26T18:40:00, in tz or UTC: where its wall clock starts and runs on from, so dates in pictures don't change (needs libfaketime) | |
| usage | No | Say, when it's stopped, what its programs used: peak memory and CPU time | |
| screen | No | WIDTHxHEIGHT (1280x800), up to 7680x4320 | |
| wayland | No | Wayland, not X11 | |
| gamepads | No | Virtual Xbox 360 pads, for use_gamepad | |
| identity | No | USER@HOST: who and where its programs are, for whoami, hostname, ls -l and $USER | |
| xwayland | No | Wayland, with Xwayland for X programs | |
| text_scale | No | Their text size, 1.25 for 125% | |
| high_contrast | No | Their high contrast style |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare it is a non-read-only, non-destructive, closed-world operation; the description goes well beyond that by guaranteeing isolation in both directions ('nothing in it reaches the desktop... nothing of theirs reaches it') and by describing the return payload, including partial-failure reporting ('anything it couldn't seal'). No contradiction with destructiveHint=false or openWorldHint=false.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with the core verb and the isolation guarantee, then the return shape, then the lifetime. The prose is slightly literary ('a sound server whose sink plays to nowhere' style is confined to the schema), but every sentence carries distinct information with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 15 parameters, no required fields, and no output schema, the description carries exactly the right load: it explains the isolation model, the lifetime, and what the returned answer contains, including what it could not seal. Nothing an agent needs to invoke or interpret this call is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all 15 parameters are already documented with formats, defaults and examples (WIDTHxHEIGHT, ISO 8601 clock, USER@HOST identity, loopback-only net). The description adds no parameter-level meaning beyond the schema, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Start a sealed session') and immediately defines what 'sealed' means: its own X11/Wayland display, its own D-Bus and accessibility bus, and a scratch home. This is unmistakably distinct from stop_session and from the action tools that operate inside a session.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The lifecycle is spelled out implicitly but clearly: the session is what spawn_program starts programs in, and it lasts until stop_session. That gives the agent enough context to know this must precede the spawn/observe tools, though it states no explicit when-not or prerequisite conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
stop_sessionADestructiveIdempotent
Stop a session start_session made, when you're done with it: every program in it is killed, unsaved work and all, and its scratch directory is deleted, with any clip record_video saved there (one saved at a path is kept). In a session started with usage, the answer says its programs' peak memory and CPU time. You needn't stop sessions before the server ends: they all stop with it.
| Name | Required | Description | Default |
|---|---|---|---|
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond destructiveHint=true: it enumerates exactly what is destroyed (every program killed, unsaved work lost, scratch directory deleted, saved clips removed except one saved to a path). It also discloses what a call returns for sessions started with usage (peak memory and CPU time), which matters because there is no output schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The critical verb and resource are front-loaded, and every clause carries new information (destruction scope, return values, lifecycle exemption). The sentences are long and comma-stacked, but there is little that could be cut without losing behavioral detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive one-parameter tool with no output schema, the description covers destruction semantics, the exception for saved clips, the return payload, and the lifecycle question of whether the call is ever required. Nothing an agent needs before invoking it is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single 'session' parameter, so the schema already carries the semantics. The description only reinforces the provenance of the name ('from start_session'), adding no format or syntax detail beyond the structured field, which is the baseline 3 case.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Starts with a specific verb+resource ('Stop a session start_session made') and immediately ties it to the sibling that creates that resource, so an agent can pair the two without ambiguity. It is unmistakably distinct from all other siblings, which are interaction/capture tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when ('when you're done with it') and an explicit when-not ('You needn't stop sessions before the server ends: they all stop with it'), which prevents an agent from making pointless calls at shutdown. No alternative tool exists for this purpose, so nothing further needs to be named.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
take_screenshotAIdempotent
Take a picture of the screen, or of one window, RETURNED AS AN IMAGE for you to look at: the only way to see colors, layout and what's drawn. Pictures cost far more than get_accessible_tree; take the window rather than the screen, and crop when you need a part. vision checks nothing is told apart by color alone; burst, as one image, is for motion, an animation or a transition. The answer gives its size, and if it was shrunk, the scale.
| Name | Required | Description | Default |
|---|---|---|---|
| crop | No | WIDTHxHEIGHT+X+Y, inside what's taken | |
| park | No | Move the pointer away first, so nothing shows its hover | |
| burst | No | Pictures to take | |
| every | No | With burst, seconds between them | |
| vision | No | protanopia (no red cones), deuteranopia (no green), tritanopia (no blue), or gray (no color) | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare safety (idempotent, non-destructive, not read-only); the description adds the cost model relative to get_accessible_tree and describes the response ('gives its size, and if it was shrunk, the scale'), which matters since there is no output schema. It does not explain state side effects such as pointer movement (park) or any rate limits, so it stops short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose and output type before the cost caveat and per-parameter guidance; no filler sentences. It is dense to the point of being telegraphic in places ('vision checks nothing is told apart by color alone'), which slightly hurts readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter capture tool with no output schema and thin annotations, the description covers modality, cost trade-off, window-vs-screen choice, cropping, burst semantics, and the shape of the return value. It omits any mention of side effects like pointer parking or failure modes when a window title matches nothing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds real meaning beyond the schema: it justifies window-over-screen, explains that crop is for taking 'a part', and frames vision as checking that nothing is distinguished by color alone. Only crop syntax and session naming are left entirely to the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a concrete verb and resource ('Take a picture of the screen, or of one window') and immediately clarifies the output modality ('RETURNED AS AN IMAGE'), which separates it from every text-based sibling. It also names the contrasting sibling get_accessible_tree, so an agent can route without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit selection rules: prefer get_accessible_tree because pictures cost far more, prefer a window over the full screen, crop for a part, and use burst only for motion/animation/transition. When-not-to-use and alternatives are both stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
type_textA
Type text into whatever has the keyboard focus, a key at a time. ASCII only: a character with no key fails the call, once what's before it is typed. For keys and shortcuts (Return, ctrl+s), use press_keys.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | The text to type | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations mark this as a non-readonly but non-destructive write, so the safety profile is covered; the description adds genuinely useful behavior beyond that: typing is per-keystroke into the focused element, and a non-keyable character fails the call at the point reached. It does not disclose what happens to already-typed text on failure or any confirmation/return behavior, keeping it short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tight clauses: purpose first, then the constraint/failure mode, then the sibling routing. Zero filler and well front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter, no-output-schema tool this covers purpose, precondition, failure mode, and the alternative path. The only omission is any statement about success confirmation or partial-typing state, which is minor here.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% (text and session are both documented, with session pointing back to start_session), so the schema already carries the parameter meaning. The description adds no per-parameter syntax or format detail, which is the expected baseline when the schema does the work.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a concrete verb ('type') and resource ('text into whatever has the keyboard focus') plus the granularity ('a key at a time'). It explicitly differentiates itself from the sibling press_keys, so an agent can disambiguate without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It names the alternative tool and the exact condition that selects it ('For keys and shortcuts (Return, ctrl+s), use press_keys'), and states the ASCII-only precondition that governs whether the call should be made at all.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
use_gamepadA
Work a virtual gamepad, in a session started with gamepads: press buttons together, tapped, held for hold seconds, or with state down left down until a call with state up; push a stick to x, y; pull a trigger to value; or neutral, letting go of everything. A stick or trigger stays where it's put, unless hold says for how long. Programs in the session see a real Xbox 360 controller, and none of the machine's. For keys and the mouse, press_keys and click_element.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | The stick, left to right | |
| y | No | The stick, up to down | |
| pad | No | Which of the session's pads (1) | |
| hold | No | Seconds to keep it down before letting go | |
| state | No | Only press, leaving it down until a call with up; or only let go | |
| value | No | The trigger, let go to pulled all the way | |
| which | No | Which stick or trigger | |
| action | Yes | Press buttons, push a stick, pull a trigger, or let go of everything | |
| buttons | No | a b x y lb rb back start guide ls rs, and the d-pad's up down left right | |
| session | Yes | Its name, from start_session |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare this is a non-destructive write, and the description adds real behavioral context beyond them: sticks/triggers persist where placed unless 'hold' bounds the duration, and the session sees a real Xbox 360 controller rather than the host's. It does not discuss permissions or concurrency, but the state-persistence disclosure is the key trait an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the core purpose and then enumerates the four action modes efficiently. The sentence is dense and semicolon-heavy, but each clause carries distinct operational information; nothing is padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 10-parameter, no-output-schema input tool with annotations already covering the read/write profile, the description covers purpose, prerequisite, action semantics, and alternatives. Return-value behavior is the only notable omission, which is minor given the interaction is fire-and-observe.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the bar is a baseline 3, but the description genuinely adds semantics by explaining how state (down until a later up call), hold (seconds kept down), and neutral interact across calls — integration meaning the per-parameter schema text does not convey on its own.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (work) over a specific resource (virtual gamepad) and enumerates the operations (press buttons, push stick, pull trigger, neutral). It explicitly distinguishes itself from siblings by routing keyboard/mouse input to press_keys and click_element, so an agent can select it without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Specifies the prerequisite ('in a session started with gamepads') and names the alternative tools for the adjacent input domains (keys and mouse). It stops short of an explicit when-not statement, but the routing to press_keys/click_element covers the main selection decision.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
wait_forARead-only
Wait, instead of sleeping, for: a window (value: its title, exactly or a regex, or app=RE, or the name of a program spawn_program started, for a window of that program's, whatever its title; giving up, it lists the windows there are), stable (the screen, or the window, unchanged for value seconds, 0.5, up to 60: after a click, for the repaint), an announcement (value: a regex in what was announced to a screen reader; each announcement is waited for once), focus (value: the name of the node with the keyboard focus, as click_element takes one), match (value: a PNG, by its path from the allow file's directory, that the screen, or the window, cropped to crop, should look like), change (from how it looked when the wait started), or line (value: a regex found in a line of file, written after the wait starts, or with from_start, anywhere in it: for a program that logs its events, rather than a sleep; a file not there yet is waited for, and one rotated or rewritten is read from the top). match and change say how long they took; line answers with the line. Fails after timeout seconds, saying so.
| Name | Required | Description | Default |
|---|---|---|---|
| crop | No | WIDTHxHEIGHT+X+Y, inside what's taken | |
| file | No | line's, by its path from the allow file's directory | |
| fuzz | No | match, change: percent a pixel may differ by (0) | |
| most | No | match, change: pixels past fuzz allowed (0) | |
| what | Yes | The kind of wait | |
| value | No | What to wait for, as what says | |
| window | No | A window's title, exactly or a regex matched in full, or app=RE for its app id | |
| session | Yes | Its name, from start_session | |
| timeout | No | Seconds before it fails (10) | |
| from_start | No | line: anywhere in file |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint and openWorldHint, and the description adds real behavioral context: failure after timeout 'saying so', match/change reporting elapsed time, line returning the line, announcements waited for only once, and rotated/rewritten files read from the top. These are substantive traits beyond the structured fields, though the result of a successful window/stable/focus wait is left implicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The content is dense and largely earns its place for a seven-mode tool, but it is delivered as a single sprawling sentence with nested parentheticals, so key facts like the timeout failure and return values are buried rather than front-loaded. Size is defensible; structure is not.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given ten parameters, seven modes, and no output schema, the description covers calling semantics and edge cases (missing/rotated files, one-shot announcements) well and states returns for match, change, and line. It leaves the return behavior of window, stable, announcement, and focus to inference, which is the one remaining gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so per-parameter meaning is already documented and the baseline is 3. The description goes further by tying parameters together across modes — which value format each 'what' requires, that crop applies inside what's taken, that fuzz/most are match/change-only, and that from_start changes line's scan origin.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Wait, instead of sleeping, for:') and then enumerates the seven wait kinds (window, stable, announcement, focus, match, change, line) with the value each expects. That is enough to tell it apart from siblings like take_screenshot or get_accessible_tree, though no sibling is named as an alternative.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It anchors usage against the obvious alternative ('instead of sleeping') and gives per-mode triggers such as 'after a click, for the repaint' for stable and 'for a program that logs its events, rather than a sleep' for line. There is no explicit when-not-to-use guidance or routing between the seven modes, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
27 tool updates
v0.8.1- Removed
accessible_tree - Removed
announcements - Added
check_accessible_names - Added
check_contrast - Added
check_focus_order - Removed
click - Added
click_element - Removed
contrast - Removed
gamepad - Added
get_accessible_tree - Added
get_program_output - Removed
lint - Added
list_announcements - Removed
program_output - Removed
record - Added
record_audio - Added
record_video - Removed
screenshot - Removed
scroll - Added
scroll_wheel - Removed
sound - Removed
spawn - Added
spawn_program - Changed
start_session2 fields changed- changed
Input schema / properties / gamepads / descriptionPrevious value: -"Virtual Xbox 360 pads, for the gamepad tool"New value: +"Virtual Xbox 360 pads, for use_gamepad" - changed
Input schema / properties / screen / descriptionPrevious value: -"WIDTHxHEIGHT (1280x800)"New value: +"WIDTHxHEIGHT (1280x800), up to 7680x4320"
- Removed
tab_walk - Added
take_screenshot - Added
use_gamepad
18 tool updates
v0.7.1- First observed
accessible_tree - First observed
announcements - First observed
click - First observed
contrast - First observed
gamepad - First observed
lint - First observed
press_keys - First observed
program_output - First observed
record - First observed
screenshot - First observed
scroll - First observed
sound - First observed
spawn - First observed
start_session - First observed
stop_session - First observed
tab_walk - First observed
type_text - First observed
wait_for
TDQS
Scored across 18 tools
Most tools target clearly distinct actions (start_session/stop_session, click_element, scroll_wheel, press_keys, type_text, record_video, record_audio, the three check_* linters), and the descriptions explicitly draw boundaries (e.g. type_text vs press_keys, screenshot vs accessible_tree). The main overlap is wait_for, which reimplements waiting on announcements, log lines, focus, and screen changes that other tools also observe, creating some potential misselection.
Every tool uses snake_case with a predictable verb_noun or verb_object pattern: start_session, stop_session, spawn_program, get_program_output, get_accessible_tree, take_screenshot, click_element, scroll_wheel, press_keys, record_video, record_audio, use_gamepad, check_accessible_names, check_contrast, check_focus_order. wait_for is the only slight deviation but remains readable and consistent in style.
18 tools is on the heavier side but each earns its place across the distinct sub-domains of session management, program spawning, input simulation, observation, recording, and accessibility checks. It is slightly large but not padded; nothing appears redundant enough to remove.
The surface covers the full lifecycle: session start/stop, program spawn/output, input (text, keys, clicks, scroll, gamepad), observation (accessible tree, screenshots, video, audio, announcements), flexible waits, and three accessibility linters. Coverage is comprehensive for a GUI/accessibility testing domain with no obvious dead ends.
Maintenance
Related MCP Connectors
Test a web app in a real browser from plain English. Passed, failed or cannot verify, with evidence.
Browser-based QA for AI-built software. Test pages with real browsers via agents.
Website checker for AI coding agents. Measures your site in a real browser, with evidence.
Accessibility pre-checks (WCAG/BFSG) in a real browser + statement drafts. Pay per call.
Related MCP Servers
- AlicenseNot gradedqualityAmaintenanceEnables AI agents to automate Linux desktop GUI by launching and interacting with Wayland applications in isolated virtual KWin sessions, or connecting to live desktops for collaborative automation.155 PyPI76MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI coding agents to autonomously interact with and test web applications in a real browser, providing DOM/Accessibility tree extraction, runtime telemetry, screenshot capture, and Markdown test reports.12 npm1MIT
- AlicenseAqualityBmaintenanceEnables AI agents to operate a real GNOME Wayland desktop through accessibility-tree widget actions, pointer/keyboard input, OCR, window management, and screen capture, optionally on a private headless session.3331 PyPI11Apache 2.0
- AlicenseNot gradedqualityAmaintenanceEnables AI agents and automation tools to interact with GUI applications in isolated X11 displays via MCP, supporting input, window management, screenshots, clipboard, and accessibility actions.4MIT