computer-use
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@computer-useTake a screenshot of the current display"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
plugin-computer-use
A persistent stdio MCP server that exposes the Anthropic
computer-use
action surface (screenshot, click, move, keyboard, clipboard, batch) against the
SideButton agent desktop on DISPLAY=:10.
This repo is the scaffold + dispatch core for the Computer Use epic (SCRUM-1399). It is delivered by SCRUM-1397:
the long-lived stdio MCP server loop (
initialize/tools/list/tools/call),the ported
computer.pydispatch base (DISPLAY targeting, screenshot → base64 PNG, coordinate scaling, single-owner lock, xdotool runner),the full tool surface declared so
tools/listreturns it,screenshotwired end-to-end as the proof action.
The individual tool bodies land in sibling tickets (SCRUM-1400…1405) and
hosting this as a runtime: "service" plugin is SCRUM-1406.
Why a persistent server
The current SideButton plugin model
(the-assistant packages/server/src/plugins)
spawns a fresh, stateless handler process per tools/call and SIGKILLs it at
a 30s timeout. That cannot host the computer-use surface, which needs cross-call
state: a held mouse button (left_mouse_down … left_mouse_up), the
screenshot→coordinate session, session grants, and holds up to ~100s. So this is
a single, long-lived child process that speaks MCP over stdio.
Related MCP server: desktop-touch-mcp
Tool surface
24 tools, grouped by the sibling ticket that owns each body. The capture group
(screenshot, zoom, SCRUM-1400), the click group (left_click / right_click
/ middle_click / double_click / triple_click, SCRUM-1401), the keyboard
group (type / key / hold_key, SCRUM-1403), and the clipboard + session
group (SCRUM-1404) are implemented; the rest are declared and return a clear
pending-owner error until their ticket lands. Full input schemas:
docs/computer-use-mcp-tools-schema.md.
Group | Ticket | Tools |
capture | SCRUM-1400 |
|
click | SCRUM-1401 |
|
move / drag / scroll | SCRUM-1402 |
|
keyboard | SCRUM-1403 |
|
clipboard + session | SCRUM-1404 |
|
utility / batch | SCRUM-1405 |
|
Clipboard + session behaviour (SCRUM-1404)
The macOS session/permission model has no XFCE/Xvfb equivalent, so these degrade gracefully instead of erroring — keeping cross-runner (macOS-authored) skills working — while honouring the native grant flags so call shapes match:
request_accessauto-grants the requestedapps(no compositor dialog), records theclipboardRead/clipboardWrite/systemKeyCombosflags (additive across calls), and returnsscreenshotFiltering: false.list_granted_applicationsechoes the allowlist + active grant flags.read_clipboard/write_clipboardshell out toxclip -selection clipboard, gated on theclipboardRead/clipboardWritegrants (a call without the grant returns anisErrorresult, matching native).open_applicationis best-effort window focus (wmctrl -a, thenxdotool search --name … windowactivate); the primary target is the single RDP window. With neither binary installed it returns a non-error no-op note.switch_displayis a no-op on the single Xvfb:10and reports the current display (accepts"auto").
Surface count. This is the 24-tool surface the epic (SCRUM-1399) specifies. The clipboard + session group follows the explicit enumeration in SCRUM-1404 (
read_clipboard/write_clipboardsplit +list_granted_applications), which is the 2-tool delta over the work plan's interim count of 22.src/tools.pyis the single source of truth;docs/computer-use-mcp-tools-schema.md(AC4) is generated from it.
Bare names + collisions. Names are the canonical Anthropic action ids.
screenshot,type,scroll,wait,clickcollide with core SideButton MCP tools, and the current loader drops the entire plugin on any collision. That is fine standalone (this server owns its namespace); namespacing on aggregation is deferred to SCRUM-1406 (recommended: bare names in the child, prefix/slug-namespace on the host).
Layout
plugin-computer-use/
├── plugin.json # generated service-plugin manifest (proposes runtime:"service")
├── src/
│ ├── server.py # stdio MCP loop: initialize / tools/list / tools/call
│ ├── computer.py # dispatch base (ported computer.py)
│ └── tools.py # canonical tool surface (single source of truth)
├── scripts/
│ └── build_manifest.py # regenerates plugin.json + the schema doc from tools.py
├── tests/ # unittest: dispatch-base unit + stdio round-trip + manifest
├── docs/
│ └── computer-use-mcp-tools-schema.md # generated; the AC4 schema doc
├── run_tests.sh # runs the suite (xvfb-wrapped when no DISPLAY)
├── pyproject.toml # dependency-free, python>=3.10
├── README.md LICENSE .gitignoreRun it standalone
# speak MCP by hand (newline-delimited JSON-RPC):
printf '%s\n' \
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' \
'{"jsonrpc":"2.0","id":2,"method":"tools/list"}' \
'{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"screenshot","arguments":{"save_to_disk":true}}}' \
'{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"zoom","arguments":{"region":[600,300,900,500]}}}' \
'{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"type","arguments":{"text":"hello"}}}' \
'{"jsonrpc":"2.0","id":6,"method":"tools/call","params":{"name":"key","arguments":{"text":"ctrl+a","repeat":1}}}' \
'{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":"hold_key","arguments":{"text":"shift","duration":2}}}' \
'{"jsonrpc":"2.0","id":8,"method":"tools/call","params":{"name":"left_click","arguments":{"coordinate":[600,300],"text":"ctrl"}}}' \
| DISPLAY=:10 python3 src/server.pyinitialize returns the handshake, tools/list the 24-tool surface, the
screenshot call a base64 PNG image block (plus a Saved to disk: <path> text
block when save_to_disk is set), and zoom a magnified PNG of the region. The
keyboard and click calls each return a short text ack (isError:false); they need
xdotool on PATH. The left_click maps its [600, 300] against the id:3
screenshot's coordinate session — a click before any screenshot returns a clear
no screenshot session yet error instead of clicking blind.
Capture & coordinates
screenshot captures DISPLAY=:10 and, when the measured size matches a model
resolution, downscales it (on the live 1920×1080 :10 it returns 1366×768).
Each capture records a screenshot → coordinate session: the measured device
geometry and the returned image geometry. Coordinates the model returns are in
image space (relative to the last screenshot); the server maps them back to
device pixels via Computer.to_device(x, y) — the foundation the click/move
siblings (SCRUM-1401/1402) consume. Both the downscale and the coordinate mapping
are derived from the same measured geometry, so they can never use different
bases (the wrong-pixel-click failure mode).
zoom takes region: (x0, y0, x1, y1) in image space, maps it to a device rect,
and crops it from a fresh full-resolution capture — genuine magnification, not
an upscale of the downscaled screenshot. It is read-only: it never moves the
click-coordinate origin (clicks still refer to the last screenshot). If no
screenshot has been taken yet, zoom establishes the session lazily.
True 1:1 (no downscale) would require pinning :10 / the RDP window to a
model-friendly size — that is provisioning (SCRUM-1396),
out of scope here.
Click group (SCRUM-1401)
Pointer clicks at a screenshot-session coordinate. The [x, y] coordinate
is image space (relative to the last screenshot) and is mapped to device pixels
via Computer.to_device — a click before any screenshot returns a clear
no screenshot session yet error (look before you click). Optional text
modifier(s) ('ctrl', 'shift+alt', …) are held for the click and always
released (keyup in a finally, the same guarantee as hold_key).
Tool | xdotool | Button |
|
| left (1) |
|
| right (3) |
|
| middle (2) |
|
| left ×2 |
|
| left ×3 |
With a modifier the click is wrapped in keydown -- <text> → click → keyup -- <text>. On-screen pixel accuracy is validated live in
SCRUM-1408 (xdotool is absent
on the current runner image, so the unit tests assert the device-pixel argv).
Keyboard group (SCRUM-1403)
Tool | xdotool | Notes |
|
| types |
|
| chords, e.g. |
|
| the hold runs in the persistent server (Python |
Test
./run_tests.sh # uses $DISPLAY if set, else wraps in xvfb-run
# or directly:
DISPLAY=:10 python3 -m unittest discover -s tests -vtests/test_dispatch_base.py— coordinate-scaling math, the screenshot → coordinate session +to_devicemapping, the measured-basis downscale target,zoomregion validation + region→device-rect math, xdotool command construction, single-owner lock, screenshot-backend detection, surface shape, plus livescreenshot/zoom+save_to_disk(DISPLAY-gated).tests/test_stdio_roundtrip.py—initialize→tools/list→tools/callscreenshot(incl.save_to_diskpath block) andzoomover a spawned server, plus error paths.tests/test_manifest.py—plugin.json+ schema doc are present and in sync withsrc/tools.py.
The screenshot round-trip needs an X display; run_tests.sh provides one via
xvfb-run when $DISPLAY is unset, so AC3 still exercises in headless CI.
System dependencies
System packages (apt), not pip — the plugin install copies no node_modules/venv
and runs no build step, so the server is stdlib-only and shells out to:
Tool | Used for | Notes |
a screenshot backend |
|
|
| pointer/keyboard actions; | required by the click/move/keyboard groups (siblings); absent on the runner image. |
|
| already on the runner; grant-gated. |
|
| best-effort; |
scrot and gnome-screenshot are absent on the runner image, so the
screenshot backend falls through to ImageMagick import -window root (verified
on DISPLAY=:10). When SCRUM-1407 adds this plugin to the agent-runners catalog,
declare xdotool, a screenshot backend, and xclip in its system_deps.
DISPLAY and single-owner
The server targets the inherited
$DISPLAY, defaulting to:10(the runner desktop). It never hardcodes a display — the screen-record plugin's bug was capturing a non-existent:1.0.It takes a process-lifetime single-owner lock (
flock,/tmp/sidebutton-computer-use.lock, override withCU_LOCK_PATH) so only one session drives the shared pointer/keyboard; a second instance exits non-zero.
Service-manifest contract (SCRUM-1406)
plugin.json targets the merged runtime: "service" tier: the SideButton server
keeps the child alive, discovers its tools via tools/list, and forwards
tools/call to it.
{
"name": "computer-use",
"runtime": "service",
"service": {
"command": "python3 src/server.py", // non-empty string; the engine splits on
// whitespace and spawns with cwd=plugin dir
"toolNamespace": "computer_use", // tools surface as computer_use_<tool>
"tools": { // per-tool timeout overrides (ms)
"hold_key": { "timeoutMs": 120000 },
"wait": { "timeoutMs": 120000 }
}
},
"tools": [] // service plugins declare no static tools
}The loader (
the-assistantpackages/server/src/plugins/loader.ts) recognizes onlycommand/timeoutMs/toolNamespace/toolsunderservice, and hard-rejects the manifest unlesscommandis a non-empty string — an array fails validation and the plugin never loads. Tools are discovered live, so the top-leveltoolsarray is normalized to[]. This repo owns onlyplugin.json; the agent-runners catalog entry +system_depsare SCRUM-1407.
Configuration (env)
Var | Default | Purpose |
|
| target X display |
|
| screen size for coordinate scaling |
|
| post-action settle before a screenshot |
|
| single-owner lock file |
|
| where |
License
MIT © 2026 SideButton
Available Tools
24 toolscomputer_batchB
Run a sequence of computer-use actions in order and return one combined result.
| Name | Required | Description | Default |
|---|---|---|---|
| actions | Yes | Ordered list of {name, arguments} actions. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description must disclose behavioral traits. It only states 'in order and return one combined result' but does not specify error handling, atomicity, or side effects like screen changes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no unnecessary words. It is front-loaded and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
As a batch tool with no output schema, the description should explain the nature of the combined result and how errors are handled. It is incomplete for a tool that coordinates multiple actions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% coverage with a description for the actions parameter. The tool description adds 'in order', which reinforces the schema, but does not add significant new meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'run' and the resource 'sequence of computer-use actions', with the result 'one combined result'. This distinguishes it from individual action tools like left_click or type.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies batch usage but provides no explicit guidance on when to use this tool versus calling individual actions sequentially. It does not mention prerequisites or when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
cursor_positionA
Return the current cursor [x, y] in model coordinates.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Describes exact return format and coordinate system. No annotations exist, but the description covers the core behavior adequately. Could mention that it is non-destructive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence with no extraneous information. Front-loaded with key action and output.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a zero-parameter query tool. Output format is specified. No output schema needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters, so schema fully covers. Description adds value by specifying 'model coordinates' for the output.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clear verb 'Return' and specific resource 'current cursor [x, y]' in model coordinates. Distinct from sibling action tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use versus alternatives. Implicitly a read-only query, but no when-not or context provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
double_clickB
Double left-click at a coordinate.
| Name | Required | Description | Default |
|---|---|---|---|
| coordinate | Yes | [x, y] in the model coordinate space (scaled to the screen). | |
| text | No | Optional modifier key(s) to hold during the click, e.g. 'ctrl' or 'shift+alt'. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must fully disclose behavior. It only states the basic action without mentioning nuances like whether it performs a full press-release-press-release sequence, any inherent delays, or side effects. This is insufficient for an agent to predict the tool's impact.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that efficiently conveys the core action. No redundant information is present, making it optimally concise for a straightforward tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and the absence of an output schema, the description is moderately complete. It defines the action and parameters, but lacks context about the typical behavior of double-clicking (e.g., triggering default actions) which would help the agent choose this tool over siblings.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides complete descriptions for both parameters (coordinate and text). The tool description adds no additional meaning beyond restating 'Double left-click at a coordinate.' Since schema coverage is 100%, a baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action 'Double left-click at a coordinate.' This distinguishes it from siblings like left_click or triple_click by specifying the double-click action. However, it doesn't elaborate on what double-clicking does in context, missing a chance to differentiate further.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. For example, when a double-click is preferred over a single click or triple click, or that it might be used to open files or invoke default actions. The agent receives no contextual cues.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
hold_keyC
Hold a key (or chord) down for a duration in seconds.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| duration | Yes | Seconds to hold (may be up to ~100s). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description only says 'Hold down' but does not clarify whether the key is automatically released after the duration, how chords are specified (e.g., separator), or any side effects. With no annotations, more detail is needed for safe usage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, front-loaded with key information. It is concise but could include more detail without becoming verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 2 required parameters, no output schema, and no annotations, the description is insufficient. It omits the required format for 'text', behavior after duration, and any constraints, making it incomplete for an agent to use correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 50% (only duration has a description). The description adds no meaning to the 'text' parameter (e.g., format for chords) beyond the schema. The tool description does not compensate for the missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Hold a key (or chord) down for a duration in seconds' explicitly states the verb (hold), resource (key or chord), and duration, clearly distinguishing it from sibling tools like 'key' or 'type' which likely involve pressing and releasing or typing characters.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use hold_key versus alternatives such as 'key' or 'type'. The description does not indicate context, prerequisites, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
keyA
Press a key or chord using xdotool key syntax, e.g. 'Return', 'ctrl+s', 'alt+Tab'.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, and the description only says 'press', failing to disclose whether it presses and releases, any prerequisites (e.g., window focus), or potential side effects. The behavioral detail is minimal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with examples, containing no redundant information. Every part earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool, the description covers purpose and syntax adequately but lacks information on return values or system behavior (e.g., whether it works globally). Without an output schema, some completeness is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The sole parameter 'text' has 0% schema coverage, but the description adds significant meaning by specifying xdotool key syntax and providing examples, effectively guiding the agent on parameter formatting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states the tool presses a key or chord using xdotool key syntax, with concrete examples ('Return', 'ctrl+s', 'alt+Tab'), making the purpose clear and distinct from siblings like 'hold_key' or 'type'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Examples imply usage but no explicit guidance on when to use this tool over alternatives like 'hold_key' or 'type'. It does not specify context or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
left_clickC
Left-click at a coordinate.
| Name | Required | Description | Default |
|---|---|---|---|
| coordinate | Yes | [x, y] in the model coordinate space (scaled to the screen). | |
| text | No | Optional modifier key(s) to hold during the click, e.g. 'ctrl' or 'shift+alt'. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It only states the action without effects, prerequisites, or outcomes (e.g., whether the cursor moves, if the click is instant, or if it can fail). This is insufficient for agent understanding.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise—one front-loaded sentence with no wasted words. It efficiently conveys the core action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of output schema and annotations, the description should explain what happens after execution (e.g., return value, success/failure behavior). It does not, leaving an incomplete picture for the agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds no parameter meaning beyond the schema, which already explains coordinate as [x,y] in model space and text as optional modifier keys.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it performs a left-click at a coordinate. It specifies the verb 'click' and the resource 'coordinate', but does not distinguish from siblings like left_click_drag or double_click, which also involve left-clicking.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., left_click_drag, double_click). There is no mention of context or exclusions, leaving the agent without direction for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
left_click_dragA
Press the left button at start_coordinate (or the current position) and drag to coordinate before releasing.
| Name | Required | Description | Default |
|---|---|---|---|
| start_coordinate | No | [x, y] in the model coordinate space (scaled to the screen). | |
| coordinate | Yes | [x, y] in the model coordinate space (scaled to the screen). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits. It describes the basic sequence (press, drag, release) but lacks details about timing, acceleration, or what happens if start_coordinate is omitted. A more thorough description would mention edge cases or potential side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with no wasted words. It front-loads the essential action and uses parentheses for optional behavior, which is well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity, 100% schema coverage, and lack of output schema, the description is largely complete. It explains the start and end points. However, it could mention that this is a mouse drag action (e.g., for moving objects) to improve intuitive understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Both parameters have schema descriptions with coordinate format details. The tool description adds value by noting that start_coordinate is optional and defaults to the current position, which is not fully captured in the schema alone (schema makes coordinate required but doesn't clarify start_coordinate's role).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action: pressing the left button at start_coordinate (or current position) and dragging to a target coordinate. It uses a specific verb and resource, distinguishing it from related tools like left_click, mouse_move, and left_mouse_down.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for drag operations but does not explicitly state when to use this tool versus alternatives like left_mouse_down + mouse_move + left_mouse_up or other click-and-drag methods. No guidance on prerequisites or when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
left_mouse_downA
Press and hold the left mouse button (released later by left_mouse_up). Requires the persistent session.
| Name | Required | Description | Default |
|---|---|---|---|
| coordinate | No | [x, y] in the model coordinate space (scaled to the screen). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It clearly discloses the hold behavior and that release requires left_mouse_up, which is key. The persistent session requirement is also stated. Could mention potential issues if already pressed, but overall adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence that efficiently conveys the action and the prerequisite. No unnecessary words or details. Perfectly front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter tool with no output schema, the description covers the essential behavior, coordinate meaning, and session requirement. It also references the companion tool for release. Minor omission: no mention of constraints like coordinate bounds or timeout.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides a complete description of the coordinate parameter (format, constraints, meaning). The description adds no further semantic value beyond what the schema states, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states 'Press and hold the left mouse button' with a clear verb and resource. It distinguishes from sibling tools like left_click (press and release) and left_click_drag (press, move, release) by noting the hold action and release via left_mouse_up.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when a persistent mouse press is needed, but no explicit when-to-use or when-not-to-use guidance is given. It mentions the persistent session requirement but does not contrast with alternatives like left_click or left_click_drag.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
left_mouse_upB
Release a left mouse button held by left_mouse_down.
| Name | Required | Description | Default |
|---|---|---|---|
| coordinate | No | [x, y] in the model coordinate space (scaled to the screen). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It does not disclose behavioral traits such as what happens if the button is not held, effects of the coordinate parameter, or any side effects beyond the release action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Very concise single sentence, but minimal structure. Adequate for a simple tool, but lacks any additional context that could help agent understanding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool is simple and has no output schema, the description could still clarify the relationship with left_mouse_down and coordinate usage. Missing details reduce completeness for confident invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% for the single parameter 'coordinate', so baseline 3 applies. The description does not add additional meaning beyond what is already in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('release') and the specific resource ('left mouse button held by left_mouse_down'), using a specific verb+resource. It implicitly distinguishes from sibling tools like left_mouse_down and left_click.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool instead of alternatives. It only mentions the relationship to left_mouse_down but does not specify context like required prior actions or error handling.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_granted_applicationsA
Return the set of applications currently granted desktop access (Linux stub: echoes the granted set).
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses the read-only nature and includes a note about Linux behavior. However, it lacks details on side effects, permissions, or what constitutes 'desktop access'.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no fluff. It is appropriately sized and front-loaded with the core purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with no parameters and no output schema, the description covers the basic functionality. It mentions the Linux stub behavior, but could be improved by clarifying the scope of 'desktop access'.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero parameters, so schema coverage is 100%. Baseline for 0 parameters is 4. The description does not need to add parameter details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Return') and the resource ('set of applications currently granted desktop access'). It effectively distinguishes from sibling tools like 'request_access' and 'open_application', which involve different operations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives such as 'request_access' or 'open_application'. The description only states what it does without usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
middle_clickC
Middle-click at a coordinate.
| Name | Required | Description | Default |
|---|---|---|---|
| coordinate | Yes | [x, y] in the model coordinate space (scaled to the screen). | |
| text | No | Optional modifier key(s) to hold during the click, e.g. 'ctrl' or 'shift+alt'. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description carries full burden. It discloses only that a middle-click occurs at a coordinate, but not whether it simulates a real click, any side effects, or system requirements. The modifier key parameter description adds minor transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise—a single sentence with no unnecessary words. It is efficiently front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the large set of sibling click tools and no output schema, the description is too minimal. It does not explain typical use cases (e.g., opening links in a new tab) or return behavior, leaving the agent without sufficient context for correct selection.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with both parameters having descriptions. The tool-level description adds no extra parameter meaning, but the schema already provides adequate explanation (e.g., coordinate format, modifier keys). Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (middle-click) and the target (at a coordinate), distinguishing it from other click tools like left_click or right_click. However, it is minimal.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use middle-click versus alternatives (e.g., left_click, double_click). Usage is only implied, with no exclusions or context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
mouse_moveA
Move the pointer to a coordinate without clicking.
| Name | Required | Description | Default |
|---|---|---|---|
| coordinate | Yes | [x, y] in the model coordinate space (scaled to the screen). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It discloses that the tool moves without clicking, but omits information about animation, boundary behavior, or side effects. Minimal but not misleading.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, 10 words, front-loaded with the verb and resource. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple move tool with one parameter and no output schema, the description is minimally adequate. It does not explain the coordinate space scaling or potential restrictions, but the schema covers the parameter.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with coordinate described as [x,y] in model coordinate space. The description adds no further parameter meaning beyond the schema, aligning with the baseline of 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (move), the resource (pointer), and a specific constraint (without clicking). This distinguishes it from clicking siblings like left_click.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as left_click_drag or scroll. With 22 sibling tools, the description should provide usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
open_applicationB
Launch or focus a desktop application by name.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It mentions 'launch or focus' which implies behavior, but does not disclose error handling (e.g., app not found), permissions, or side effects like potential blocking.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded with action, no wasted words. Appropriate size for the tool's simplicity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, no annotations, and a single parameter, the description lacks details on return values or error conditions. An agent cannot determine success or failure handling.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description only says 'by name', which adds minimal context beyond the parameter name. Does not specify format, case sensitivity, or possible values.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the tool launches or focuses a desktop application by name, providing a specific verb and resource. It distinguishes from sibling tools like 'list_granted_applications' which lists apps rather than launching them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives (e.g., when the application might already be running). No exclusions or prerequisites mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
read_clipboardA
Read the X clipboard contents (via xclip).
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. Description mentions the underlying tool (xclip) but does not disclose behavior on empty clipboard, missing xclip, or side effects. Minimal transparency for a simple read operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, no wasted words, directly conveys purpose and mechanism. Excellent conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Simple tool with no output schema or parameters. Description mentions the specific clipboard source (X clipboard) and tool (xclip). Lacks detail on return format but adequate for a straightforward read.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist, so schema coverage is 100%. Description adds no param info, which is acceptable per guidelines. Baseline score of 4 for zero-parameter tools.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Read the X clipboard contents (via xclip)', providing a specific verb, resource, and mechanism. It effectively distinguishes from sibling 'write_clipboard'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives (e.g., when to read vs write clipboard). No context about prerequisites or situations where reading might fail.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
request_accessB
Request a session grant for one or more applications (Linux stub: auto-grants and returns screenshotFiltering=false; the real grant model lands with the service engine).
| Name | Required | Description | Default |
|---|---|---|---|
| applications | No | Applications to request access to. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description carries full burden. It discloses the Linux stub behavior and future model, but lacks details on non-Linux platform behavior, error handling, or side effects. Partially transparent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One concise sentence with a parenthetical that adds important context. No fluff, but the parenthetical might be slightly awkward. Good front-loading.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and one parameter, the description covers Linux stub behavior but omits essential context like non-Linux behavior, error conditions, grant lifecycle, and usage on other platforms. Incomplete for an access-granting tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a single parameter 'applications' described as 'Applications to request access to.' The description adds no additional meaning beyond 'one or more applications,' which is already clear from the schema. Baseline 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: to request a session grant for applications. It provides specific verb ('Request'), resource ('session grant'), and distinguishes from siblings like list_granted_applications by mentioning Linux stub behavior.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. It describes Linux stub behavior but does not specify prerequisites, when not to use, or differences from sibling tools like open_application or list_granted_applications.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
right_clickC
Right-click at a coordinate.
| Name | Required | Description | Default |
|---|---|---|---|
| coordinate | Yes | [x, y] in the model coordinate space (scaled to the screen). | |
| text | No | Optional modifier key(s) to hold during the click, e.g. 'ctrl' or 'shift+alt'. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It only states the action without explaining behavioral traits like whether it requires focus, what coordinate system is used (though the param schema notes model space), or what effects occur (e.g., opening context menu). The description adds no behavioral context beyond the name.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise at just 4 words, but it is appropriately structured for a simple action. However, it could be more informative without becoming verbose, so it scores average.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the low complexity and complete schema coverage, the description is still too terse. It does not explain the return value (e.g., success indicator or new cursor position) or how it integrates with other tools (e.g., requiring prior mouse movement). This leaves the agent with incomplete information.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters: coordinate as [x,y] in model space and text as optional modifier keys. The tool description does not add any additional meaning or usage nuances beyond what the schema provides, resulting in a baseline score of 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it performs a right-click at a coordinate. The verb 'right-click' and the resource 'coordinate' are specific. However, it does not differentiate from sibling tools like left_click or double_click, which have similar structures.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives like left_click or triple_click. The description does not mention any context or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
screenshotB
Capture the active display (DISPLAY=:10) and return a base64-encoded PNG. Implemented in SCRUM-1397 as the proof action.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. Description only specifies the display (DISPLAY=:10) and adds an implementation reference, but fails to disclose permissions, failure modes, or effects like whether it captures full screen or window.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences; the first is essential and clear. The second sentence adds implementation context but is not critical. Still concise overall.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, so description should provide more detail on the returned data (e.g., size, details) or behavior (e.g., asynchronous, permissions). States only output format. Incomplete for a tool with zero annotations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters, so baseline 4. The description already states the output format, which is useful beyond the empty schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states verb 'capture', resource 'active display', and output format 'base64-encoded PNG'. Distinguishes from sibling tools that perform other actions like clicking or moving.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives. Does not mention when not to use or provide context for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrollC
Scroll in a direction by an amount at a coordinate.
| Name | Required | Description | Default |
|---|---|---|---|
| coordinate | Yes | [x, y] in the model coordinate space (scaled to the screen). | |
| scroll_direction | Yes | ||
| scroll_amount | Yes | Number of scroll 'clicks'. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Lacks details on effects of off-screen coordinates, scroll amount scaling, or side effects. Annotations absent, so description carries full burden but falls short.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence is concise and front-loaded, but slightly too brief for a tool with three parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a trivial scroll action but omits crucial details like coordinate system, units, and limitations, leaving room for misuse.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage 67%; description adds no extra meaning. The scroll_direction enum lacks any explanation, and coordinate/scroll_amount descriptions are already in schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the action 'scroll' with direction, amount, and coordinate, distinguishing it from sibling mouse tools like click or move.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use scroll versus alternatives like mouse_move or click-drag. No prerequisites or context provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
switch_displayC
Switch the display the session targets, e.g. ':10'.
| Name | Required | Description | Default |
|---|---|---|---|
| display | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description is responsible for disclosing behavior. It only states the action and an example, but does not mention effects on the session, safety (destructive or not), or any side effects. This is insufficient for a mutation-like tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely short (one sentence), but it omits critical information, making it under-specified rather than concise. Every sentence should add value; here it fails to provide sufficient context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity (no nested objects, no output schema), the description should be complete. It is not: missing parameter format, usage context, and behavioral details. The tool's job is clear but the description is incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must explain the parameter. It gives an example (':10') but does not describe the format, valid values, or how the 'display' string is interpreted. The agent lacks needed semantic detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('switch the display') and provides an example format (':10'), making the purpose easy to understand. However, the term 'session targets' could be more explicit, but it's still adequate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is given on when to use this tool versus the many sibling tools (e.g., mouse_move, scroll). The description does not mention prerequisites or conditions for usage, leaving the agent to infer context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
triple_clickB
Triple left-click at a coordinate.
| Name | Required | Description | Default |
|---|---|---|---|
| coordinate | Yes | [x, y] in the model coordinate space (scaled to the screen). | |
| text | No | Optional modifier key(s) to hold during the click, e.g. 'ctrl' or 'shift+alt'. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; the description is minimal and does not disclose any behavioral traits beyond the bare action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is short and front-loaded; every word is necessary, but could be slightly more structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the low complexity and no output schema, the description is minimally complete but lacks guidance on behavior or use cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3; the description adds no additional meaning beyond what the schema already provides for the parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Triple left-click at a coordinate' clearly specifies the action and target, distinguishing it from siblings like double_click and left_click.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use triple click versus alternatives; the description only states the action without context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
typeB
Type a string of text at the current focus.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Since no annotations are provided, the description must fully disclose behavior. It mentions the tool types 'at the current focus', but does not specify what happens if no element is focused, how special characters are handled, or whether the action is destructive. Additional context about keystroke simulation or speed is missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no extraneous information. Every word is necessary and contributes to understanding the tool's purpose. It is appropriately concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description provides the core functionality. However, it lacks edge-case handling (e.g., no focus), behavioral details, and usage context. It is minimally adequate but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter 'text' has no schema description (0% coverage), so the description must compensate. It describes it as 'a string of text', which adds little beyond the schema. No details on format, length limits, or allowed characters are provided.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Type' and the resource 'a string of text', with context 'at the current focus'. This distinguishes it from sibling tools like 'key' (single key presses) and 'write_clipboard' (clipboard operations).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as 'key' for individual keystrokes or 'hold_key' for modifier keys. The description does not mention scenarios or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
waitA
Wait for a duration in seconds (then optionally screenshot).
| Name | Required | Description | Default |
|---|---|---|---|
| duration | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses the primary behavior (wait for duration, optionally screenshot) but lacks details on blocking, error handling, or edge cases.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with no excess. Every word contributes to understanding the tool's function.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description covers the essential behavior and optional screenshot. It could mention return behavior (likely none) but is largely complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds value by specifying 'in seconds' for the duration parameter, which the schema (number, required) does not provide. This clarifies the unit and usage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'wait' and the resource 'duration in seconds', with an optional screenshot action. It distinguishes itself from sibling tools as the only wait-related tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use or avoid the tool. No mention of alternatives or prerequisites, leaving the agent to infer usage from context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
write_clipboardB
Write text to the X clipboard (via xclip).
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses the action and context but does not specify that it overwrites the current clipboard content, nor does it mention permissions or side effects. With no annotations, more behavioral detail is expected.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence that is clear and front-loaded. Every word serves a purpose with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description is mostly adequate. However, it lacks usage guidelines and parameter elaboration, leaving some gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With schema coverage at 0%, the description does not add any meaning to the 'text' parameter beyond the schema. It could clarify length limits, encoding, or format but does not.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Write', the resource 'text to the X clipboard', and the mechanism 'via xclip'. It immediately distinguishes from the sibling 'read_clipboard' tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives like 'type' or 'key'. No mention of when not to use it or prerequisite conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
zoomB
Capture and return a magnified PNG of a sub-region of the screen.
| Name | Required | Description | Default |
|---|---|---|---|
| region | Yes | [x, y, width, height] region to magnify. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It states the output is a 'magnified PNG' but does not disclose the magnification factor, potential permissions, or whether the region is relative to the screen or application. Insufficient detail for safe invocation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence that is clear and free of superfluous content. Every word contributes to understanding the tool's function.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one parameter, no output schema, no annotations), the description covers the basic function. However, it omits important details like the output format (file path vs. base64), magnification factor, and how to interpret coordinates. A 3 reflects that it is minimally adequate but has clear gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% for the 'region' parameter, so the baseline is 3. The description adds no extra meaning beyond the schema, only reiterating the purpose. It does not clarify coordinate system or constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Capture and return a magnified PNG') and the resource ('sub-region of the screen'). The verb 'Capture' and modifier 'magnified' differentiate it from the sibling 'screenshot' tool, but it could be more explicit about the distinction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like 'screenshot' or 'mouse_move'. There are no exclusions or contextual cues for appropriate usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
24 tool updates
v0.1.0- First observed
computer_batch - First observed
cursor_position - First observed
double_click - First observed
hold_key - First observed
key - First observed
left_click - First observed
left_click_drag - First observed
left_mouse_down - First observed
left_mouse_up - First observed
list_granted_applications - First observed
middle_click - First observed
mouse_move - First observed
open_application - First observed
read_clipboard - First observed
request_access - First observed
right_click - First observed
screenshot - First observed
scroll - First observed
switch_display - First observed
triple_click - First observed
type - First observed
wait - First observed
write_clipboard - First observed
zoom
TDQS
Scored across 24 tools
Most tools have distinct purposes like left_click vs right_click vs double_click, but some overlap exists (e.g., left_click_drag vs left_mouse_down/move/up sequence) and could cause confusion despite clear descriptions.
Naming is mostly snake_case but inconsistent in verb_noun patterns: some are verbs (type, scroll), some nouns (screenshot, key), and some are compound (computer_batch). Not chaotic but lacks a uniform style.
24 tools is reasonable for a computer-use server covering mouse, keyboard, clipboard, screenshots, and display management. Not too many, not too few.
Covers core desktop automation actions well (clicks, drag, keyboard, clipboard, screenshots, scrolling, application launching). Minor gaps like file operations or advanced gestures are acceptable.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
- QuallaaOAuthcom.quallaa
Talk to your public-facing AI from any MCP client — Claude, ChatGPT, Cursor, Cline, Windsurf.
OCR, transcription, file extraction, and image generation for AI agents via MCP.
Shared memory and actions for Claude, Kiro, OpenAI, Cursor, and other MCP-compatible AI clients.
An agent-first office suite Claude & ChatGPT read and write over one MCP URL.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceAn MCP server that gives any AI assistant eyes and hands on your desktop — screenshots, clicking, typing, OCR, window management, accessibility-tree queries, workflow recording.5Apache 2.0
- AlicenseAqualityAmaintenanceAllows AI clients to see and control Windows 10/11 desktops via MCP, with screenshots, UI Automation, Chrome CDP, keyboard/mouse, and terminal using semantic element targeting.30383MIT
- AlicenseNot gradedqualityAmaintenanceEnables Windows desktop automation via MCP, allowing AI agents to control mouse, keyboard, and screen capture with the same interface as Anthropic's computer-use tool.3MIT
- AlicenseNot gradedqualityCmaintenanceA framework-agnostic computer-use MCP server that exposes core desktop operations (screen capture, mouse, keyboard, and file access) as standard MCP tools, enabling any MCP-compatible agent to drive a computer.347MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/sidebutton/plugin-computer-use'
If you have feedback or need assistance with the MCP directory API, please join our Discord server