use-computer-mcp
Provides desktop automation for a GNOME Shell session on Wayland, including screenshots, mouse and keyboard control, accessibility tree inspection, OCR, clipboard access, app launching, and audio monitoring.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@use-computer-mcpTake a screenshot and tell me what's on the screen"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
use-computer
Lets an agent (Claude, via MCP) see and operate a GNOME desktop on Wayland the way a person does: screenshots, mouse, keyboard, the accessibility tree with clickable refs, OCR, the clipboard, and listening to whatever the computer is playing. Think Claude in Chrome, but for the whole desktop instead of one browser tab.
It ships as three things that share one daemon: an MCP server (use-computer-mcp), a CLI
(use-computer), and a Claude skill (skills/use-computer) that teaches the workflow.
Real limitations, up front:
GNOME Shell on Wayland only. It uses Mutter's private RemoteDesktop/ScreenCast D-Bus API. KDE, Sway, Hyprland and X11 sessions are not supported and nothing here pretends to be.
Single monitor (the primary one) today.
It drives your real session. The pointer moves; typing goes to the focused window. Don't use the machine at the same time.
Accessibility coverage depends on the app. GTK/GNOME apps and Firefox are good; Chromium/Electron apps expose little unless started with
--force-renderer-accessibility; games and canvases are pixels only.Audio transcription is CPU-only here: about real time on long audio, ~15 s minimum per call.
What works today
Capability | How | Measured on GNOME 50 / i7-1265U |
Screenshot | Mutter ScreenCast → PipeWire → GStreamer, downscaled to 1280 px wide WebP | ~0.3 s first, ~15 KB |
Zoom | full-resolution crop of a region | ~0.06 s |
Click / double / triple / right / middle, hover, drag, scroll | Mutter RemoteDesktop absolute pointer events | verified against a GTK4 test app's own event log |
Key combos | X keysyms ( | verified, incl. |
Typing any text | ASCII as keys; everything else pasted via Mutter's clipboard, old clipboard restored |
|
Typed-text verification | reads the focused field back through AT-SPI |
|
Accessibility tree + refs | AT-SPI; refs clickable via the element's own action | 10–20 ms for a small window |
Screen positions for refs | window-relative extents + window origin (see below) | exact for X11 apps; matched for Wayland apps |
| EditableText / Value / toggle action, mouse fallback | |
OCR | tesseract, 2× upscale, sparse-text mode | ~0.7 s full 1080p screen |
Wait for text / element / idle screen | OCR polling, AT-SPI polling, damage-driven frame timing | |
Open app | Activities search (Super, type, Enter) | |
Clipboard get/set | Mutter selection API | round-trips Unicode |
Listen / transcribe |
| 62 s speech in 62 s, 2.1 GB RAM |
Kill switch | stopping screen sharing from the top bar revokes control until resumed | |
Multi-agent safety | one daemon, a 30 s control lease per client |
Related MCP server: deskwright
What does not exist yet
Multi-monitor support.
Window management beyond "open/switch app via search" (no move/resize/list-geometry API — GNOME blocks
Shell.Introspectfor outside callers).A way to see the mouse cursor in screenshots (it's hidden on purpose).
Non-GNOME compositors.
Install
With ownbox:
ownbox install use-computerSetup (ownbox.sh setup) does the following, and ownbox uninstall undoes the non-package parts:
Installs missing system packages with pacman (asks sudo):
python-gobject,gst-plugin-pipewire,at-spi2-core,wl-clipboard,tesseract,tesseract-data-eng,tesseract-data-osd. More OCR languages:USE_COMPUTER_TESSERACT_LANGS="eng osd srp srp_latn".Creates
.venvwith system site-packages (PyGObject comes from the distro).Sets
org.gnome.desktop.interface toolkit-accessibility true(restored on uninstall if it was false). Apps started after this expose their accessibility tree.Registers the MCP server in
~/.claude.json(user scope, the same entryclaude mcp add --scope userwould write). Start a new Claude session afterwards.Runs
use-computer doctor.
Audio is opt-in because the model is 1.6 GB:
use-computer audio setupManual install:
git clone https://github.com/BogdanStamenovic/use-computer && cd use-computer
./ownbox.sh setupUsage
From Claude: just ask for something that needs the desktop; the skill loads and the
use-computer MCP tools do the work.
From a shell:
use-computer doctor
use-computer screenshot # prints a PNG path
use-computer tree # accessibility tree of the focused window
use-computer find "Save"
use-computer click --ref ref_12
use-computer click --at 640,360 # screenshot pixels; add --screen for real pixels
use-computer type "Zdravo, svete"
use-computer key "ctrl+s"
use-computer ocr --region 0,0,640,360
use-computer audio listen 20 # what the computer plays, for 20 s
use-computer stop # end the session, indicator disappearsCommand | What it does |
| capture to a file |
|
|
| pointer |
| keyboard |
| accessibility |
| reading and waiting |
| desktop |
| JSON list of |
| hearing |
| plumbing |
--json prints raw results. Exit codes: 0 ok, 1 failed, 2 usage error, 3 control revoked by the user.
How it works
Claude ──stdio──► use-computer-mcp ─┐
Bash ─────────► use-computer (CLI)├─ unix socket ─► daemon (one per D-Bus session)
┘ │
┌─────────────────────────────────────────────┼──────────────────────────┐
│ Mutter RemoteDesktop session: pointer, keysyms, clipboard │
│ Mutter ScreenCast stream ─► PipeWire ─► GStreamer appsink (latest frame)│
│ AT-SPI (accessibility bus): tree, refs, actions, text │
│ tesseract (subprocess) │
└────────────────────────────────────────────────────────────────────────┘
listen/transcribe run in the calling process (pw-record + faster-whisper), not the daemon.Why a daemon. A Mutter session dies with the D-Bus connection that created it, CLI calls are separate processes, and several Claude sessions must not fight over one pointer. The daemon is single-threaded (it pumps GLib between requests), holds the session and the newest frame, stops the session after 90 s idle (the sharing indicator disappears) and exits after 30 min.
Why Mutter's API and not the portal. No consent dialog, no restore tokens, absolute coordinates. The cost is GNOME-only.
Why typing pastes non-ASCII. Mutter turns a keysym into a key press only if the active
layout has it, and drops it silently otherwise. Switching layouts from a remote session is
refused unless the session installed the keymap itself. Pasting is the only path that works
for all text; the clipboard is restored afterwards, and handed to wl-copy when the session
ends so it isn't lost.
Why screen positions need a trick. Wayland apps don't know where their window is, so
AT-SPI gives window-relative boxes and bogus screen boxes. GNOME Shell's own accessibility tree
does expose every window actor's screen rectangle, but that includes client-side shadows of
unknown per-side size. locate.py finds the content rectangle inside the actor by scoring every
possible offset on edge strength along its perimeter (shadows are smooth, window borders are
sharp). X11 apps report real screen coordinates and skip this.
Screenshots settle. The screencast is damage-driven, so "no new frame for 120 ms" means the screen stopped changing; screenshots wait for that (max 1 s).
Development
.venv/bin/pytest -q # unit tests, no desktop neededEnd-to-end tests run against a separate headless GNOME Shell so they never touch your desktop:
tests/e2e/headless-shell.sh /tmp/uc.env & # isolated shell, bus, runtime dir
. /tmp/uc.env
python tests/e2e/harness.py /tmp/h.log & # GTK4 app that logs every event it receives
python tests/e2e/mcp_smoke.py /tmp # drives the MCP server over stdio
tests/e2e/headless-stop.shRead the comments in headless-shell.sh before changing it. Running a second shell with the
real XDG_RUNTIME_DIR clobbers the desktop's accessibility bus socket and document portal
mount (this happened; see docs/development-log.md). The headless shell also doesn't register
itself with AT-SPI, so window-position resolution can only be tested on a real session.
License
MIT
Available Tools
13 toolsclipboardBDestructive
Read or replace the clipboard text.
| Name | Required | Description | Default |
|---|---|---|---|
| text | No | set: new clipboard text | |
| action | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare destructiveHint=true, which aligns with the 'replace' operation overwriting clipboard contents. The description adds the read/write distinction that the schema enum implies, but it does not disclose further side effects, such as the clipboard affecting other applications. With destructive hint already present, the added value is modest but acceptable.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One short sentence, front-loaded with the verb and resource, with zero filler. The brevity is well suited to a simple two-action tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description carries the burden of explaining results. 'Read' implies returning clipboard text, but the return format is not explicit, and there is no mention of what 'set' returns. For a simple tool with a destructive annotation, this is a minor but real gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 50%: 'text' is documented as 'set: new clipboard text', while 'action' only has an enum. The description adds semantic meaning by mapping get→read and set→replace, helping the agent understand the action parameter. However, it does not clarify that text is ignored for get or required for set, so compensation is partial.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses specific verbs 'read' and 'replace' tied to the 'clipboard text' resource, and these map directly to the get/set actions in the schema. It does not explicitly name a sibling alternative, but no sibling tool appears to handle clipboard operations, so ambiguity is low.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance about when to use this tool versus alternatives like computer, form_input, or session. No exclusions, prerequisites, or context cues are provided beyond the action enum, leaving the agent to infer use cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
computerADestructive
Mouse, keyboard and screen on the user's GNOME desktop.
Coordinates are pixels in the most recent screenshot (it is downscaled, e.g. 1280x720 for a 1920x1080 screen; the tool maps them back). A screenshot waits briefly for the screen to stop changing. Text outside ASCII is pasted via the clipboard, which is restored afterwards. Actions other than screenshot/zoom return a short confirmation, not an image: take a screenshot when you need to see the outcome.
| Name | Required | Description | Default |
|---|---|---|---|
| ref | No | Element ref from read_screen/find; used instead of coordinate. left_click on a ref uses the element's accessibility action when it has one. | |
| text | No | type: the text (any Unicode). key: space-separated combos like 'ctrl+s' or 'Tab Return' | |
| mouse | No | With ref: click with the real pointer even if an accessibility action exists | |
| action | Yes | What to do | |
| region | No | zoom: [x0, y0, x1, y1] in screenshot pixels | |
| repeat | No | key: repeat count | |
| duration | No | wait: seconds | |
| modifiers | No | Held during a click/scroll/drag, e.g. 'ctrl' or 'ctrl+shift' | |
| coordinate | No | ||
| scroll_amount | No | Wheel steps | |
| scroll_direction | No | ||
| start_coordinate | No | left_click_drag start [x, y] |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses several non-obvious behaviors beyond the annotations: screenshots are downscaled and coordinates are mapped back, screenshots wait for the screen to stabilize, non-ASCII text is pasted via a clipboard that is restored, and non-screenshot/zoom actions return a confirmation rather than an image. These details substantially enrich the openWorldHint and destructiveHint annotations without contradicting them.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four short sentences, each carrying one distinct fact: scope, coordinate system, screenshot/clipboard behavior, and output format. The most important scope statement is front-loaded, and there is no filler or repeated schema content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 12-parameter, 13-action GUI tool with no output schema, the description covers the essential non-schema semantics: coordinate space, output behavior, clipboard side effect, and screenshot timing. Combined with the detailed schema and annotations, an agent has enough information to invoke actions and interpret results correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds important meaning beyond the schema, especially the coordinate mapping from downscaled screenshots and the clipboard-based handling of non-ASCII text. Since schema description coverage is already high at 83%, the description does not need to repeat every parameter, but it adds the key behavioral context that makes coordinate and text parameters safer to use.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the resource scope: 'Mouse, keyboard and screen on the user's GNOME desktop.' This distinguishes it from perception-oriented siblings like read_screen and ocr. However, it lacks an explicit verb such as 'control' or 'operate', so it is clear by implication rather than through a direct action statement.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The scope line implies this is the GUI interaction tool, and the closing advice to 'take a screenshot when you need to see the outcome' gives workflow guidance. However, it never names sibling tools or conditions for choosing computer over read_screen, form_input, or computer_batch, so the usage guidance is only implied, not explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
computer_batchADestructive
Run several steps in one round trip, e.g. click a field, type, press Return, wait, screenshot. Returns every step's text and any images, in order.
| Name | Required | Description | Default |
|---|---|---|---|
| actions | Yes | Steps run in order, stopping at the first error. Each is {name, input} where name is computer|read_screen|find|form_input|ocr|wait_for|clipboard|open_app and input is exactly that tool's arguments. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare openWorldHint=true and destructiveHint=true, covering the safety profile. The description adds a useful behavioral detail: it returns every step's text and images in order, which is not covered by annotations. However, it does not elaborate on side effects beyond the example, and it omits the 'stopping at the first error' behavior that appears in the schema description. The added context is modest but not contradictory.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, focused sentence that immediately states the purpose, provides a clarifying example, and mentions the return format. There is zero waste, and the core concept is front-loaded. It earns its place without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with one parameter and rich schema documentation, the description is nearly complete. It clarifies the batching concept and the output (text and images in order). Given the openWorld and destructive annotations, it lacks an explicit warning about real-world impact, but that is covered by the annotations. The only minor gap is the error-handling behavior, which is already in the schema, so the description is adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides 100% description coverage for the single 'actions' parameter, including the exact format ({name, input}) and allowed names. The tool description adds a concrete example sequence but does not convey any semantic information beyond what the schema already specifies. With full schema coverage, the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Run') and a clear resource ('several steps in one round trip'), with concrete examples (click, type, press Return, wait, screenshot) that immediately convey the tool's function. It clearly distinguishes itself from the individual sibling tools by emphasizing batching, so an agent can tell it apart from computer, read_screen, etc.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context: use this when you want to execute multiple steps in a single round trip, as opposed to calling individual tools sequentially. It provides a concrete example of a multi-step sequence. However, it does not explicitly mention when not to use it (e.g., if you need conditional logic or to inspect intermediate results), leaving the exclusion criteria to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
findARead-only
Find elements by name. Returns refs with boxes (accessibility) or, if nothing accessible matches, OCR text boxes with a click_at point in screenshot pixels.
| Name | Required | Description | Default |
|---|---|---|---|
| ocr | No | Fall back to OCR when accessibility finds nothing | |
| query | Yes | Text to look for: a button label, field name, visible text | |
| scope | No | 'all', 'focused', or app/title substring | all |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With only readOnlyHint=true in annotations, the description adds meaningful behavioral detail: it is non-mutating, tries accessibility first, falls back to OCR, and returns click coordinates in screenshot pixels. This enriches the agent's understanding beyond the single annotation, though it does not cover edge cases like zero matches or result ordering.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, information-dense sentence with no filler. It front-loads the core purpose and return behavior, and every clause contributes useful operational detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description correctly takes responsibility for explaining return values: accessibility boxes or OCR boxes with click_at coordinates. It is missing minor details such as what happens when nothing is found or whether multiple matches are returned, but for a read-only lookup tool with well-documented parameters it is largely complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all three parameters are already self-explanatory. The description adds no parameter-level detail, but none is needed: query, ocr, and scope are adequately documented in the schema itself.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action ('Find elements by name') and immediately describes what is returned: accessibility refs with boxes, falling back to OCR text boxes with click_at points. This clearly distinguishes it from sibling tools like ocr and read_screen by revealing the accessibility-first and OCR-fallback behavior.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The fallback condition 'if nothing accessible matches' implies when OCR behavior kicks in, but the description never explicitly says when to choose this tool over siblings like ocr, read_screen, or form_input. Usage context is implied rather than stated, and no exclusions or alternatives are named.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
form_inputADestructive
Set a form control's value directly through accessibility, without typing.
| Name | Required | Description | Default |
|---|---|---|---|
| ref | Yes | Element ref | |
| value | Yes | Text for fields (replaces the content), true/false for checkboxes and switches, a number for sliders/spin buttons |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds context beyond the destructiveHint annotation by stating that the value is set 'directly through accessibility,' indicating programmatic assignment rather than keystroke simulation. It does not contradict the annotation and offers useful behavioral nuance.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, tightly worded sentence that is front-loaded with the key verb and resource. Every word contributes meaning, with no redundant filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter, no-output-schema mutation tool with a destructiveHint annotation, the description plus schema cover the essential call contract. The overwrite behavior is already stated in the schema's value description, so nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with 'ref' explained as 'Element ref' and 'value' detailing type-specific behavior (text replaces content, booleans for checkboxes/switches, numbers for sliders/spin buttons). The description restates 'value' at a high level but adds no parameter meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('set'), a specific resource ('form control's value'), and a mechanism ('through accessibility, without typing'). This clearly distinguishes it from sibling tools like read_screen or find, which observe rather than modify the UI.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'without typing' implies a contrast with keyboard simulation, but the description never names an alternative tool or states when to use this over it. Usage is implied rather than explicitly guided, and there are no exclusions or conditions provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
listenARead-only
Record audio for a fixed time and transcribe it with Whisper (local, CPU). The recording is deleted afterwards. Start playback before calling. Transcription takes about as long as the audio (minimum ~15 s, Whisper works in 30-second windows).
| Name | Required | Description | Default |
|---|---|---|---|
| source | No | 'output' = what the computer is playing; 'mic' = the microphone. Only use mic when the user asked for it. | output |
| seconds | Yes | How long to record | |
| language | No | ISO code like 'en' or 'sr'; auto-detect if omitted |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds valuable behavioral context beyond the readOnlyHint annotation: the recording is deleted afterwards, Whisper runs locally on CPU, and transcription takes roughly the audio duration in 30-second windows. The ephemeral nature is also consistent with readOnlyHint, so there is no contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four short sentences, each earning its place: core purpose, ephemeral recording, prerequisite, and performance characteristics. There is no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 3-parameter tool with no output schema, the description covers purpose, prerequisites, persistence, and latency well. It could more explicitly state that the result is the transcription text, but 'transcribe it' strongly implies the return value.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents seconds, source, and language well. The description only reinforces the fixed-time aspect and adds no new meaning for the parameters, so a baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action: record audio for a fixed time and transcribe it with Whisper. This is clear and resource-specific, but it does not explicitly distinguish itself from the sibling 'transcribe' tool, so the differentiation is left largely implicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives a clear prerequisite ('Start playback before calling') and implies the tool is for live capture. However, it never says when to prefer this tool over the sibling 'transcribe' tool or when not to use it, so the routing decision is mostly inferred.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ocrARead-only
Read text from the screen with tesseract: lines with confidence and boxes in screenshot pixels. Use when accessibility has nothing (images, canvases, terminals, remote desktops). ~0.7 s for a full 1080p screen.
| Name | Required | Description | Default |
|---|---|---|---|
| lang | No | tesseract languages, e.g. 'eng', 'eng+srp_latn', 'srp' | |
| region | No | [x0, y0, x1, y1] in screenshot pixels; whole screen if omitted |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With readOnlyHint=true already covering safety, the description adds valuable behavioral detail: it returns lines with confidence and boxes in screenshot pixels, and it reports a ~0.7 s latency for a full 1080p screen. This is useful operational context beyond the annotations, and there is no contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences deliver the core action, output format, usage guidance, and performance expectation with no filler. The main purpose is front-loaded, and every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having no output schema, the description adequately covers the return format (lines, confidence, boxes in screenshot pixels), the usage context, and expected latency. The schema handles parameter details, and the annotations handle safety, so nothing critical is missing for an agent to select and invoke this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%: both `lang` and `region` are already documented with meaningful examples and coordinate semantics. The description adds no additional parameter-level meaning beyond reinforcing the screenshot-pixel context, so the high-coverage baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource, 'Read text from the screen with tesseract', and clarifies the output shape ('lines with confidence and boxes in screenshot pixels'). It also scopes the tool to cases where accessibility has nothing, helping distinguish it from sibling screen-reading alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly states when to use the tool: 'Use when accessibility has nothing', with concrete examples like images, canvases, terminals, and remote desktops. It does not explicitly name alternative sibling tools or state exclusion criteria, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
open_appADestructive
Open or switch to an app through the Activities search (Super, type, Enter). Confirm with a screenshot afterwards.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | App name as it appears in GNOME's search, e.g. 'Firefox', 'Files', 'Settings' |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses the interaction method (Super, type, Enter) and instructs the agent to confirm with a screenshot, which adds behavioral context. However, it does not explain the meaning of the destructiveHint annotation – opening an app could potentially change focus or system state, but the description does not clarify the risk or why this is flagged as destructive. It adds some value but leaves a significant aspect unexplained.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise, consisting of two sentences with the purpose front-loaded. Every word contributes to explaining the operation and the confirmation step, with no fluff or redundancy. The structure is efficient and easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple open/switch tool, the description covers the main operation and the confirmation step, but it lacks information about error handling (e.g., if the app is not found) and does not address the destructiveHint annotation. Given the tool's simplicity and complete schema, it is mostly adequate but could benefit from clarifying the destructive aspect to give the agent a fuller picture.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already provides a detailed description of the 'name' parameter (app name as it appears in GNOME's search), so the description adds no additional semantic information. With 100% schema description coverage, the baseline of 3 is appropriate – the description does not need to compensate for missing schema details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (open or switch) on a resource (an app) using a specific method (Activities search). It distinguishes itself from sibling tools like 'windows' or 'find' by focusing on launching an app, though it doesn't explicitly name alternatives. The verb and object are unambiguous, making the tool's purpose immediately understandable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool (when an app needs to be opened or switched to) but provides no explicit guidance on when not to use it or mention alternative tools. There are no exclusion criteria or conditions, so an agent might not know when to prefer this over other tools like 'windows' for listing open apps or 'find' for searching.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
read_screenARead-only
Accessibility tree of windows: one line per element with [ref], role, name, @(x,y,w,h) box in screenshot pixels, value, states and actions.
Prefer refs over pixel guessing. Apps that do not expose accessibility (games, canvases, some Electron apps) show little or nothing; use screenshots and ocr there. 'positions=unresolved' means refs still work for actions but boxes are unknown.
| Name | Required | Description | Default |
|---|---|---|---|
| scope | No | 'focused' (active window), 'all' (every visible window), or an app name / window title substring | focused |
| filter | No | 'interactive' keeps controls; 'all' adds labels and text | interactive |
| ref_id | No | Only the subtree under this ref | |
| max_chars | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses meaningful behavior beyond the readOnlyHint annotation: the output is a single line per element with specific fields, and it explains that 'positions=unresolved' still allows actions via refs. It also warns about the failure mode of non-accessible apps, which is valuable behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded: it immediately defines the output format, then provides actionable usage guidance, then clarifies an edge-case state. Every sentence earns its place with no repetition of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given there is no output schema, the description adequately documents the return format, coordinate system, semantics of unresolved positions, and limitations of the tool. It supplies enough context for an agent to call read_screen correctly and interpret its results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 75%, so the schema already explains scope, filter, ref_id, and max_chars well. The description adds context about refs being preferred over pixel coordinates, but does not add significant meaning to the parameters themselves. This aligns with the baseline for high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb/resource ('Accessibility tree of windows') and precisely describes the output format, including refs, roles, names, boxes, values, states, and actions. It also distinguishes itself from screenshot/OCR-based tools by identifying when the accessibility tree is the right source.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit usage guidance: 'Prefer refs over pixel guessing' and directs the agent to use screenshots/OCR for apps that do not expose accessibility. This clearly helps select between read_screen and its siblings like ocr or computer.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
sessionB
Inspect or end the control session.
| Name | Required | Description | Default |
|---|---|---|---|
| action | Yes | status; stop = end control now (removes the sharing indicator); resume = only after the user explicitly agreed to give control back |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. The description adds one key behavioral detail: 'stop' removes the sharing indicator, and 'resume' requires explicit user consent. However, it doesn't disclose other behaviors like what 'status' returns or the effects of 'resume' on the session. It lacks depth but does add value beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is short and to the point. It front-loads the main purpose and provides a parenthetical clarification. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (single enum parameter, no output schema, no annotations), the description covers the important caveat about 'resume' but misses defining what 'status' returns and what 'resume' actually does beyond the consent requirement. It is adequate but leaves some gaps for an agent to reason about.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the schema already describes each action with a brief note. The description repeats the 'resume' condition and adds the 'removes the sharing indicator' detail for 'stop'. This adds marginal value beyond the schema, so a baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the general purpose ('inspect or end the control session') but is somewhat vague on what 'inspect' means. It doesn't explicitly name the resources or actions beyond the schema's enum. It mentions 'end' but the schema also includes 'resume', so the description partially covers the actions. It is not a tautology, but it is not fully specific.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no explicit when-to-use guidance. It does clarify 'resume' is only after user agreement, which is helpful, but it doesn't contrast with sibling tools or explain when to use 'status' vs 'stop' etc. The absence of alternatives or exclusions leaves the agent to infer usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
transcribeBRead-only
Transcribe an audio/video file with Whisper (local, CPU).
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | Audio or video file path | |
| language | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations provide readOnlyHint=true, and the description aligns with that. The description adds minor context (Whisper model, local CPU execution) but doesn't disclose other behaviors like error handling, output format, or file format restrictions. Given annotations cover the safety profile, a 3 is appropriate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence that is front-loaded with the core action and resource. Every word contributes to the purpose, and there is no redundant or filler content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with two parameters and no output schema, the description is minimally adequate. It covers the core action and model context, but it omits details like output format, supported file types, or prerequisites. The missing language parameter explanation is a notable gap, though the schema's default null offers some hint. Overall, it's sufficient but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 50% (path has a description, language does not). The description does not compensate for the undocumented 'language' parameter, nor does it add any parameter-specific meaning beyond what the schema already provides. The agent is left without guidance on how to use the optional language parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Transcribe' and the resource 'audio/video file', and adds the specific method 'Whisper (local, CPU)'. It distinguishes the tool from siblings like 'listen' by indicating file-based transcription rather than real-time listening, though it doesn't explicitly name alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus alternatives. It only states what the tool does, with no mention of scenarios, exclusions, or why one might choose this over siblings like 'listen'. This leaves the agent to infer usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
wait_forBRead-only
Wait for something instead of sleeping. Returns met=true/false.
| Name | Required | Description | Default |
|---|---|---|---|
| gone | No | Wait for it to disappear instead | |
| name | No | Accessible element name to wait for | |
| role | No | With name: required role, e.g. 'button' | |
| text | No | Visible text to wait for (OCR) | |
| timeout | No | ||
| screen_idle | No | Wait until the screen stops changing |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The readOnlyHint annotation covers the safety profile, and the description adds that the tool returns met=true/false. It does not disclose important behaviors like whether it blocks until timeout, whether met=false means timeout, or whether it returns immediately if the condition is already satisfied.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is short and front-loaded, with no filler words. However, 'something' is an unhelpfully generic placeholder that could be replaced with a more specific phrase, making the first sentence slightly less informative than it could be.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With six parameters and no output schema, the description is minimal but serviceable when combined with the schema descriptions. The main gap is the absence of timeout semantics and the meaning of met=false, which an agent would need to handle failed waits correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is high at 83%, so the description does not need to repeat parameter meanings. The description adds no extra parameter semantics, such as how name, role, and text combine or the effect of the gone flag, leaving the schema to carry that burden.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names an action ('wait') and a result ('met=true/false'), but 'something' is vague and does not specify what can be waited on, such as an element, text, or the screen becoming idle. It also does not clearly differentiate wait_for from sibling tools like find or read_screen beyond the waiting behavior.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Instead of sleeping' gives a clear directive for when to prefer this tool over a raw sleep, which is useful. However, it does not explain when not to use it or how it relates to alternatives such as find or read_screen for checking current state.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
windowsARead-only
List open windows (app, title, active, ref) from accessibility.
| Name | Required | Description | Default |
|---|---|---|---|
| positions | No | Also resolve on-screen boxes (slower) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, so the safety profile is known. The description adds the data source 'from accessibility', implying reliance on the OS accessibility tree rather than screen capture, which is useful context. However, it doesn't disclose potential failure modes, permission requirements, or the semantics of 'active' and 'ref' fields, so transparency is basic.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One short sentence with no filler, front-loaded with the verb 'List' and immediately specifying the resource and fields. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a low-complexity tool with one optional boolean parameter and no output schema, the description covers the essential context: it lists windows, enumerates the returned fields, and notes the accessibility source. It doesn't explain field meanings or failure cases, but is largely adequate for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already fully describes the only parameter (positions) with meaning and a performance note ('slower'), and description coverage is 100%. The tool description adds no additional parameter semantics, so it relies on the schema – baseline 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the specific verb 'List' and resource 'open windows', and specifies the output fields (app, title, active, ref). The qualifier 'from accessibility' clearly separates it from visual screen-reading siblings like read_screen and ocr, so an agent can distinguish it without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives no guidance on when to use this tool versus alternatives such as open_app, read_screen, or find. There are no when-not-to-use conditions or references to sibling tools, leaving the agent to infer applicability from the action alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
13 tool updates
v0.1.0- First observed
clipboard - First observed
computer - First observed
computer_batch - First observed
find - First observed
form_input - First observed
listen - First observed
ocr - First observed
open_app - First observed
read_screen - First observed
session - First observed
transcribe - First observed
wait_for - First observed
windows
TDQS
Scored across 13 tools
Most tools have distinct roles: read_screen, find, and ocr separate accessibility-tree reading from element lookup from text extraction, while listen and transcribe split recording from file transcription. The main ambiguity is between computer and computer_batch, but the batch tool is clearly described as a multi-step version of the same action set.
Names are readable and consistently lowercase snake_case, but conventions are mixed: some are imperative verb phrases (read_screen, open_app, wait_for, form_input) while others are bare nouns (computer, ocr, windows, clipboard, session) or awkward compounds (computer_batch). This is mildly inconsistent but not chaotic.
13 tools is well-scoped for a desktop-control server; input, screen perception, accessibility, OCR, window management, waiting, clipboard, batch execution, audio transcription, and session control each earn a place. There is no obvious bloat or redundancy at the tool-count level.
The surface covers core desktop-automation workflows well: mouse/keyboard input, screenshots, accessibility trees, OCR, app/window management, form filling, waiting, clipboard, and session control. Minor gaps exist, such as no dedicated audio-output tool and relying on computer for primitive actions like screenshot or typing, but these are workable.
Maintenance
Related MCP Connectors
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Eyes and hands on real Windows PCs — observe, click, type via Glasswarp API.
- TabfleetOAuthcom.tabfleet
Launch, inspect, control, and share isolated cloud browsers for your agents.
Drive real devices from your AI Coding tool. Embed a client SDK (Unity, Godot, Flutter, iOS/macOS, Android, React Native, Web) in your app, then capture screenshots, traverse the UI tree, inject taps and key events, and run automated test tasks on the physical device over a secure relay.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceEnables LLMs to capture screenshots, inspect windows, and interact with the Hyprland desktop environment through mouse clicks and keyboard input.1-
- AlicenseAqualityCmaintenanceEnables AI agents to operate a real GNOME Wayland desktop through accessibility-tree widget actions, pointer/keyboard input, OCR, window management, and screen capture, optionally on a private headless session.3326 PyPI8Apache 2.0
- AlicenseNot gradedqualityDmaintenanceEnables an AI agent to see and control a Linux desktop via Wayland/Hyprland, providing screenshots, structured desktop state, pointer/keyboard input, semantic window/workspace tools, and a policy engine for safe execution.Apache 2.0
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to observe and control the user's live desktop by listing apps, reading accessibility trees, and performing clicks, typing, scrolling, dragging, and other input actions with per-action approvals and local audit archives.1MIT