Skip to main content
Glama
BogdanStamenovic

use-computer-mcp

use-computer

Lets an agent (Claude, via MCP) see and operate a GNOME desktop on Wayland the way a person does: screenshots, mouse, keyboard, the accessibility tree with clickable refs, OCR, the clipboard, and listening to whatever the computer is playing. Think Claude in Chrome, but for the whole desktop instead of one browser tab.

It ships as three things that share one daemon: an MCP server (use-computer-mcp), a CLI (use-computer), and a Claude skill (skills/use-computer) that teaches the workflow.

Real limitations, up front:

  • GNOME Shell on Wayland only. It uses Mutter's private RemoteDesktop/ScreenCast D-Bus API. KDE, Sway, Hyprland and X11 sessions are not supported and nothing here pretends to be.

  • Single monitor (the primary one) today.

  • It drives your real session. The pointer moves; typing goes to the focused window. Don't use the machine at the same time.

  • Accessibility coverage depends on the app. GTK/GNOME apps and Firefox are good; Chromium/Electron apps expose little unless started with --force-renderer-accessibility; games and canvases are pixels only.

  • Audio transcription is CPU-only here: about real time on long audio, ~15 s minimum per call.

What works today

Capability

How

Measured on GNOME 50 / i7-1265U

Screenshot

Mutter ScreenCast → PipeWire → GStreamer, downscaled to 1280 px wide WebP

~0.3 s first, ~15 KB

Zoom

full-resolution crop of a region

~0.06 s

Click / double / triple / right / middle, hover, drag, scroll

Mutter RemoteDesktop absolute pointer events

verified against a GTK4 test app's own event log

Key combos

X keysyms (ctrl+shift+t, Tab Return, F5)

verified, incl. shift+TabISO_Left_Tab

Typing any text

ASCII as keys; everything else pasted via Mutter's clipboard, old clipboard restored

Hi šđčćž Ћирилица + done arrives intact

Typed-text verification

reads the focused field back through AT-SPI

verified: true/false/null

Accessibility tree + refs

AT-SPI; refs clickable via the element's own action

10–20 ms for a small window

Screen positions for refs

window-relative extents + window origin (see below)

exact for X11 apps; matched for Wayland apps

form_input

EditableText / Value / toggle action, mouse fallback

OCR

tesseract, 2× upscale, sparse-text mode

~0.7 s full 1080p screen

Wait for text / element / idle screen

OCR polling, AT-SPI polling, damage-driven frame timing

Open app

Activities search (Super, type, Enter)

Clipboard get/set

Mutter selection API

round-trips Unicode

Listen / transcribe

pw-record of the default output's monitor + faster-whisper large-v3-turbo int8

62 s speech in 62 s, 2.1 GB RAM

Kill switch

stopping screen sharing from the top bar revokes control until resumed

Multi-agent safety

one daemon, a 30 s control lease per client

Related MCP server: deskwright

What does not exist yet

  • Multi-monitor support.

  • Window management beyond "open/switch app via search" (no move/resize/list-geometry API — GNOME blocks Shell.Introspect for outside callers).

  • A way to see the mouse cursor in screenshots (it's hidden on purpose).

  • Non-GNOME compositors.

Install

With ownbox:

ownbox install use-computer

Setup (ownbox.sh setup) does the following, and ownbox uninstall undoes the non-package parts:

  1. Installs missing system packages with pacman (asks sudo): python-gobject, gst-plugin-pipewire, at-spi2-core, wl-clipboard, tesseract, tesseract-data-eng, tesseract-data-osd. More OCR languages: USE_COMPUTER_TESSERACT_LANGS="eng osd srp srp_latn".

  2. Creates .venv with system site-packages (PyGObject comes from the distro).

  3. Sets org.gnome.desktop.interface toolkit-accessibility true (restored on uninstall if it was false). Apps started after this expose their accessibility tree.

  4. Registers the MCP server in ~/.claude.json (user scope, the same entry claude mcp add --scope user would write). Start a new Claude session afterwards.

  5. Runs use-computer doctor.

Audio is opt-in because the model is 1.6 GB:

use-computer audio setup

Manual install:

git clone https://github.com/BogdanStamenovic/use-computer && cd use-computer
./ownbox.sh setup

Usage

From Claude: just ask for something that needs the desktop; the skill loads and the use-computer MCP tools do the work.

From a shell:

use-computer doctor
use-computer screenshot                 # prints a PNG path
use-computer tree                       # accessibility tree of the focused window
use-computer find "Save"
use-computer click --ref ref_12
use-computer click --at 640,360         # screenshot pixels; add --screen for real pixels
use-computer type "Zdravo, svete"
use-computer key "ctrl+s"
use-computer ocr --region 0,0,640,360
use-computer audio listen 20            # what the computer plays, for 20 s
use-computer stop                       # end the session, indicator disappears

Command

What it does

screenshot, zoom X0,Y0,X1,Y1

capture to a file

click, double-click, triple-click, right-click, middle-click

--at X,Y or --ref; --modifiers ctrl; --mouse forces a real click on refs

move, drag FROM TO, scroll [--at] DIR [N]

pointer

key KEYS [--repeat N], type TEXT [--ref]

keyboard

tree [--scope all|APP] [--all] [--ref], find QUERY, form-input REF VALUE, action REF NAME

accessibility

ocr [--region] [--lang], wait-for --text/--name/--idle [--gone]

reading and waiting

windows, open-app NAME, clipboard get|set

desktop

batch FILE|-

JSON list of {"op", "args"}

audio setup|status|listen SECONDS|transcribe PATH

hearing

status, stop, resume, shutdown, daemon, doctor, mcp-register, mcp-unregister

plumbing

--json prints raw results. Exit codes: 0 ok, 1 failed, 2 usage error, 3 control revoked by the user.

How it works

Claude ──stdio──► use-computer-mcp ─┐
Bash   ─────────► use-computer (CLI)├─ unix socket ─► daemon (one per D-Bus session)
                                    ┘                   │
          ┌─────────────────────────────────────────────┼──────────────────────────┐
          │ Mutter RemoteDesktop session: pointer, keysyms, clipboard              │
          │ Mutter ScreenCast stream ─► PipeWire ─► GStreamer appsink (latest frame)│
          │ AT-SPI (accessibility bus): tree, refs, actions, text                  │
          │ tesseract (subprocess)                                                 │
          └────────────────────────────────────────────────────────────────────────┘
listen/transcribe run in the calling process (pw-record + faster-whisper), not the daemon.

Why a daemon. A Mutter session dies with the D-Bus connection that created it, CLI calls are separate processes, and several Claude sessions must not fight over one pointer. The daemon is single-threaded (it pumps GLib between requests), holds the session and the newest frame, stops the session after 90 s idle (the sharing indicator disappears) and exits after 30 min.

Why Mutter's API and not the portal. No consent dialog, no restore tokens, absolute coordinates. The cost is GNOME-only.

Why typing pastes non-ASCII. Mutter turns a keysym into a key press only if the active layout has it, and drops it silently otherwise. Switching layouts from a remote session is refused unless the session installed the keymap itself. Pasting is the only path that works for all text; the clipboard is restored afterwards, and handed to wl-copy when the session ends so it isn't lost.

Why screen positions need a trick. Wayland apps don't know where their window is, so AT-SPI gives window-relative boxes and bogus screen boxes. GNOME Shell's own accessibility tree does expose every window actor's screen rectangle, but that includes client-side shadows of unknown per-side size. locate.py finds the content rectangle inside the actor by scoring every possible offset on edge strength along its perimeter (shadows are smooth, window borders are sharp). X11 apps report real screen coordinates and skip this.

Screenshots settle. The screencast is damage-driven, so "no new frame for 120 ms" means the screen stopped changing; screenshots wait for that (max 1 s).

Development

.venv/bin/pytest -q                    # unit tests, no desktop needed

End-to-end tests run against a separate headless GNOME Shell so they never touch your desktop:

tests/e2e/headless-shell.sh /tmp/uc.env &      # isolated shell, bus, runtime dir
. /tmp/uc.env
python tests/e2e/harness.py /tmp/h.log &       # GTK4 app that logs every event it receives
python tests/e2e/mcp_smoke.py /tmp             # drives the MCP server over stdio
tests/e2e/headless-stop.sh

Read the comments in headless-shell.sh before changing it. Running a second shell with the real XDG_RUNTIME_DIR clobbers the desktop's accessibility bus socket and document portal mount (this happened; see docs/development-log.md). The headless shell also doesn't register itself with AT-SPI, so window-position resolution can only be tested on a real session.

License

MIT

Available Tools

13 tools
clipboardB
Destructive

Read or replace the clipboard text.

ParametersJSON Schema
NameRequiredDescriptionDefault
textNoset: new clipboard text
actionYes

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare destructiveHint=true, which aligns with the 'replace' operation overwriting clipboard contents. The description adds the read/write distinction that the schema enum implies, but it does not disclose further side effects, such as the clipboard affecting other applications. With destructive hint already present, the added value is modest but acceptable.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One short sentence, front-loaded with the verb and resource, with zero filler. The brevity is well suited to a simple two-action tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description carries the burden of explaining results. 'Read' implies returning clipboard text, but the return format is not explicit, and there is no mention of what 'set' returns. For a simple tool with a destructive annotation, this is a minor but real gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 50%: 'text' is documented as 'set: new clipboard text', while 'action' only has an enum. The description adds semantic meaning by mapping get→read and set→replace, helping the agent understand the action parameter. However, it does not clarify that text is ignored for get or required for set, so compensation is partial.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses specific verbs 'read' and 'replace' tied to the 'clipboard text' resource, and these map directly to the get/set actions in the schema. It does not explicitly name a sibling alternative, but no sibling tool appears to handle clipboard operations, so ambiguity is low.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance about when to use this tool versus alternatives like computer, form_input, or session. No exclusions, prerequisites, or context cues are provided beyond the action enum, leaving the agent to infer use cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

computerA
Destructive

Mouse, keyboard and screen on the user's GNOME desktop.

Coordinates are pixels in the most recent screenshot (it is downscaled, e.g. 1280x720 for a 1920x1080 screen; the tool maps them back). A screenshot waits briefly for the screen to stop changing. Text outside ASCII is pasted via the clipboard, which is restored afterwards. Actions other than screenshot/zoom return a short confirmation, not an image: take a screenshot when you need to see the outcome.

ParametersJSON Schema
NameRequiredDescriptionDefault
refNoElement ref from read_screen/find; used instead of coordinate. left_click on a ref uses the element's accessibility action when it has one.
textNotype: the text (any Unicode). key: space-separated combos like 'ctrl+s' or 'Tab Return'
mouseNoWith ref: click with the real pointer even if an accessibility action exists
actionYesWhat to do
regionNozoom: [x0, y0, x1, y1] in screenshot pixels
repeatNokey: repeat count
durationNowait: seconds
modifiersNoHeld during a click/scroll/drag, e.g. 'ctrl' or 'ctrl+shift'
coordinateNo
scroll_amountNoWheel steps
scroll_directionNo
start_coordinateNoleft_click_drag start [x, y]

TDQS

A4.2/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses several non-obvious behaviors beyond the annotations: screenshots are downscaled and coordinates are mapped back, screenshots wait for the screen to stabilize, non-ASCII text is pasted via a clipboard that is restored, and non-screenshot/zoom actions return a confirmation rather than an image. These details substantially enrich the openWorldHint and destructiveHint annotations without contradicting them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four short sentences, each carrying one distinct fact: scope, coordinate system, screenshot/clipboard behavior, and output format. The most important scope statement is front-loaded, and there is no filler or repeated schema content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 12-parameter, 13-action GUI tool with no output schema, the description covers the essential non-schema semantics: coordinate space, output behavior, clipboard side effect, and screenshot timing. Combined with the detailed schema and annotations, an agent has enough information to invoke actions and interpret results correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds important meaning beyond the schema, especially the coordinate mapping from downscaled screenshots and the clipboard-based handling of non-ASCII text. Since schema description coverage is already high at 83%, the description does not need to repeat every parameter, but it adds the key behavioral context that makes coordinate and text parameters safer to use.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the resource scope: 'Mouse, keyboard and screen on the user's GNOME desktop.' This distinguishes it from perception-oriented siblings like read_screen and ocr. However, it lacks an explicit verb such as 'control' or 'operate', so it is clear by implication rather than through a direct action statement.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The scope line implies this is the GUI interaction tool, and the closing advice to 'take a screenshot when you need to see the outcome' gives workflow guidance. However, it never names sibling tools or conditions for choosing computer over read_screen, form_input, or computer_batch, so the usage guidance is only implied, not explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

computer_batchA
Destructive

Run several steps in one round trip, e.g. click a field, type, press Return, wait, screenshot. Returns every step's text and any images, in order.

ParametersJSON Schema
NameRequiredDescriptionDefault
actionsYesSteps run in order, stopping at the first error. Each is {name, input} where name is computer|read_screen|find|form_input|ocr|wait_for|clipboard|open_app and input is exactly that tool's arguments.

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare openWorldHint=true and destructiveHint=true, covering the safety profile. The description adds a useful behavioral detail: it returns every step's text and images in order, which is not covered by annotations. However, it does not elaborate on side effects beyond the example, and it omits the 'stopping at the first error' behavior that appears in the schema description. The added context is modest but not contradictory.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, focused sentence that immediately states the purpose, provides a clarifying example, and mentions the return format. There is zero waste, and the core concept is front-loaded. It earns its place without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with one parameter and rich schema documentation, the description is nearly complete. It clarifies the batching concept and the output (text and images in order). Given the openWorld and destructive annotations, it lacks an explicit warning about real-world impact, but that is covered by the annotations. The only minor gap is the error-handling behavior, which is already in the schema, so the description is adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides 100% description coverage for the single 'actions' parameter, including the exact format ({name, input}) and allowed names. The tool description adds a concrete example sequence but does not convey any semantic information beyond what the schema already specifies. With full schema coverage, the baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Run') and a clear resource ('several steps in one round trip'), with concrete examples (click, type, press Return, wait, screenshot) that immediately convey the tool's function. It clearly distinguishes itself from the individual sibling tools by emphasizing batching, so an agent can tell it apart from computer, read_screen, etc.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context: use this when you want to execute multiple steps in a single round trip, as opposed to calling individual tools sequentially. It provides a concrete example of a multi-step sequence. However, it does not explicitly mention when not to use it (e.g., if you need conditional logic or to inspect intermediate results), leaving the exclusion criteria to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

findA
Read-only

Find elements by name. Returns refs with boxes (accessibility) or, if nothing accessible matches, OCR text boxes with a click_at point in screenshot pixels.

ParametersJSON Schema
NameRequiredDescriptionDefault
ocrNoFall back to OCR when accessibility finds nothing
queryYesText to look for: a button label, field name, visible text
scopeNo'all', 'focused', or app/title substringall

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With only readOnlyHint=true in annotations, the description adds meaningful behavioral detail: it is non-mutating, tries accessibility first, falls back to OCR, and returns click coordinates in screenshot pixels. This enriches the agent's understanding beyond the single annotation, though it does not cover edge cases like zero matches or result ordering.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, information-dense sentence with no filler. It front-loads the core purpose and return behavior, and every clause contributes useful operational detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description correctly takes responsibility for explaining return values: accessibility boxes or OCR boxes with click_at coordinates. It is missing minor details such as what happens when nothing is found or whether multiple matches are returned, but for a read-only lookup tool with well-documented parameters it is largely complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all three parameters are already self-explanatory. The description adds no parameter-level detail, but none is needed: query, ocr, and scope are adequately documented in the schema itself.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('Find elements by name') and immediately describes what is returned: accessibility refs with boxes, falling back to OCR text boxes with click_at points. This clearly distinguishes it from sibling tools like ocr and read_screen by revealing the accessibility-first and OCR-fallback behavior.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The fallback condition 'if nothing accessible matches' implies when OCR behavior kicks in, but the description never explicitly says when to choose this tool over siblings like ocr, read_screen, or form_input. Usage context is implied rather than stated, and no exclusions or alternatives are named.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

form_inputA
Destructive

Set a form control's value directly through accessibility, without typing.

ParametersJSON Schema
NameRequiredDescriptionDefault
refYesElement ref
valueYesText for fields (replaces the content), true/false for checkboxes and switches, a number for sliders/spin buttons

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds context beyond the destructiveHint annotation by stating that the value is set 'directly through accessibility,' indicating programmatic assignment rather than keystroke simulation. It does not contradict the annotation and offers useful behavioral nuance.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, tightly worded sentence that is front-loaded with the key verb and resource. Every word contributes meaning, with no redundant filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter, no-output-schema mutation tool with a destructiveHint annotation, the description plus schema cover the essential call contract. The overwrite behavior is already stated in the schema's value description, so nothing critical is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with 'ref' explained as 'Element ref' and 'value' detailing type-specific behavior (text replaces content, booleans for checkboxes/switches, numbers for sliders/spin buttons). The description restates 'value' at a high level but adds no parameter meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('set'), a specific resource ('form control's value'), and a mechanism ('through accessibility, without typing'). This clearly distinguishes it from sibling tools like read_screen or find, which observe rather than modify the UI.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'without typing' implies a contrast with keyboard simulation, but the description never names an alternative tool or states when to use this over it. Usage is implied rather than explicitly guided, and there are no exclusions or conditions provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

listenA
Read-only

Record audio for a fixed time and transcribe it with Whisper (local, CPU). The recording is deleted afterwards. Start playback before calling. Transcription takes about as long as the audio (minimum ~15 s, Whisper works in 30-second windows).

ParametersJSON Schema
NameRequiredDescriptionDefault
sourceNo'output' = what the computer is playing; 'mic' = the microphone. Only use mic when the user asked for it.output
secondsYesHow long to record
languageNoISO code like 'en' or 'sr'; auto-detect if omitted

TDQS

A4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds valuable behavioral context beyond the readOnlyHint annotation: the recording is deleted afterwards, Whisper runs locally on CPU, and transcription takes roughly the audio duration in 30-second windows. The ephemeral nature is also consistent with readOnlyHint, so there is no contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four short sentences, each earning its place: core purpose, ephemeral recording, prerequisite, and performance characteristics. There is no filler or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 3-parameter tool with no output schema, the description covers purpose, prerequisites, persistence, and latency well. It could more explicitly state that the result is the transcription text, but 'transcribe it' strongly implies the return value.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents seconds, source, and language well. The description only reinforces the fixed-time aspect and adds no new meaning for the parameters, so a baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action: record audio for a fixed time and transcribe it with Whisper. This is clear and resource-specific, but it does not explicitly distinguish itself from the sibling 'transcribe' tool, so the differentiation is left largely implicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives a clear prerequisite ('Start playback before calling') and implies the tool is for live capture. However, it never says when to prefer this tool over the sibling 'transcribe' tool or when not to use it, so the routing decision is mostly inferred.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ocrA
Read-only

Read text from the screen with tesseract: lines with confidence and boxes in screenshot pixels. Use when accessibility has nothing (images, canvases, terminals, remote desktops). ~0.7 s for a full 1080p screen.

ParametersJSON Schema
NameRequiredDescriptionDefault
langNotesseract languages, e.g. 'eng', 'eng+srp_latn', 'srp'
regionNo[x0, y0, x1, y1] in screenshot pixels; whole screen if omitted

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With readOnlyHint=true already covering safety, the description adds valuable behavioral detail: it returns lines with confidence and boxes in screenshot pixels, and it reports a ~0.7 s latency for a full 1080p screen. This is useful operational context beyond the annotations, and there is no contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences deliver the core action, output format, usage guidance, and performance expectation with no filler. The main purpose is front-loaded, and every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having no output schema, the description adequately covers the return format (lines, confidence, boxes in screenshot pixels), the usage context, and expected latency. The schema handles parameter details, and the annotations handle safety, so nothing critical is missing for an agent to select and invoke this tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%: both `lang` and `region` are already documented with meaningful examples and coordinate semantics. The description adds no additional parameter-level meaning beyond reinforcing the screenshot-pixel context, so the high-coverage baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource, 'Read text from the screen with tesseract', and clarifies the output shape ('lines with confidence and boxes in screenshot pixels'). It also scopes the tool to cases where accessibility has nothing, helping distinguish it from sibling screen-reading alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly states when to use the tool: 'Use when accessibility has nothing', with concrete examples like images, canvases, terminals, and remote desktops. It does not explicitly name alternative sibling tools or state exclusion criteria, so it stops short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

open_appA
Destructive

Open or switch to an app through the Activities search (Super, type, Enter). Confirm with a screenshot afterwards.

ParametersJSON Schema
NameRequiredDescriptionDefault
nameYesApp name as it appears in GNOME's search, e.g. 'Firefox', 'Files', 'Settings'

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses the interaction method (Super, type, Enter) and instructs the agent to confirm with a screenshot, which adds behavioral context. However, it does not explain the meaning of the destructiveHint annotation – opening an app could potentially change focus or system state, but the description does not clarify the risk or why this is flagged as destructive. It adds some value but leaves a significant aspect unexplained.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise, consisting of two sentences with the purpose front-loaded. Every word contributes to explaining the operation and the confirmation step, with no fluff or redundancy. The structure is efficient and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple open/switch tool, the description covers the main operation and the confirmation step, but it lacks information about error handling (e.g., if the app is not found) and does not address the destructiveHint annotation. Given the tool's simplicity and complete schema, it is mostly adequate but could benefit from clarifying the destructive aspect to give the agent a fuller picture.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already provides a detailed description of the 'name' parameter (app name as it appears in GNOME's search), so the description adds no additional semantic information. With 100% schema description coverage, the baseline of 3 is appropriate – the description does not need to compensate for missing schema details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (open or switch) on a resource (an app) using a specific method (Activities search). It distinguishes itself from sibling tools like 'windows' or 'find' by focusing on launching an app, though it doesn't explicitly name alternatives. The verb and object are unambiguous, making the tool's purpose immediately understandable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use the tool (when an app needs to be opened or switched to) but provides no explicit guidance on when not to use it or mention alternative tools. There are no exclusion criteria or conditions, so an agent might not know when to prefer this over other tools like 'windows' for listing open apps or 'find' for searching.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

read_screenA
Read-only

Accessibility tree of windows: one line per element with [ref], role, name, @(x,y,w,h) box in screenshot pixels, value, states and actions.

Prefer refs over pixel guessing. Apps that do not expose accessibility (games, canvases, some Electron apps) show little or nothing; use screenshots and ocr there. 'positions=unresolved' means refs still work for actions but boxes are unknown.

ParametersJSON Schema
NameRequiredDescriptionDefault
scopeNo'focused' (active window), 'all' (every visible window), or an app name / window title substringfocused
filterNo'interactive' keeps controls; 'all' adds labels and textinteractive
ref_idNoOnly the subtree under this ref
max_charsNo

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses meaningful behavior beyond the readOnlyHint annotation: the output is a single line per element with specific fields, and it explains that 'positions=unresolved' still allows actions via refs. It also warns about the failure mode of non-accessible apps, which is valuable behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded: it immediately defines the output format, then provides actionable usage guidance, then clarifies an edge-case state. Every sentence earns its place with no repetition of schema details.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given there is no output schema, the description adequately documents the return format, coordinate system, semantics of unresolved positions, and limitations of the tool. It supplies enough context for an agent to call read_screen correctly and interpret its results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 75%, so the schema already explains scope, filter, ref_id, and max_chars well. The description adds context about refs being preferred over pixel coordinates, but does not add significant meaning to the parameters themselves. This aligns with the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb/resource ('Accessibility tree of windows') and precisely describes the output format, including refs, roles, names, boxes, values, states, and actions. It also distinguishes itself from screenshot/OCR-based tools by identifying when the accessibility tree is the right source.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit usage guidance: 'Prefer refs over pixel guessing' and directs the agent to use screenshots/OCR for apps that do not expose accessibility. This clearly helps select between read_screen and its siblings like ocr or computer.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

sessionB

Inspect or end the control session.

ParametersJSON Schema
NameRequiredDescriptionDefault
actionYesstatus; stop = end control now (removes the sharing indicator); resume = only after the user explicitly agreed to give control back

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. The description adds one key behavioral detail: 'stop' removes the sharing indicator, and 'resume' requires explicit user consent. However, it doesn't disclose other behaviors like what 'status' returns or the effects of 'resume' on the session. It lacks depth but does add value beyond the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short and to the point. It front-loads the main purpose and provides a parenthetical clarification. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (single enum parameter, no output schema, no annotations), the description covers the important caveat about 'resume' but misses defining what 'status' returns and what 'resume' actually does beyond the consent requirement. It is adequate but leaves some gaps for an agent to reason about.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the schema already describes each action with a brief note. The description repeats the 'resume' condition and adds the 'removes the sharing indicator' detail for 'stop'. This adds marginal value beyond the schema, so a baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states the general purpose ('inspect or end the control session') but is somewhat vague on what 'inspect' means. It doesn't explicitly name the resources or actions beyond the schema's enum. It mentions 'end' but the schema also includes 'resume', so the description partially covers the actions. It is not a tautology, but it is not fully specific.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no explicit when-to-use guidance. It does clarify 'resume' is only after user agreement, which is helpful, but it doesn't contrast with sibling tools or explain when to use 'status' vs 'stop' etc. The absence of alternatives or exclusions leaves the agent to infer usage.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

transcribeB
Read-only

Transcribe an audio/video file with Whisper (local, CPU).

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYesAudio or video file path
languageNo

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations provide readOnlyHint=true, and the description aligns with that. The description adds minor context (Whisper model, local CPU execution) but doesn't disclose other behaviors like error handling, output format, or file format restrictions. Given annotations cover the safety profile, a 3 is appropriate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, concise sentence that is front-loaded with the core action and resource. Every word contributes to the purpose, and there is no redundant or filler content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with two parameters and no output schema, the description is minimally adequate. It covers the core action and model context, but it omits details like output format, supported file types, or prerequisites. The missing language parameter explanation is a notable gap, though the schema's default null offers some hint. Overall, it's sufficient but not thorough.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 50% (path has a description, language does not). The description does not compensate for the undocumented 'language' parameter, nor does it add any parameter-specific meaning beyond what the schema already provides. The agent is left without guidance on how to use the optional language parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Transcribe' and the resource 'audio/video file', and adds the specific method 'Whisper (local, CPU)'. It distinguishes the tool from siblings like 'listen' by indicating file-based transcription rather than real-time listening, though it doesn't explicitly name alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this tool versus alternatives. It only states what the tool does, with no mention of scenarios, exclusions, or why one might choose this over siblings like 'listen'. This leaves the agent to infer usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

wait_forB
Read-only

Wait for something instead of sleeping. Returns met=true/false.

ParametersJSON Schema
NameRequiredDescriptionDefault
goneNoWait for it to disappear instead
nameNoAccessible element name to wait for
roleNoWith name: required role, e.g. 'button'
textNoVisible text to wait for (OCR)
timeoutNo
screen_idleNoWait until the screen stops changing

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The readOnlyHint annotation covers the safety profile, and the description adds that the tool returns met=true/false. It does not disclose important behaviors like whether it blocks until timeout, whether met=false means timeout, or whether it returns immediately if the condition is already satisfied.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short and front-loaded, with no filler words. However, 'something' is an unhelpfully generic placeholder that could be replaced with a more specific phrase, making the first sentence slightly less informative than it could be.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With six parameters and no output schema, the description is minimal but serviceable when combined with the schema descriptions. The main gap is the absence of timeout semantics and the meaning of met=false, which an agent would need to handle failed waits correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is high at 83%, so the description does not need to repeat parameter meanings. The description adds no extra parameter semantics, such as how name, role, and text combine or the effect of the gone flag, leaving the schema to carry that burden.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names an action ('wait') and a result ('met=true/false'), but 'something' is vague and does not specify what can be waited on, such as an element, text, or the screen becoming idle. It also does not clearly differentiate wait_for from sibling tools like find or read_screen beyond the waiting behavior.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Instead of sleeping' gives a clear directive for when to prefer this tool over a raw sleep, which is useful. However, it does not explain when not to use it or how it relates to alternatives such as find or read_screen for checking current state.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

windowsA
Read-only

List open windows (app, title, active, ref) from accessibility.

ParametersJSON Schema
NameRequiredDescriptionDefault
positionsNoAlso resolve on-screen boxes (slower)

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, so the safety profile is known. The description adds the data source 'from accessibility', implying reliance on the OS accessibility tree rather than screen capture, which is useful context. However, it doesn't disclose potential failure modes, permission requirements, or the semantics of 'active' and 'ref' fields, so transparency is basic.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One short sentence with no filler, front-loaded with the verb 'List' and immediately specifying the resource and fields. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a low-complexity tool with one optional boolean parameter and no output schema, the description covers the essential context: it lists windows, enumerates the returned fields, and notes the accessibility source. It doesn't explain field meanings or failure cases, but is largely adequate for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already fully describes the only parameter (positions) with meaning and a performance note ('slower'), and description coverage is 100%. The tool description adds no additional parameter semantics, so it relies on the schema – baseline 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses the specific verb 'List' and resource 'open windows', and specifies the output fields (app, title, active, ref). The qualifier 'from accessibility' clearly separates it from visual screen-reading siblings like read_screen and ocr, so an agent can distinguish it without opening schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no guidance on when to use this tool versus alternatives such as open_app, read_screen, or find. There are no when-not-to-use conditions or references to sibling tools, leaving the agent to infer applicability from the action alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 13 tool updatesv0.1.0
    • First observedclipboard
    • First observedcomputer
    • First observedcomputer_batch
    • First observedfind
    • First observedform_input
    • First observedlisten
    • First observedocr
    • First observedopen_app
    • First observedread_screen
    • First observedsession
    • First observedtranscribe
    • First observedwait_for
    • First observedwindows

TDQS

A3.6/5.0

Scored across 13 tools

Disambiguation4/5

Most tools have distinct roles: read_screen, find, and ocr separate accessibility-tree reading from element lookup from text extraction, while listen and transcribe split recording from file transcription. The main ambiguity is between computer and computer_batch, but the batch tool is clearly described as a multi-step version of the same action set.

Naming Consistency3/5

Names are readable and consistently lowercase snake_case, but conventions are mixed: some are imperative verb phrases (read_screen, open_app, wait_for, form_input) while others are bare nouns (computer, ocr, windows, clipboard, session) or awkward compounds (computer_batch). This is mildly inconsistent but not chaotic.

Tool Count5/5

13 tools is well-scoped for a desktop-control server; input, screen perception, accessibility, OCR, window management, waiting, clipboard, batch execution, audio transcription, and session control each earn a place. There is no obvious bloat or redundancy at the tool-count level.

Completeness4/5

The surface covers core desktop-automation workflows well: mouse/keyboard input, screenshots, accessibility trees, OCR, app/window management, form filling, waiting, clipboard, and session control. Minor gaps exist, such as no dedicated audio-output tool and relying on computer for primitive actions like screenshot or typing, but these are workable.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to operate a real GNOME Wayland desktop through accessibility-tree widget actions, pointer/keyboard input, OCR, window management, and screen capture, optionally on a private headless session.
    33
    26 PyPI
    8
    Apache 2.0
  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables an AI agent to see and control a Linux desktop via Wayland/Hyprland, providing screenshots, structured desktop state, pointer/keyboard input, semantic window/workspace tools, and a policy engine for safe execution.
    Apache 2.0
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI agents to observe and control the user's live desktop by listing apps, reading accessibility trees, and performing clicks, typing, scrolling, dragging, and other input actions with per-action approvals and local audit archives.
    1
    MIT