Skip to main content
Glama
liufeicc

cc-computer-use

by liufeicc

cc-computer-use — Computer-Use MCP Server

中文文档 | English

An MCP server that lets an AGENT (Claude Code, etc.) drive a Linux desktop directly.

Core idea: perceive the UI through the accessibility tree (AT-SPI) and act through element-level actions — instead of the conventional "screenshot + coordinate click". This removes two pain points at the root:

Pain point

Conventional approach

This project

① Screenshots burn tokens

1000+ tokens per screenshot, every round

Read the a11y tree / OCR text — both compact text

② Clicks miss the target

LLM estimates coordinates from pixels; DPI and multi-monitor stack up the error

Element-level do_action: zero coordinates, immune to focus / resolution / DPI

Feasibility was established by measurement in demo/ — the conclusion there was that element-level actions beat coordinate clicking outright, which is why the execution layer here is "element-level first, coordinate clicking only as fallback".

For a full deployment walkthrough see docs/安装说明.md (Chinese).


1. Capabilities (14 MCP tools)

Tool

Purpose

get_ui_tree

Read the accessibility tree as compact text, each actionable element tagged with a [ref]the primary sense for apps that expose a tree

get_screen_text

OCR the screen into text plus coordinates ([ref] text @ (x,y)) — the primary sense for grey-area apps, far cheaper than screenshots. Defaults to scope="window" (active window only): a dialog takes 0.3–3 s while a dense full screen takes 8–10 s

find_element

Search elements by text / role / app; returns candidates carrying refs

element_info

Inspect one element: value, states, rect, available actions

click

Three-tier fallback: element-level do_action → calibrated coordinate click → screenshot fallback. In grey areas you can pass raw x+y. Coordinate clicks return landing evidence: which window received it, what text sits under the point, the active window afterwards, plus a crosshair image (red cross = the exact pixel clicked), a computed aim offset (expect="Save" → "missed, off by (+17,-42), 45px total — try its center (517,258)"), and a post-click change percentage. preview=false disables the visual feedback

get_last_click_image

Look back at the most recent coordinate click (index=-1 latest, -2 before that). When a click produces no visible reaction, inspect where it actually landed before blindly clicking again

type_text

Input: element-level set_value first, keyboard injection as fallback. Reports the active window after typing, so a stolen focus is immediately visible

press_key

Key combos (ctrl+s, alt+F4, …); reports the active window afterwards

act_sequence

Batch a run of actions in one call (click / type / key / wait / sleep / list_windows / screenshot), saving round trips. Putting screenshot last returns "what it looks like now" inside the same call

launch_app

Launch an app on the target display (the proper way to get an app into the sandbox)

list_windows

Visible windows (id / title / PID / geometry, largest first)

wait_window

Wait until a window title matches (server-side polling, single call)

get_screen_layout

Monitor layout and virtual desktop size

screenshot

Screenshot — last resort, JPEG by default; use only when pixel-level judgement is genuinely needed

  • Apps with a tree: launch_appget_ui_tree to see the structure → locate the target [ref] (or find_element) → click(ref) / type_text(ref).

  • Grey-area apps (SWT/Java, custom-drawn widgets, games, remote desktops — no a11y tree): get_screen_text to obtain text and coordinates → click(ref) or click(x=.., y=..). Do not default to screenshots and let the model guess coordinates: in a measured DBeaver session of 8.3 minutes, tool calls accounted for 21 seconds while the remaining 478 seconds went entirely into "look at image → estimate pixel → convert back to screen coordinates".

  • When unsure about a coordinate, pass expect: click(x=512, y=384, expect="Save") makes the program compute how far off you were and which coordinate to use instead. If a click does nothing, call get_last_click_image before retrying — never repeat clicks blindly.

  • Batch any predictable sequence into a single act_sequence. Every extra call costs a model-think + read-result round trip (30–70 s measured). Split calls only when you need to branch on the result.

Two mechanisms that save round trips

  1. Landing evidence — the return value of a coordinate click or key injection only tells you the event was sent (xdotool has no idea what it hit). So clicks also report:

    coordinate click executed @(874,578) | landed on: window "Landing Story" 310x212
    | text under point: "OK" | active window after: none (previous "Landing Story" is gone)

    The model immediately knows it hit "OK" and the dialog really closed — no confirming screenshot needed. (Element-level do_action doesn't need this; there is no "did it hit" question.)

  2. The screenshot step of act_sequence — two thirds of screenshots in the measured session existed purely to confirm a preceding sequence. Moving them into the last step of the same call eliminates that round trip entirely. The image is returned as an extra image content block alongside the step log.

Click feedback: the ring, the crosshair, the computed offset

Coordinate clicks exist only in tier ② (fallback) and raw-coordinate grey-area use, and all of the feedback below is attached automatically by backend.click_at.

  • Ring (backend/linux/ring.py) — for the human watching the sandbox: a solid red ring lights up at the landing point and self-destructs after 1.5 s. Without it, a click in the virtual screen is invisible to the user. Disable with CC_CU_CLICK_RING=0.

  • Crosshair preview image — for the model: a 480x300 crop grabbed before the click with a red cross on the landing point, unscaled (1 px on the image = 1 px on screen, so no arithmetic is needed). ~190 tokens per call. preview=false skips the capture entirely (zero cost); disable globally with CC_CU_CLICK_PREVIEW=0.

  • Computed aim offset — the landing coordinate is one the program itself sent (exact), and the text block positions come from OCR (which reports left/top/width/height). The offset is therefore pure arithmetic, not a guess. Passing expect turns a heuristic into an exact number.

  • Post-click change — a before/after pixel comparison of the same region (<0.5 % none, <5 % slight, ≥5 % obvious). This is an independent second signal that needs no knowledge of intent. Read the two signals together: off by 45 px but the UI changed means the button's hit area is larger than its text; off by 45 px with no change means the coordinate needs fixing.


Related MCP server: linux-computer-use

2. Requirements

Item

Requirement

OS

Linux (Phase 1 is Linux only; Windows is Phase 3)

Session type

X11 (under Wayland, global input injection is restricted)

Python

3.12+

System components

xdotool (injection), xserver-xephyr (sandbox virtual screen), xclip (clipboard path for non-ASCII input), wmctrl (app enumeration), tesseract-ocr + chi_sim (OCR grey-area sensing), zenity (tests), AT-SPI (gir1.2-atspi-2.0 / libatspi)

Accessibility

Must be enabled (see below)

Key constraint: AT-SPI depends on the system's libatspi and the Atspi typelib — OS-level components that cannot be bundled into a single binary. _bootstrap.py points GI_TYPELIB_PATH at the system directory at runtime, so even as a frozen executable these components must still be provided by the host system.


3. Installation

# 1. One-time: enable the accessibility switch (otherwise GTK app trees are unreadable)
gsettings set org.gnome.desktop.interface toolkit-accessibility true

# 2. System dependencies (Debian/Ubuntu) — OS-level, not bundled into the build artifact
sudo apt install -y xdotool xserver-xephyr xclip wmctrl zenity \
                    gir1.2-atspi-2.0 \
                    tesseract-ocr tesseract-ocr-chi-sim

# 3. conda environment (Python >= 3.12); pygobject must come from conda-forge — pip cannot install gi
conda create -n cc-computer-use python=3.12 -y
conda install -n cc-computer-use -c conda-forge pygobject -y
conda activate cc-computer-use

# 4. Python dependencies + this project (editable)
PYTHONNOUSERSITE=1 pip install -U mcp cryptography pyinstaller pytest
PYTHONNOUSERSITE=1 pip install -e .

Why PYTHONNOUSERSITE=1: if another mcp lives in ~/.local, it shadows the conda environment's package and causes version confusion (affecting both development and packaging). All run/build commands should carry this prefix.

Note: chi_sim is the Simplified Chinese OCR language pack; without it Chinese text comes out as garbage. If your UI is English-only, set CC_CU_OCR_LANG=eng — noticeably faster.

Moving to a new machine / deploying from scratch / troubleshooting: see docs/安装说明.md (Chinese, step-by-step).


4. Running the MCP server (development mode)

# Self-check: prints backend status, tool list, screen layout (does not enter the stdio loop)
PYTHONNOUSERSITE=1 python -m computer_use_mcp.server --selftest

# Normal run (stdio transport, for MCP clients)
PYTHONNOUSERSITE=1 python -m computer_use_mcp.server

Isolation sandbox (on by default)

X11 has a single physical pointer and focus: while the agent injects input, the user cannot use the machine. So this server defaults to a visible Xephyr virtual screen (the display number is allocated automatically, one per Claude session). All injection, screenshots and geometry queries target that virtual screen — the host desktop is untouched and the user can keep working. You can also reach into the Xephyr window with your own mouse and keyboard at any time; the agent's injection politely yields until you leave.

The virtual screen starts only when this MCP is first actually used, not when Claude Code launches — otherwise every session would spawn a stray Xephyr window.

Each Claude session gets its own private screen (allocated by the X server, race-free), so concurrent sessions never fight over focus or the clipboard. Within one session, the main agent and its sub-agents share that screen, and injection is therefore mutually exclusive: whoever fails to acquire the screen lock receives an immediate "screen is busy, retry later" error rather than being silently queued (queueing would execute decisions made against a stale view of the UI).

If the sandbox is unavailable (Xephyr missing, or startup failed), injection is refused with an explicit error rather than silently falling back to your real desktop. To operate the real desktop on purpose, set CC_CU_DISPLAY_MODE=real.

Environment variable

Default

Effect

CC_CU_DISPLAY_MODE

isolated

Set to real to disable the sandbox and operate the real desktop

CC_CU_SANDBOX_DISPLAY

unset

Unset = allocate a free display per session; set explicitly (e.g. :99) = fixed display, and attach to it rather than restarting if it already exists

CC_CU_SANDBOX_SCREEN

1600x1000

Virtual screen resolution

CC_CU_SANDBOX_WAIT_USER

30

Max seconds injection yields while the user is inside the sandbox

CC_CU_SANDBOX_WM

auto

Set to none to skip launching the i3 window manager inside the sandbox

CC_CU_SANDBOX_AT_SPI_BUS

unset

Override the AT-SPI bus address for sandboxed apps (debugging)

CC_CU_OCR_LANG

chi_sim+eng

OCR languages; eng for English-only UIs is noticeably faster

CC_CU_CLICK_RING

1

Set to 0 to disable the visual click ring

CC_CU_CLICK_PREVIEW

1

Set to 0 to disable the crosshair preview image globally

PYTHONNOUSERSITE

Set to 1 for every command so ~/.local packages cannot shadow the conda env

a11y isolation: the sandbox has its own AT-SPI bus

Xephyr isolates the X11 channel (injection, screenshots) but not AT-SPI. Accessibility runs over the session D-Bus, and the sandbox shares one at-spi2-registryd with the host. Measured on 2026-09-15: traversing the a11y tree of an app inside the sandbox crashed the host GNOME Shell. Offline core analysis confirmed the root cause — gnome-shell's own atk-bridge calling g_object_ref on an already-freed GObject (a use-after-free), crashing on its main loop thread.

Solution: when the sandbox starts it brings up a private AT-SPI bus (its own dbus-daemon + at-spi2-registryd in a private directory). Sandboxed apps and the MCP's a11y reader both connect to it — the host bus cannot see any sandboxed app, while a11y capability is fully preserved.

host GNOME Shell ── host AT-SPI bus ── host apps (gnome-shell / chrome / …)
                                    ✗ mutually invisible
sandboxed apps ──── private sandbox AT-SPI bus ── the MCP's a11y reader
  • If the private bus fails to start, apps receive a dead address (no a11y connection): the host stays safe but no tree is readable inside the sandbox. This is deliberate — better to have no a11y than to let traffic reach the host bus.

  • Why a bus address rather than NO_AT_BRIDGE=1: the latter is a GTK3-only switch (GTK4 uses GTK_A11Y), and SWT (Java apps such as DBeaver) honours neither. AT_SPI_BUS_ADDRESS is toolchain-agnostic — GTK3/GTK4/SWT/Qt/Electron all route a11y through libatspi, all read it, and none of them fall back to the session bus (measured).

  • Verification script: PYTHONNOUSERSITE=1 python tests/manual_a11y_isolation.py

Operational consequence: if the sandbox crashes and is rebuilt, the bus address changes, and libatspi's atspi_init() can only ever bind once per process — so a11y stays permanently unavailable for the rest of that session, leaving only OCR + coordinate clicking. Restarting Claude Code is the only recovery.


5. Building a standalone executable

bash build.sh
# Artifacts (onedir, not a single file):
#   dist/computer-use-mcp-bin/     directory holding the real executable, ~455 MB
#   dist/computer-use-mcp          thin shell wrapper forwarding to it

# Verify the frozen artifact
PYTHONNOUSERSITE=1 ./dist/computer-use-mcp --selftest

Why onedir: onefile re-extracts itself to a temp directory on every launch, making cold start an order of magnitude slower. The wrapper exists to preserve the registered path (~/.claude.json points at dist/computer-use-mcp) — do not delete it.

Key points baked into build.sh:

  • --collect-all gi + --hidden-import gi.repository.Atspi/GLib/GObject to package PyGObject.

  • --collect-submodules mcp.server (not all of mcp, which would pull in mcp.cli and its typer dependency).

  • --copy-metadata: mcp/pydantic and friends read versions via importlib.metadata.

  • LD_LIBRARY_PATH must prepend conda's lib: otherwise PyInstaller pairs the system's old libcrypto with conda's new libssl and fails at runtime with OPENSSL_3.3.0 not found.

  • The Atspi typelib / libatspi are not bundled; _bootstrap points at the system copies at runtime.

  • --collect-submodules Xlib plus explicit --hidden-import Xlib.ext.shape: the click ring's X extension modules are imported dynamically by name, so they appear in no static import graph. Omit them and the ring silently works in development but vanishes from the frozen build.

Distributing to another machine requires the whole dist/ directory; the target machine still needs the system dependencies installed (xdotool / xserver-xephyr / tesseract / AT-SPI, etc.).


6. Wiring it into Claude Code

Add one of the following to your project or ~/.claude MCP configuration:

A. Using the packaged executable (recommended, no conda needed)

{
  "mcpServers": {
    "cc-computer-use": {
      "type": "stdio",
      "command": "/absolute/path/cc-computer-use/dist/computer-use-mcp",
      "args": [],
      "env": { "PYTHONNOUSERSITE": "1" }
    }
  }
}

B. Running from the conda environment (development; source edits take effect immediately)

{
  "mcpServers": {
    "cc-computer-use": {
      "type": "stdio",
      "command": "/path/to/conda/envs/cc-computer-use/bin/python",
      "args": ["-m", "computer_use_mcp.server"],
      "env": {
        "PYTHONNOUSERSITE": "1",
        "PYTHONPATH": "/absolute/path/cc-computer-use/src"
      }
    }
  }
}

⚠️ The server name must be cc-computer-use, not computer-use — the latter fails to register in practice; renaming fixes it.

Claude Code must be running inside an X11 graphical session (the server subprocess inherits DISPLAY so it can read the screen and inject input).

Once connected you can give natural-language tasks, for example:

  • "What buttons are clickable in the current window?" → the agent calls get_ui_tree

  • "Click 'Yes' in this dialog" → find_element + click(ref), element-level, zero coordinates

  • "Type hello in the editor and save" → type_text + press_key('ctrl+s')


7. Testing

# Unit tests (no desktop needed): geometry calibration / ref mapping / serializer denoising /
# injection and sandbox logic. 293 tests collected; without the e2e flag: 287 passed, 6 skipped
PYTHONNOUSERSITE=1 python -m pytest -q

# End-to-end (needs X11 + zenity + Xephyr): runs INSIDE the sandbox by default, never touching the
# real desktop; requires explicit opt-in. 293 passed
CC_CU_E2E=1 PYTHONNOUSERSITE=1 python -m pytest -q

# Manual stories (6 of them; full list in docs/安装说明.md §5.4)

# Sandbox story (drives a real server): launches zenity in the sandbox → element-level click →
# asserts the host was left completely undisturbed
PYTHONNOUSERSITE=1 python tests/manual_sandbox_story.py

# Grey-area + landing-evidence story (drives the frozen artifact; run `bash build.sh` first):
# OCR yields coordinates → click → asserts the dialog actually closed
PYTHONNOUSERSITE=1 python tests/manual_ocr_story.py

⚠️ While manual_sandbox_story.py runs, do not touch your mouse or keyboard — it asserts that the host's active window and pointer are unchanged.

The suite is split deliberately: unit tests never need a desktop, end-to-end tests require X11 plus zenity and Xephyr (and skip automatically when unavailable), and manual_*.py files are manual stories that drive a real server or the frozen artifact through complete task flows.


8. Architecture

Claude Code / MCP client
        │ MCP protocol (stdio, JSON-RPC)
┌───────▼──────────────────────────────────────────┐
│  server.py   (FastMCP/MCPServer entry)            │
│  tools/          14 tool definitions (schema +    │
│                  guiding descriptions)            │
│  core/coordinator   semantic orchestration: the   │
│                     three-tier fallback           │
│  core/serializer    tree → compact text (token    │
│                     saving) + ref allocation      │
│  core/geometry      coordinate calibration        │
│  core/display       isolation sandbox: the single │
│                     source of DISPLAY             │
│  utils/refs         ref ↔ live element mapping    │
│                     (in-process cache)            │
└───────┬──────────────────────────────────────────┘
        │ backend/base.py  abstract contract
┌───────▼──────────┐
│ backend/linux/   │  AT-SPI reads (atspi.py) + xdotool injection (inject/)
└──────────────────┘  (Windows backend = Phase 3)

The three-tier fallback (coordinator.click):

  1. element: backend.invoke() → AT-SPI do_action, zero coordinates (preferred).

  2. coord: geometry calibrates an absolute screen coordinate → xdotool focuses the window and clicks (fallback).

  3. screenshot: when not even an element can be found (grey area) → fall back to pixels, with the LLM making the visual decision.

The returned ActionResult states which tier actually took effect.


9. Project layout

cc-computer-use/
├── src/computer_use_mcp/
│   ├── _bootstrap.py        # runs first: sets GI_TYPELIB_PATH before importing gi
│   ├── server.py            # MCP entry + tool registration + --selftest
│   ├── backend/
│   │   ├── base.py          # Backend ABC + Rect/UINode/Element/TextBlock dataclasses
│   │   └── linux/
│   │       ├── atspi.py     # AT-SPI reads + do_action/set_value
│   │       ├── inject/      # xdotool injection + window geometry
│   │       │                #   (apps/base/keyboard/pointer/screens/windows)
│   │       ├── grab.py      # screen capture (shared by screenshot and OCR)
│   │       ├── ocr.py       # grey-area sensing: tesseract → text blocks with coordinates
│   │       ├── ring.py      # visual click ring (feedback for the human)
│   │       └── backend.py   # LinuxBackend composite implementation
│   ├── core/
│   │   ├── serializer.py    # tree serialization (denoise / save tokens / allocate refs)
│   │   ├── geometry.py      # coordinate calibration
│   │   ├── screen_lock.py   # screen-exclusive lock (mutual exclusion for injection)
│   │   ├── display/         # isolation sandbox (Xephyr + private AT-SPI bus +
│   │   │                    #   DISPLAY provisioning + yielding to the user)
│   │   └── coordinator/     # three-tier fallback orchestration
│   ├── tools/               # ui_tree/find/action/layout/screenshot/screen_text/windows/apps
│   └── utils/               # refs/errors/logging/blocks/temps
├── tests/                   # unit tests + zenity end-to-end + manual_* stories
├── demo/                    # Phase 0 feasibility verification
├── docs/                    # deployment guide
├── entry.py                 # PyInstaller entry point
├── build.sh                 # packaging script
└── pyproject.toml

The core/display, core/coordinator and backend/linux/inject packages were split out of single modules by responsibility. Their public namespaces are unchanged (display.MANAGER, inject.XdotoolInjector, coordinator.Coordinator are re-exported from __init__.py), so every existing import site kept working.


10. Known limitations

  • Scenes with no accessibility tree (custom-drawn widgets, games, video, remote desktops, DRM): fall back to get_screen_text. Known limitation: a line of text immediately adjacent to a dark icon can fail to recognise — narrow the region and retry.

  • Coordinate drift: some GTK dialogs return window-relative coordinates from get_extents(SCREEN). The geometry layer compensates, and element-level do_action bypasses coordinates entirely.

  • Wayland: the MVP targets X11; injection under Wayland is restricted and would need libei (under evaluation for Phase 2+).

  • uinput injection: xdotool (XTest) is used today, which suffices for X11; uinput (needs a udev rule) is planned for Phase 2.3.

  • Non-ASCII input goes through the clipboard: xdotool type rewrites the global keyboard mapping for CJK input, which makes the user's physical keyboard unresponsive for the duration and races with input-method XKB state. Only pure ASCII uses xdotool type.


Roadmap

  • Phase 0 Feasibility verification (demo/)

  • Phase 1 Linux MVP (this project: 14 tools + backend + core + packaging)

  • Phase 1.5 Isolation sandbox (Xephyr virtual screen by default + private AT-SPI bus + launch_app)

  • Phase 2 Precision hardening + token optimization (tree diffing, uinput, focus handling)

  • Phase 3 Windows backend (UIA + SendInput)

  • Phase 4 Grey-area fallbacks + visual parsing + extra tools (scroll/drag)


License

GNU Affero General Public License v3.0 — Copyright (C) 2026 刘飞 (liufei)

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Because this is an AGPL network-copyleft license, if you run a modified version of this software to provide a service over a network, you must offer the corresponding source to that service's users.

Available Tools

14 tools
act_sequenceA

一次调用批量执行一串动作——省往返的关键工具。⭐ 使用判据:凡是下一步不依赖上一步结果的连续动作,一律一次提交;每多一次单独调用,就多一个「模型思考 + 读结果」的来回(实测每轮 30~70 秒,而工具本身只要零点几秒)。只有需要看结果做分支判断时才拆成多次调用。steps 每项含 op 字段:click{ref?,text?,role?,app?,button?,x?,y?} / type{text,ref?,clear_first?} / key{combo} / wait{title_contains?,window_id?,timeout?} / sleep{seconds} / list_windows{} / screenshot{region?,max_side?}。把 screenshot 放在最后一步,可以在同一次调用里拿到「这一串做完之后长什么样」,省掉单独截图的整个来回(确认类截图占实测截图的三分之二)。stop_on_error=true 时某步失败即停止后续。返回每步 ok/message 的 JSON;点击类步骤的 message 里带落点证据(落在哪扇窗、底下什么字、点后活动窗口),据此判断有没有点偏,不必再截图确认。

ParametersJSON Schema
NameRequiredDescriptionDefault
stepsYes
stop_on_errorNo

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden, and it excels. It discloses the return format (JSON with per-step ok/message), the stop_on_error semantics, the step op types and their optional fields, and the landing-point evidence in click messages. It also reveals performance characteristics (measured round-trip times) and the advice about screenshots, giving an agent a realistic mental model of execution and side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but every sentence earns its place. It front-loads the core value proposition and usage rule, then systematically covers step ops, the screenshot tip, stop_on_error, and return details. The bold and star markers guide attention. For a tool with this many op variants and options, the density is appropriate—no filler, all actionable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (7 op types with numerous optional fields) and the absence of an output schema, the description covers all critical aspects: op syntax, usage heuristics, error behavior, return structure, and a practical optimization. It even addresses a common failure mode (off-target clicks) and how to verify without extra calls. An agent has everything needed to call it correctly and efficiently.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description fully compensates. It enumerates every op (click, type, key, wait, sleep, list_windows, screenshot) with their sub-fields, and explains the stop_on_error parameter's default and effect. This is far richer than the generic 'steps: array of objects' schema, enabling an agent to construct valid step objects without guessing.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear purpose: batch-executing a sequence of actions in one call to save round trips. It distinguishes itself from sibling tools (click, type_text, etc.) by framing it as the aggregation tool, and explicitly names the batching criterion. The verb 'execute' and resource 'a string of actions' are concrete, and it even quantifies the benefit (30–70s per round trip vs. sub-second tool call).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides explicit, actionable usage criteria: submit consecutive actions whose next step doesn't depend on the previous result, and only split when you need to branch on results. It also gives a specific optimization—placing screenshot as the last step to save a round trip—and explains stop_on_error behavior. This directly tells an agent when to use this tool versus individual sibling calls.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

clickA

点击一个元素。优先传 ref(来自 get_ui_tree/find_element/get_screen_text);也可传 text(+app) 现场匹配第一个元素。执行采用三级降级:①元素级 do_action(零坐标,首选)→ ②校准坐标点击(兜底)→ ③都失败则提示改用 screenshot。返回结果会标明实际生效的层级。坐标级点击会回报「落点证据」(落在哪扇窗、底下什么字、点后活动窗口)、一张点击前抓的准星小图(红十字 = 你刚才点的那个像素,未缩放、图上 1 像素 = 屏幕 1 像素),以及程序算好的偏差数值:最近文字块离落点多远;若你传了 expect(如 expect="保存"),还会直接给出「未命中,偏差 (+17,-42) 共45px,建议改点其中心 (517,258)」——按建议坐标重试即可,不必再截图估算。另有「点后界面变化」百分比作为命中参考(无变化 = 大概率点空)。preview=false 可关掉这套画面反馈(更省 token,代价是失去位置依据)。灰区应用(无元素树,如 SWT/自绘控件)可传裸坐标 x+y(屏幕绝对坐标,可先用 get_screen_text 读出坐标)直接坐标级点击,无需再借道 shell。

ParametersJSON Schema
NameRequiredDescriptionDefault
xNo
yNo
appNo
refNo
roleNo
textNo
actionNo
buttonNo
expectNo
previewNo

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations are absent, so the description carries the full disclosure burden and discharges it thoroughly. It reveals the three-tier internal execution fallback (element-level do_action → calibrated coordinate click → suggest screenshot), that the return states the effective tier, and the full set of coordinate-click return artifacts: landing evidence, unscaled crosshair snapshot, computed offset from the nearest text block, expect-driven corrective coordinates, and the post-click change percentage as a miss indicator.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long, but every clause earns its place given the 0% schema coverage and absent annotations, and it is front-loaded with the core purpose and preferred mode before details. The ①②③ fallback chain, bolded key terms, and arrow/percent framing keep the density structured; the crosshair pixel-mapping passage is valuable but could arguably be tightened.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 10-parameter tool with no annotations, no schema descriptions, and no output schema, the description supplies everything an agent needs to call it correctly: targeting modes, fallback mechanics, return artifacts, the expect retry loop, and the preview tradeoff. Remaining gaps are minor: role is undocumented, button is underexplained, and the error-return format beyond the effective-tier indicator is not specified.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description compensates strongly: it explains ref (source tools and precedence), text(+app) first-match semantics, x/y as absolute screen coordinates, expect (enables miss-offset plus suggested retry coordinates), preview (toggles the visual feedback payload), and action via the element-level do_action reference. However, role is never mentioned and button's semantics are left entirely to its default value, so coverage of the 10 parameters is strong but not complete.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States the purpose immediately and specifically: '点击一个元素' (click an element), a clear verb+resource. It then enumerates the three targeting modes (ref, text+app, bare coordinates), and distinguishes itself from siblings by naming get_ui_tree/find_element/get_screen_text as the sources of ref and screenshot as the fallback path, so an agent can tell it apart from the find/read/keyboard siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit priority ordering: prefer ref, fall back to text(+app) first-match, and use bare x+y only for gray-zone apps lacking an element tree. It also states when not to use the feedback payload (preview=false to save tokens) and when to abandon the tool (both fallback tiers fail → switch to screenshot), which is explicit when/when-not guidance with named alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

element_infoA

按 ref 查看单个元素的详情:角色、名字、值、状态、屏幕矩形、可用 actions。在 click/type 前用它确认元素是否正确、有哪些可执行动作。

ParametersJSON Schema
NameRequiredDescriptionDefault
refYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the behavioral burden. The verb '查看' (view/look up) makes the read-only nature clear, and the list of returned fields tells the agent what to expect. It doesn't discuss invalid/stale refs or error behavior, but for a simple inspection tool this is a minor gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two compact sentences: the first front-loads the purpose and output fields, the second adds the usage context. Every clause carries information and there is no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter read-only tool with an output schema, the description covers the main need: what the tool returns and when to use it. The main missing piece is the provenance/lifetime of the ref value, but the sibling tool list and simple signature keep this a minor gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must explain the ref parameter; it only restates '按 ref' (by ref) as a lookup key. It never says what a ref is, where it comes from (e.g., get_ui_tree/find_element), or that it must be from the current UI snapshot. This leaves the only parameter under-specified.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('view') and names the resource ('single element by ref'), then enumerates exactly what detail fields are returned. This distinguishes it from sibling tools like click/type_text (which perform actions) and get_ui_tree/find_element (which search/collect elements) without needing to open the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit usage context: use before click/type to confirm the target element and its available actions. It doesn't name alternative tools or state when not to use it, so it misses the full when/when-not structure, but the context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

find_elementA

按文本/角色/应用搜索可交互元素,返回候选列表(每个含 [ref])。用 ref 配合 click/type_text 精确操作,避免坐标估算。text 为名字子串(如『保存』『是』),role 为角色子串(如 'push button';注意 GTK 输入框的 role 是 'text' 而非 'entry'),app 限定应用名关键词。至少提供 text 或 role 之一。强烈建议带 app 或 text 限定:全桌面裸搜很慢,且遍历大型应用(浏览器/Electron)时有节点数熔断,结果可能不全。

ParametersJSON Schema
NameRequiredDescriptionDefault
appNo
roleNo
textNo
limitNo
interactive_onlyNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

无注解,描述承担全部责任。披露了关键行为:全桌面裸搜慢、遍历大型应用有节点数熔断可能导致结果不全,以及 GTK 输入框 role 为 'text' 而非 'entry' 的细节。这些超出可推断范围,提供了有价值的限制信息。

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

描述为中文段落,信息密度高,包含用法、性能警示和技术细节,但语句组织有序,无冗余。虽稍长,但每句都提供实用信息,属于效率高的结构。

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

有输出 schema 因此返回格式有定义,描述已说明返回含 [ref] 的候选列表。参数虽多但主要参数已解释,且提供了使用建议和性能约束,整体足以让代理正确调用。小幅缺失在于未覆盖 limit 和 interactive_only 的细节,但影响有限。

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema 描述覆盖率为 0%,描述必须补偿。详细解释了 text(子串匹配)、role(含 GTK 特例)、app(限定应用名)三个参数的含义,但未说明 limit 和 interactive_only 的语义。interactive_only 可从描述中'可交互'推断,但 limit 完全未提,覆盖不完整。

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

明确说明'按文本/角色/应用搜索可交互元素',动词(搜索)和资源(可交互元素)清晰,且返回候选列表含 [ref] 的用途明确。虽未直接命名兄弟工具,但'用 ref 配合 click/type_text 精确操作'暗示了与其他交互工具的分工,足以区分。

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

提供了明确的使用条件:至少提供 text 或 role 之一,并强烈建议带 app 或 text 限定,说明全桌面裸搜慢且有熔断风险。虽未明确指向替代工具(如 element_info),但给出了使用时的最佳实践和性能提示,较清晰。

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_last_click_imageA

回看最近一次坐标点击的准星小图与结论(index=-1 最近,-2 上上次)。⭐ 什么时候用它:坐标点击之后界面没有预期反应(没弹窗、没跳转、没变化)时,先用它看清「刚才到底点在哪、程序算的偏差是多少、建议改点哪个坐标」,再据此修正重试;不要盲目重复点击(实测那是最容易把界面点花的做法),也不必重新截图重新估算。返回里含:落点坐标、当时程序算出的偏差与建议坐标、点后界面变化百分比。记录只保留本会话内最近 8 次坐标点击;元素级点击零坐标、不产生记录。

ParametersJSON Schema
NameRequiredDescriptionDefault
indexNo

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full behavioral disclosure. It states what the return contains (click coordinates, computed deviation, suggested coordinates, change percentage), and discloses retention limits (last 8 coordinate clicks) and an exclusion (element-level clicks produce no record).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but every sentence earns its place: purpose/index, when-to-use, return contents, and retention constraints. It is front-loaded with the core function and uses bold and an emoji for scannability without adding filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Since there is no output schema and no annotations, the description must stand alone—and it does. It explains what the tool returns, when to use it, how the index parameter works, and data retention limits. Nothing needed for correct invocation or interpretation is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides only a default value of -1 with no description. The tool description adds complete semantic meaning: index=-1 is the most recent click, -2 is the one before that, and records are limited to the last 8 clicks. This fully compensates for the schema's 0% coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action: review the crosshair image and conclusion of the most recent coordinate click, with index semantics explained. It clearly sets this apart from sibling tools like click or screenshot, which perform actions or capture screens rather than diagnose past clicks.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit trigger condition: use it when a coordinate click produces no expected UI reaction (no popup, navigation, or change). It also tells the agent not to blindly repeat clicks and says re-screenshotting is unnecessary, which is direct, actionable usage guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_screen_layoutA

获取显示器布局:各屏幕名称、分辨率、在虚拟桌面中的偏移(x,y)、是否主屏,以及虚拟桌面总尺寸。多屏/坐标问题排查时先调它理解几何环境。返回 JSON 文本。

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses the return format ('返回 JSON 文本') and the specific data fields, and '获取' implies a read-only operation. It doesn't explicitly state absence of side effects, but for a getter this is sufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no redundancy. The first sentence lists what it returns, the second provides usage guidance. Information is front-loaded and every phrase adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Output schema exists, so return structure is already documented. The description covers purpose, usage context, and return format. A minor gap is the lack of an explicit non-destructive statement, but given the tool's simplicity, it is nearly complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so baseline is 4. The description doesn't need to explain parameters and instead focuses on the output structure, which is acceptable.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('获取') and resource ('显示器布局'), then enumerates the exact data fields: screen names, resolutions, offsets, primary flag, and virtual desktop size. This clearly differentiates it from sibling action tools like click or type_text.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly says '多屏/坐标问题排查时先调它理解几何环境', giving a concrete scenario and priority ('call it first'). It doesn't explicitly contrast with alternatives, but the guidance is clear and actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_screen_textA

读取屏幕上的文字及其坐标,返回紧凑文本列表(每行 [ref] 文字 @ (x,y) 宽x高)。灰区应用(无元素树:SWT/Java、自绘控件、游戏、远程桌面)首选感知方式——它比 screenshot 省得多:你读文字就能定位,不需要看图、也不需要从图像里估算像素位置再换算回屏幕坐标(那正是灰区操作慢和点偏的主因)。拿到结果后可直接 click(x=..,y=..),或 click(ref=N) 由服务端代点(避免抄错坐标)。scope='window'(默认)只识别当前活动窗口那块区域,23 秒;scope='screen' 整屏,文字密集时要 810 秒,非必要别用。也可传 region=[x,y,w,h] 自己指定区域。注意:坐标是快照,界面变化(切窗/弹新对话框)后请重新调用,不要复用旧 ref。⚠️ 已知限制:文字紧邻深色图标时那一行会识别失败(实测「确认删除该文件吗?」挨着问号图标时被认成乱码)。若发现某行文字明显不对,把 region 收窄到只含该文字的小块再调一次即可(可先用本次返回的该块 bbox 定位)。

ParametersJSON Schema
NameRequiredDescriptionDefault
scopeNowindow
regionNo
min_confNo
max_itemsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden and delivers richly: it discloses timing expectations (2-3s vs 8-10s), the snapshot semantics of coordinates with the warning not to reuse stale refs after UI changes, and a concrete known failure mode (text next to dark icons misrecognized) plus a mitigation (narrowing the region). This is exemplary disclosure for a sensing tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but front-loaded and tightly organized: function+format, when to use, how to chain with click, scope timing, region, then snapshot warning and known limitations. Each sentence earns its place given the tool's timing and failure-mode complexity, though the size is on the higher end and could tolerate minor trimming.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 4 parameters, no annotations, and an output schema, the description covers nearly everything an agent needs: return format, scoping behavior with latencies, follow-up click usage, snapshot invalidation, and a failure-mode workaround. Small gaps remain — the semantics of min_conf/max_items, behavior when no text is found, and the coordinate reference frame (screen vs window-relative) are only implied.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate. It adds deep semantics for scope ('window' default targeting the active window, 'screen' full-screen with heavy cost) and region (arbitrary [x,y,w,h] specification), which the bare schema cannot convey. However, min_conf and max_items receive no explanation at all, leaving two of the four parameters semantically undocumented.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: '读取屏幕上的文字及其坐标' (read on-screen text and its coordinates), and specifies the exact return format `[ref] 文字 @ (x,y) 宽x高`. It differentiates from siblings by positioning itself as the preferred sensing method for gray-area apps with no element tree (SWT/Java, custom-drawn controls, games, remote desktop), which clearly separates it from get_ui_tree/find_element and screenshot.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use it (gray-area applications) and contrasts it against the alternative (screenshot): reading text avoids guessing pixel positions, which is 'the main cause of slow and offset clicks'. It also gives scope-specific guidance — window scope for the active window (2-3s) vs screen scope (8-10s) with an explicit '非必要别用' (don't use unless necessary) — and prescribes the follow-up action of click(x,y) or click(ref=N).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_ui_treeA

读取桌面无障碍元素树(结构化文本),这是感知屏幕的首选方式,不要默认用 screenshot。返回紧凑文本树,每个可操作元素带 [ref] 编号,后续用 click(ref)/type_text(ref) 精确操作,无需估算坐标。scope=active_window 最省 token(默认);找特定应用用 scope=app+app=名字关键词;整桌面用 scope=desktop(大,慎用)。interactive_only=true 只列可操作元素,进一步省 token。

ParametersJSON Schema
NameRequiredDescriptionDefault
appNo
scopeNoactive_window
max_nodesNo
interactive_onlyNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and does well: it explains the compact text-tree return, the [ref] mechanism for click/type_text, and the token-cost implications of scope and interactive_only. It does not disclose edge behaviors like failure modes or output size limits, but the core behavioral profile is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single dense paragraph with no filler, front-loading the most important guidance (preferred over screenshot) before explaining return format and parameters. Every clause adds operational value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given an output schema exists, the description does not need to detail return structure, and it covers the main use cases, scope routing, and ref-based interactions. The only notable gap is the undocumented max_nodes parameter and a missing note about when screenshot might still be necessary, but overall the tool can be invoked correctly from this description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It adds real meaning for scope values, the app keyword matching, and interactive_only, but never explains max_nodes or its default of 400, leaving one parameter semantically underspecified.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('读取') and resource ('桌面无障碍元素树'), and immediately differentiates itself from screenshot by declaring itself the preferred screen-perception method. This makes it unambiguously distinct from siblings like screenshot, get_screen_text, and get_screen_layout.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It clearly states when to use this tool ('这是感知屏幕的首选方式') and explicitly warns against defaulting to screenshot, while also giving concrete scope-selection guidance for active_window, app, and desktop. It does not state explicit exclusions or when screenshot would be preferable, so it stops short of a perfect 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

launch_appA

在目标 display 启动应用:isolated 模式即放进 Xephyr 沙箱(把应用放上虚拟屏的唯一正路,宿主桌面不受影响);real 模式即普通启动。command 用 shlex 解析(如 'zenity --entry --title=T'、'gedit'),返回 {pid,command,display} 的 JSON。启动后配合 wait_window 等窗口出现,再 get_ui_tree/click 操作。

ParametersJSON Schema
NameRequiredDescriptionDefault
settleNo
commandYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the transparency burden and does well: it discloses sandboxing behavior, that commands are parsed with shlex, and that the result is a JSON object with pid, command, and display. Failure behavior and permissions are not covered, but key launch semantics are.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded: the core action comes first, followed by the key behavioral caveat, examples, return shape, and suggested workflow. It is dense but not wasteful.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description provides useful workflow and return info, but it omits the meaning of settle and does not explain how the isolated/real mode is chosen given the schema only exposes command and settle. This leaves a meaningful gap for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It documents command with shlex parsing and examples, but it never explains the settle parameter, and it introduces isolated/real modes without saying how those modes are selected through the provided schema. An agent cannot fully determine argument semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states it launches an application on the target display and distinguishes isolated vs real modes. This clearly separates launch_app from sibling tools, which are all interaction/inspection operations like click, type_text, and get_ui_tree.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explains when isolated mode is appropriate (the only way to put an app on the virtual screen without affecting the host) and gives a follow-up workflow: use wait_window, then get_ui_tree/click. It does not name alternative launch tools, but the context is clear enough.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_windowsA

一次列出当前可见窗口:[{id,title,pid,x,y,w,h,area}],按面积降序(面积最大者通常是主窗口;未映射的孤儿窗/隐藏辅助窗已被过滤)。用于甄别同名窗、定位目标窗口 id,替代多次 shell 查询。返回 JSON 文本。

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It discloses that the output is a JSON list sorted by area descending, filters out unmapped orphan windows and hidden helper windows, and returns JSON text. This is substantial behavioral context for a read-only listing tool, though it doesn't mention rate limits or authentication (likely irrelevant). Score 4 is appropriate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, dense sentence that packs purpose, output format, sorting, filtering, and use case without unnecessary fluff. It is front-loaded with the main action. It could be slightly more readable but is still efficient, so a 4 is justified.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The presence of an output schema means the description need not explain return values in detail, and it does list the fields. However, the 'limit' parameter is completely undocumented, and there is no mention of error conditions or edge cases. For a tool with a single optional parameter, this is a notable omission, making the description incomplete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has one optional parameter 'limit' with a default of 100, and schema description coverage is 0%. The description never mentions 'limit' or explains its purpose or effect. Since the description does not compensate for the schema's lack of documentation, the parameter semantics are poorly conveyed. A score of 2 reflects this gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists visible windows with a specific output structure (id, title, pid, x, y, w, h, area) and indicates sorting and filtering. It gives a concrete use case (distinguishing same-name windows, locating target window id). However, it does not explicitly differentiate itself from sibling tools like get_screen_layout or get_ui_tree, so it falls short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context: it is meant for identifying windows by id and replacing multiple shell queries. It implies when to use it, but does not explicitly state when not to use it or name alternative tools. This meets 'clear context, no exclusions', which is a 4.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

press_keyA

发送快捷键(全局键盘注入)。combo 形如 'ctrl+s'、'alt+F4'、'Return'、'Tab'、'ctrl+shift+t'。修饰键用 ctrl/alt/shift/super;常见别名(PageDown/Enter/Esc/方向键等)会自动归一为 xdotool keysym;键名不识别会明确报错而非静默无操作。会回报按键后的活动窗口标题(快捷键生效与否取决于焦点在谁身上)。

ParametersJSON Schema
NameRequiredDescriptionDefault
comboYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully covers behavioral details: it explains the alias normalization to xdotool keysym, states that unrecognized key names cause explicit errors rather than silent failures, and notes that the returned active window title depends on focus. This is comprehensive for a single-purpose tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is information-dense but not bloated—each sentence adds value (purpose, format, modifiers, aliases, errors, return value). It is structured logically, though slightly long; a tighter phrasing could earn a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (one parameter) and the presence of an output schema (which the description complements by mentioning the active window title), the description is complete. It covers invocation, input format, edge cases, and return behavior—nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% (the combo parameter has no schema description), but the description provides exhaustive guidance: exact syntax examples, accepted modifier keys, alias behavior, and error handling. It fully compensates for the missing schema information.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: '发送快捷键' (send hotkey) and '全局键盘注入' (global keyboard injection), which clearly distinguishes it from siblings like type_text (text input) and click (mouse action). The format examples further clarify the exact operation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The purpose is immediately evident—sending key combinations—and the global scope is explicit. However, it does not explicitly state when NOT to use it (e.g., for typing text use type_text), though the distinction is strongly implied by the sibling names and the term 'keyboard injection'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

screenshotA

截图(灰区兜底,默认不要主动用)。仅当目标无元素树(自绘控件/游戏/视频/远程桌面)或 get_ui_tree/find_element 无法定位时才使用——它费 token。可传 region=[x,y,w,h] 裁剪感兴趣区域;max_side 控制降采样长边(默认1280,省 token)。inline=false 时不返回图像、只落盘并返回文件路径(更省 token/更快),适合只需存档或后续自行读取的场景。此时可用 save_path 指定落盘位置——已存在的文件不会被覆盖(会直接报错),需要改路径或留空让本工具自动存到临时目录(自动路径只保留最近 5 张、旧的会被回收,要长期留存必须显式传 save_path)。优先用 get_ui_tree + click(ref) 完成任务。

ParametersJSON Schema
NameRequiredDescriptionDefault
inlineNo
regionNo
max_sideNo
save_pathNo

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description fully discloses behavior: token cost, region cropping, max_side downsampling, inline=false behavior (file path instead of image), non-overwrite error, and automatic path retention (only last 5 images). These are all behavioral traits an agent needs to know and are not derivable from the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single dense paragraph that front-loads the critical caution ('default not to use') before diving into parameters. Every sentence adds value, but the length is slightly high; still, given the complexity of 4 parameters and the need for behavioral context, it remains well-structured and efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 4 parameters, no annotations, and no output schema, the description covers usage context, parameter semantics, and behavioral constraints thoroughly. An agent has everything needed to decide when and how to call it correctly, including edge cases like file overwrite and auto-path cleanup.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must explain each parameter. It does: inline (return image vs. file path), region (crop area), max_side (downsample long side, default 1280 to save tokens), save_path (specify location, with overwrite and retention warnings). This adds substantial meaning beyond the schema's type/default info.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function (take a screenshot) and explicitly positions it as a fallback tool for cases where element trees are unavailable. It distinguishes itself from siblings like get_ui_tree and find_element by naming them and specifying when they fail, making the purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use (no element tree, self-drawn controls, games, video, remote desktop) and when-not-to-use (default, prefer get_ui_tree + click). It also mentions the alternative tool directly, leaving no ambiguity about the decision process.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

type_textA

输入文本。若给 ref(文本框/输入区),优先用元素级 set_value(最稳,不受焦点影响);否则/失败时降级为键盘注入(xdotool type)。clear_first=true 先全选删除原内容再输入。建议:先 click(ref) 聚焦目标输入框,再 type_text。键盘注入会回报输入后的活动窗口标题——焦点被别的应用抢走时一眼可见,不必再截图核对。

ParametersJSON Schema
NameRequiredDescriptionDefault
appNo
refNo
textYes
clear_firstNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses the two execution modes (set_value vs keyboard injection), the effect of clear_first (select-all delete), and the behavior of reporting the active window title after keyboard injection, which aids verification. It doesn't mention potential failures or edge cases, but for a text-input tool this is reasonably transparent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise (about three sentences) and front-loaded with the core purpose. The fallback logic, suggestion, and reporting note are each purposeful, with no redundant information. It is well-structured and easy to scan.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (4 params, no annotations) and the presence of an output schema, the description covers the key operational details: method selection, clear behavior, focus suggestion, and verification hint. The undocumented 'app' parameter is a minor gap, but overall the description provides enough context for an agent to call the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It explains ref (triggers set_value) and clear_first (clears existing content), and text is implied. However, the 'app' parameter is not explained, leaving a gap. Since it covers two of four parameters meaningfully but not all, a score of 3 reflects partial compensation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: '输入文本' (input text). It specifies the primary method (element-level set_value) and the fallback (keyboard injection), which distinguishes it from sibling tools like press_key (key presses) and click (clicking). The verb and resource are explicit, and the behavior is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidance: it instructs to prefer set_value when a ref is given, otherwise fall back to keyboard injection, and it recommends clicking the target input first. It explains when to use clear_first. While it doesn't explicitly contrast with sibling tools, the internal decision logic and actionable suggestion give clear context for usage.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

wait_windowA

等待窗口标题满足条件(server 内部轮询,单次调用完成)。title_contains 忽略大小写;window_id 指定则只盯该窗口,否则盯活动窗口;超时返回超时提示。用于等页面加载/应用启动,替代外部反复轮询。

ParametersJSON Schema
NameRequiredDescriptionDefault
timeoutNo
window_idNo
title_containsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It explains internal polling, single-call completion, case-insensitive matching, window targeting behavior (specific window vs active window), and timeout behavior, all beyond what the schema conveys. This is strong transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and well-structured: core behavior first, then parameter semantics, then usage guidance. Every sentence adds value, with no filler or repetition of the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers purpose, behavior, parameters, and usage context well. However, it does not clarify what happens when title_contains is null despite being optional in the schema, nor the timeout unit. Since there is an output schema, return format is not a concern, but these two gaps slightly reduce completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description compensates for all three parameters: title_contains (case-insensitive matching), window_id (scoping to a specific window or active window), and timeout (timeout returns a timeout message). However, it does not specify the timeout unit (seconds vs milliseconds), which is a meaningful gap for correct invocation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: waiting for a window title to meet a condition, with server-side polling in a single call. This makes the tool's function obvious and distinguishes it from action-oriented siblings. However, it does not explicitly differentiate from any sibling tool by name, so it misses the top score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to use it: 'for waiting for page load/app startup', and what it replaces: 'external repeated polling'. This gives an agent clear guidance on when to choose this tool over manual polling loops, satisfying the usage-guideline criterion.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 14 tool updatesv0.1.0
    • First observedact_sequence
    • First observedclick
    • First observedelement_info
    • First observedfind_element
    • First observedget_last_click_image
    • First observedget_screen_layout
    • First observedget_screen_text
    • First observedget_ui_tree
    • First observedlaunch_app
    • First observedlist_windows
    • First observedpress_key
    • First observedscreenshot
    • First observedtype_text
    • First observedwait_window

TDQS

A4.2/5.0

Scored across 14 tools

Disambiguation5/5

Every tool has a clearly distinct purpose: input (press_key, click, type_text), perception (get_ui_tree, find_element, get_screen_text, screenshot, list_windows, get_screen_layout), automation (act_sequence), waiting (wait_window), launching (launch_app), and inspection (element_info). No two tools appear to do the same thing; even perception tools are differentiated by method and use case.

Naming Consistency4/5

Most tools follow a consistent verb_noun pattern (press_key, type_text, get_screen_text, list_windows, wait_window, launch_app, get_ui_tree, find_element). A few are single words (click, screenshot) or noun-centric (element_info), but they are still clear and not confusing. Minor deviation from a strict convention.

Tool Count5/5

14 tools is well-scoped for a computer-use server. The set covers perception, input, automation, window management, and app launching without redundancy or bloat. Each tool earns its place for a comprehensive GUI automation domain.

Completeness4/5

The tool surface covers the core computer-use workflow: perceive (tree, text, screenshot, windows), interact (click, type, key), automate (sequence), wait, and launch. Minor gaps include no explicit scroll or drag-and-drop, but these can be handled via key combos or coordinates. Overall, no dead ends for typical tasks.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI assistants to interact with native Linux desktop applications through AT-SPI2 accessibility interfaces. Provides semantic element targeting, natural language search, and automation capabilities (clicking, typing, keyboard shortcuts) across GTK, Qt, and Electron applications.
    6
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI agents to control Linux/X11 desktops by providing tools for taking screenshots, clicking, typing, and managing windows via AT-SPI and xdotool.
    3
    MIT
  • A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to operate a real GNOME Wayland desktop through accessibility-tree widget actions, pointer/keyboard input, OCR, window management, and screen capture, optionally on a private headless session.
    33
    26 PyPI
    8
    Apache 2.0
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI agents to drive Linux desktop applications through the AT-SPI2 accessibility bus, addressing UI elements by role and name to read text, set values, and invoke actions without relying on pixels or synthetic keystrokes.
    MIT