cc-computer-use
An MCP server that lets an AI agent perceive and operate a Linux desktop (by default an isolated private virtual screen) through accessibility-tree/element-level actions rather than screenshots and coordinate guessing.
Read the UI without screenshots:
get_ui_treereturns a compact accessibility tree with[ref]IDs;get_screen_textOCRs text + coordinates for grey-area apps; both are far cheaper than screenshots.Find and inspect elements:
find_elementsearches by text/role/app;element_infoshows an element's value, states, rect, and available actions.Act on elements without coordinates:
click(ref)/type_text(ref)use element-level AT-SPI actions; coordinate fallback is automatic when needed.Coordinate clicking with feedback:
click(x,y)/double_click/scroll/drag, with landing evidence, crosshair preview, computed aim offset, and post-click change detection;get_last_click_imagelets you review the last click.Type and send keys:
type_textwith element-first input;press_keyfor combos likectrl+s; both report the active window afterward.Batch actions:
act_sequenceruns multiple steps in one call, optionally ending withui_tree/screenshotto see the resulting state.Manage apps and windows:
launch_appstarts apps in the sandbox/target display;list_windows,wait_window, andget_screen_layoutprovide window/display context.Sandbox isolation by default: the agent operates on its own Xephyr virtual screen with a private AT-SPI bus;
CC_CU_DISPLAY_MODE=realswitches to the real desktop intentionally.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@cc-computer-use open the terminal and run 'ls -la'"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
cc-computer-use — Computer-Use MCP Server
中文文档 | English
An MCP server that lets an AGENT (Claude Code, etc.) drive a Linux desktop — without taking over yours.
Most computer-use servers do the same thing: they seize your mouse, your keyboard and your screen for the whole run. This one puts the agent on its own private virtual screen by default, so you keep working while it does. When you do want it driving the real desktop, one environment variable switches it over.
Conventional computer-use MCP | This project | |
While the agent works | Mouse / keyboard / screen are seized — you can't use your machine | Untouched — the agent gets a private Xephyr virtual screen |
Isolating the agent | You build the VM or container yourself | On by default, zero setup |
Sandbox unavailable | Silently falls back to your real desktop | Refuses to run and says why |
The a11y tree | One AT-SPI bus shared with your desktop | Private AT-SPI bus — the host bus cannot see sandboxed apps |
Anthropic's own computer-use reference says running such an agent outside a VM is "strongly discouraged" and that there are "no safeguards". This server takes that seriously — isolation is the default here, not a setting you have to get right.
How it perceives
Perception goes through the accessibility tree (AT-SPI) and action through element-level operations, instead of the conventional "screenshot + coordinate click". That removes two more problems at the root:
Pain point | Conventional approach | This project |
Screenshots burn tokens | 1000+ tokens per screenshot, every round | Read the a11y tree / OCR text — both compact text |
Clicks miss the target | LLM estimates coordinates from pixels; DPI and multi-monitor stack up the error | Element-level |
Feasibility was established by measurement in
demo/— the conclusion there was that element-level actions beat coordinate clicking outright, which is why the execution layer here is "element-level first, coordinate clicking only as fallback".For a full deployment walkthrough see
docs/安装说明.md(Chinese).
Related MCP server: linux-computer-use
1. Capabilities (17 MCP tools)
Tool | Purpose |
| Read the accessibility tree as compact text, each actionable element tagged with a |
| OCR the screen into text plus coordinates ( |
| Search elements by text / role / app; returns candidates carrying |
| Inspect one element: value, states, rect, available actions |
| Three-tier fallback: element-level |
| Look back at the most recent coordinate click ( |
| Input: element-level |
| Key combos ( |
| Batch a run of actions in one call (click / type / key / wait / sleep / list_windows / ui_tree / screenshot) — the main round-trip saver. Putting |
| Double click — coordinate-level only, because element-level actions have no double-click semantics, so you must pass screen coordinates. The two clicks are timed server-side (100 ms, far below the ~400 ms system threshold), so it genuinely is a double click; two separate |
| Scroll at a point. |
| Drag from |
| Launch an app on the target display (the proper way to get an app into the sandbox) |
| Visible windows (id / title / PID / geometry, largest first) |
| Wait until a window title matches (server-side polling, single call) |
| Monitor layout and virtual desktop size |
| Screenshot — last resort, JPEG by default; use only when pixel-level judgement is genuinely needed |
Recommended workflow
Apps with a tree:
launch_app→get_ui_treeto see the structure → locate the target[ref](orfind_element) →click(ref)/type_text(ref).Grey-area apps (SWT/Java, custom-drawn widgets, games, remote desktops — no a11y tree):
get_screen_textto obtain text and coordinates →click(ref)orclick(x=.., y=..). Do not default to screenshots and let the model guess coordinates: in a measured DBeaver session of 8.3 minutes, tool calls accounted for 21 seconds while the remaining 478 seconds went entirely into "look at image → estimate pixel → convert back to screen coordinates".When unsure about a coordinate, pass
expect:click(x=512, y=384, expect="Save")makes the program compute how far off you were and which coordinate to use instead. If a click does nothing, callget_last_click_imagebefore retrying — never repeat clicks blindly.Batch any predictable sequence into a single
act_sequence. Every extra call costs a model-think + read-result round trip (30–70 s measured). Split calls only when you need to branch on the result. Unsure what the target is called? Pass a list of candidate names (text: ["Save","保存"]) — the server tries them in order and stops at the first one that exists, instead of spending one round trip per guess.
Three mechanisms that save round trips
Landing evidence — the return value of a coordinate click or key injection only tells you the event was sent (xdotool has no idea what it hit). So clicks also report:
coordinate click executed @(874,578) | landed on: window "Landing Story" 310x212 | text under point: "OK" | active window after: none (previous "Landing Story" is gone)The model immediately knows it hit "OK" and the dialog really closed — no confirming screenshot needed. (Element-level
do_actiondoesn't need this; there is no "did it hit" question.)The
ui_treeandscreenshotsteps ofact_sequence— "open a menu → read what's in it" used to take two calls. Two thirds of the screenshots in the measured session existed purely to confirm a preceding sequence. Both now fold into the same call:ui_treefor structure (text, cheap),screenshotfor pixels (expensive — grey-area apps with no tree have only this option). The image comes back as an extra image content block alongside the step log.The server nudges the model to batch. The MCP
instructionshanded to the model at session start state the rule outright: one round trip costs 30–70 s while the action itself takes fractions of a second, so consecutive actions that don't depend on the previous result belong in oneact_sequence. Without that nudge the model reads a step-by-step workflow and defaults to one call per action — which is exactly what the 96% round-trip overhead above consists of.
Click feedback: the ring, the crosshair, the computed offset
Coordinate clicks exist only in tier ② (fallback) and raw-coordinate grey-area use, and all of the
feedback below is attached automatically by backend.click_at.
Ring (
backend/linux/ring.py) — for the human watching the sandbox: a solid red ring lights up at the landing point and self-destructs after 1.5 s. Without it, a click in the virtual screen is invisible to the user. Disable withCC_CU_CLICK_RING=0.Crosshair preview image — for the model: a 480x300 crop grabbed before the click with a red cross on the landing point, unscaled (1 px on the image = 1 px on screen, so no arithmetic is needed). ~190 tokens per call.
preview=falseskips the capture entirely (zero cost); disable globally withCC_CU_CLICK_PREVIEW=0.Computed aim offset — the landing coordinate is one the program itself sent (exact), and the text block positions come from OCR (which reports
left/top/width/height). The offset is therefore pure arithmetic, not a guess. Passingexpectturns a heuristic into an exact number.Post-click change — a before/after pixel comparison of the same region (<0.5 % none, <5 % slight, ≥5 % obvious). This is an independent second signal that needs no knowledge of intent. Read the two signals together: off by 45 px but the UI changed means the button's hit area is larger than its text; off by 45 px with no change means the coordinate needs fixing.
2. Requirements
Item | Requirement |
OS | Linux (Phase 1 is Linux only; Windows is Phase 3) |
Session type | X11 (under Wayland, global input injection is restricted) |
Python | 3.12+ |
System components |
|
Accessibility | Must be enabled (see below) |
Key constraint: AT-SPI depends on the system's
libatspiand theAtspitypelib — OS-level components that cannot be bundled into a single binary._bootstrap.pypointsGI_TYPELIB_PATHat the system directory at runtime, so even as a frozen executable these components must still be provided by the host system.
3. Installation
# 1. One-time: enable the accessibility switch (otherwise GTK app trees are unreadable)
gsettings set org.gnome.desktop.interface toolkit-accessibility true
# 2. System dependencies (Debian/Ubuntu) — OS-level, not bundled into the build artifact
sudo apt install -y xdotool xserver-xephyr xclip wmctrl zenity \
gir1.2-atspi-2.0 \
tesseract-ocr tesseract-ocr-chi-sim
# 3. conda environment (Python >= 3.12); pygobject must come from conda-forge — pip cannot install gi
conda create -n cc-computer-use python=3.12 -y
conda install -n cc-computer-use -c conda-forge pygobject -y
conda activate cc-computer-use
# 4. Python dependencies + this project (editable)
PYTHONNOUSERSITE=1 pip install -U mcp cryptography pyinstaller pytest
PYTHONNOUSERSITE=1 pip install -e .Why
PYTHONNOUSERSITE=1: if anothermcplives in~/.local, it shadows the conda environment's package and causes version confusion (affecting both development and packaging). All run/build commands should carry this prefix.Note:
chi_simis the Simplified Chinese OCR language pack; without it Chinese text comes out as garbage. If your UI is English-only, setCC_CU_OCR_LANG=eng— noticeably faster.Moving to a new machine / deploying from scratch / troubleshooting: see
docs/安装说明.md(Chinese, step-by-step).
4. Running the MCP server (development mode)
# Self-check: prints backend status, tool list, screen layout (does not enter the stdio loop)
PYTHONNOUSERSITE=1 python -m computer_use_mcp.server --selftest
# Normal run (stdio transport, for MCP clients)
PYTHONNOUSERSITE=1 python -m computer_use_mcp.serverIsolation sandbox (on by default)
X11 has a single physical pointer and focus: while the agent injects input, the user cannot use the machine. So this server defaults to a visible Xephyr virtual screen (the display number is allocated automatically, one per Claude session). All injection, screenshots and geometry queries target that virtual screen — the host desktop is untouched and the user can keep working. You can also reach into the Xephyr window with your own mouse and keyboard at any time; the agent's injection politely yields until you leave.
The virtual screen starts only when this MCP is first actually used, not when Claude Code launches — otherwise every session would spawn a stray Xephyr window.
Each Claude session gets its own private screen (allocated by the X server, race-free), so concurrent sessions never fight over focus or the clipboard. Within one session, the main agent and its sub-agents share that screen, and injection is therefore mutually exclusive: whoever fails to acquire the screen lock receives an immediate "screen is busy, retry later" error rather than being silently queued (queueing would execute decisions made against a stale view of the UI).
If the sandbox is unavailable (Xephyr missing, or startup failed), injection is refused with an
explicit error rather than silently falling back to your real desktop. To operate the real desktop
on purpose, set CC_CU_DISPLAY_MODE=real.
Environment variable | Default | Effect |
|
| Set to |
| unset | Unset = allocate a free display per session; set explicitly (e.g. |
|
| Virtual screen resolution |
|
| Max seconds injection yields while the user is inside the sandbox |
|
| Set to |
| unset | Override the AT-SPI bus address for sandboxed apps (debugging) |
|
| OCR languages; |
|
| Set to |
|
| Set to |
| — | Set to |
a11y isolation: the sandbox has its own AT-SPI bus
Xephyr isolates the X11 channel (injection, screenshots) but not AT-SPI. Accessibility runs over
the session D-Bus, and the sandbox shares one at-spi2-registryd with the host. Measured on
2026-09-15: traversing the a11y tree of an app inside the sandbox crashed the host GNOME Shell.
Offline core analysis confirmed the root cause — gnome-shell's own atk-bridge calling g_object_ref
on an already-freed GObject (a use-after-free), crashing on its main loop thread.
Solution: when the sandbox starts it brings up a private AT-SPI bus (its own dbus-daemon +
at-spi2-registryd in a private directory). Sandboxed apps and the MCP's a11y reader both connect to
it — the host bus cannot see any sandboxed app, while a11y capability is fully preserved.
host GNOME Shell ── host AT-SPI bus ── host apps (gnome-shell / chrome / …)
✗ mutually invisible
sandboxed apps ──── private sandbox AT-SPI bus ── the MCP's a11y readerIf the private bus fails to start, apps receive a dead address (no a11y connection): the host stays safe but no tree is readable inside the sandbox. This is deliberate — better to have no a11y than to let traffic reach the host bus.
Why a bus address rather than
NO_AT_BRIDGE=1: the latter is a GTK3-only switch (GTK4 usesGTK_A11Y), and SWT (Java apps such as DBeaver) honours neither.AT_SPI_BUS_ADDRESSis toolchain-agnostic — GTK3/GTK4/SWT/Qt/Electron all route a11y through libatspi, all read it, and none of them fall back to the session bus (measured).Verification script:
PYTHONNOUSERSITE=1 python tests/manual_a11y_isolation.py
Operational consequence: if the sandbox crashes and is rebuilt, the bus address changes, and libatspi's
atspi_init()can only ever bind once per process — so a11y stays permanently unavailable for the rest of that session, leaving only OCR + coordinate clicking. Restarting Claude Code is the only recovery.
5. Building a standalone executable
bash build.sh
# Artifacts (onedir, not a single file):
# dist/computer-use-mcp-bin/ directory holding the real executable, ~455 MB
# dist/computer-use-mcp thin shell wrapper forwarding to it
# Verify the frozen artifact
PYTHONNOUSERSITE=1 ./dist/computer-use-mcp --selftestWhy onedir: onefile re-extracts itself to a temp directory on every launch, making cold start an
order of magnitude slower. The wrapper exists to preserve the registered path (~/.claude.json
points at dist/computer-use-mcp) — do not delete it.
Key points baked into build.sh:
Targeted
gicollection (--collect-submodules/--collect-data/--collect-binaries gi) plus explicit--exclude-modulefor the GTK family:--collect-all gidrags inGtk, which trips PyInstaller's GTK hook into collecting 185 MB of icons and 42 MB of themes — none of which this project ever imports.--collect-submodules mcp.server(not all ofmcp, which would pull inmcp.cliand itstyperdependency).--copy-metadata: mcp/pydantic and friends read versions viaimportlib.metadata.LD_LIBRARY_PATHmust prepend conda'slib: otherwise PyInstaller pairs the system's oldlibcryptowith conda's newlibssland fails at runtime withOPENSSL_3.3.0 not found.packaging/embed-atspi.shbundleslibatspi.so.0and theAtspi-2.0.typelib(into_internal/and_internal/gi_typelibs/), so the artifact no longer needs the target machine to havelibatspi2.0-0/gir1.2-atspi-2.0. Both locations are forced: the latter is the only place PyInstaller'spyi_rth_gihook looks (it assignsGI_TYPELIB_PATH, not appends), and the former sits under_MEIPASSso the subprocess-env stripping logic removes it automatically. The script also walks the typelib dependency closure —Atspi-2.0depends onDBus-1.0, which comes fromgir1.2-freedesktop; copying just the one file works fine on a dev box and fails on the target.--collect-submodules Xlibplus explicit--hidden-import Xlib.ext.shape: the click ring's X extension modules are imported dynamically by name, so they appear in no static import graph. Omit them and the ring silently works in development but vanishes from the frozen build.
Distributing to another machine requires the whole dist/ directory; the target machine still
needs the system dependencies installed (xdotool / xserver-xephyr / tesseract / AT-SPI, etc.).
Building a self-contained .deb (recommended for Ubuntu users)
Unlike build.sh above (which uses this machine's 24.04 base and therefore cannot be handed to a
22.04 user), this path builds inside a container pinned to the oldest supported base:
mkdir -p dist
# 挂到容器内的 /out/dist(脚本的最终产物路径就是它),于是包直接落在本仓库的 dist/ 下。
# 中间产物走 /out/work,留在容器里 —— 分开是为了不覆盖 build.sh 的同名产物
# dist/computer-use-mcp-bin/(两者基座不同:22.04 vs 本机 24.04)。
# CC_CU_CHOWN 是必要的:容器以 root 写入宿主目录,不交还属主的话组装目录
# 在宿主上没有 sudo 删不掉。
docker run --rm -v "$PWD":/src -v "$PWD/dist":/out/dist \
-e CC_CU_CHOWN="$(id -u):$(id -g)" \
-v /tmp/Miniforge3-Linux-x86_64.sh:/miniforge.sh:ro \
-e CC_CU_MINIFORGE_SH=/miniforge.sh \
ubuntu:22.04 bash -c 'bash /src/packaging/build-in-container.sh'
# → dist/cc-computer-use_0.1.0-1_amd64.deb (68 MB measured)
# Accept it in a clean container (install → run → click → uninstall; run it on both 22.04 and 24.04)
bash packaging/deb/verify-install.sh dist/*.deb ubuntu:22.04
# Probe the packaged artifacts for the sandbox-shell fixes (the 10-step acceptance above
# never calls launch_app("gnome-terminal"), so it can't tell which source tree was built)
bash packaging/deb/probe-deb-fix.sh dist/*.deb ubuntu:22.04Only the .deb is produced — the .mcpb (Claude Desktop bundle) is opt-in via CC_CU_MCPB=1,
since it costs ~2 minutes of zipping and isn't part of this distribution.
Re-packaging without the 19-minute rebuild: dist/cc-computer-use/ is the assembled tree the
.deb is built from, so a change to the packaging scripts alone takes ~1 minute:
docker run --rm -v "$PWD":/src:ro -v "$PWD/dist":/d -v /tmp/o:/o ubuntu:22.04 \
bash -c 'bash /src/packaging/deb/build-deb.sh /d/cc-computer-use /o'On the target machine, sudo apt install ./cc-computer-use_*.deb is all it takes — nothing else
needs installing. The nine system binaries (xdotool / Xephyr / i3 / tesseract / …), their
dependency closure, the Atspi typelib and the OCR language data all ship inside. Afterwards
cc-computer-use-setup flips the accessibility switch and registers the MCP server if Claude Code
is present (existing config is never overwritten); cc-computer-use-doctor reports on every piece.
The build must run inside ubuntu:22.04: the binaries collected on 24.04 require GLIBC 2.38,
which would exclude 22.04 — the oldest supported base. See
docs/安装说明.md section 7 for the details.
6. Wiring it into Claude Code
Add one of the following to your project or ~/.claude MCP configuration:
A. Using the packaged executable (recommended, no conda needed)
{
"mcpServers": {
"cc-computer-use": {
"type": "stdio",
"command": "/absolute/path/cc-computer-use/dist/computer-use-mcp",
"args": [],
"env": { "PYTHONNOUSERSITE": "1" }
}
}
}B. Running from the conda environment (development; source edits take effect immediately)
{
"mcpServers": {
"cc-computer-use": {
"type": "stdio",
"command": "/path/to/conda/envs/cc-computer-use/bin/python",
"args": ["-m", "computer_use_mcp.server"],
"env": {
"PYTHONNOUSERSITE": "1",
"PYTHONPATH": "/absolute/path/cc-computer-use/src"
}
}
}
}⚠️ The server name must be
cc-computer-use, notcomputer-use— the latter fails to register in practice; renaming fixes it.Claude Code must be running inside an X11 graphical session (the server subprocess inherits
DISPLAYso it can read the screen and inject input).
Once connected you can give natural-language tasks, for example:
"What buttons are clickable in the current window?" → the agent calls
get_ui_tree"Click 'Yes' in this dialog" →
find_element+click(ref), element-level, zero coordinates"Type hello in the editor and save" →
type_text+press_key('ctrl+s')
7. Testing
# Unit tests (no desktop needed): geometry calibration / ref mapping / serializer denoising /
# injection and sandbox logic. 310 tests collected; without the e2e flag: 304 passed, 6 skipped
PYTHONNOUSERSITE=1 python -m pytest -q
# End-to-end (needs X11 + zenity + Xephyr): runs INSIDE the sandbox by default, never touching the
# real desktop; requires explicit opt-in. 310 passed
CC_CU_E2E=1 PYTHONNOUSERSITE=1 python -m pytest -q
# Manual stories (6 of them; full list in docs/安装说明.md §5.4)
# Sandbox story (drives a real server): launches zenity in the sandbox → element-level click →
# asserts the host was left completely undisturbed
PYTHONNOUSERSITE=1 python tests/manual_sandbox_story.py
# Grey-area + landing-evidence story (drives the frozen artifact; run `bash build.sh` first):
# OCR yields coordinates → click → asserts the dialog actually closed
PYTHONNOUSERSITE=1 python tests/manual_ocr_story.py⚠️ While
manual_sandbox_story.pyruns, do not touch your mouse or keyboard — it asserts that the host's active window and pointer are unchanged.
The suite is split deliberately: unit tests never need a desktop, end-to-end tests require X11 plus
zenity and Xephyr (and skip automatically when unavailable), and manual_*.py files are manual
stories that drive a real server or the frozen artifact through complete task flows.
8. Architecture
Claude Code / MCP client
│ MCP protocol (stdio, JSON-RPC)
┌───────▼──────────────────────────────────────────┐
│ server.py (FastMCP/MCPServer entry) │
│ tools/ 17 tool definitions (schema + │
│ guiding descriptions) │
│ core/coordinator semantic orchestration: the │
│ three-tier fallback │
│ core/serializer tree → compact text (token │
│ saving) + ref allocation │
│ core/geometry coordinate calibration │
│ core/display isolation sandbox: the single │
│ source of DISPLAY │
│ utils/refs ref ↔ live element mapping │
│ (in-process cache) │
└───────┬──────────────────────────────────────────┘
│ backend/base.py abstract contract
┌───────▼──────────┐
│ backend/linux/ │ AT-SPI reads (atspi.py) + xdotool injection (inject/)
└──────────────────┘ (Windows backend = Phase 3)The three-tier fallback (coordinator.click):
element:
backend.invoke()→ AT-SPIdo_action, zero coordinates (preferred).coord:
geometrycalibrates an absolute screen coordinate →xdotoolfocuses the window and clicks (fallback).screenshot: when not even an element can be found (grey area) → fall back to pixels, with the LLM making the visual decision.
The returned ActionResult states which tier actually took effect.
9. Project layout
cc-computer-use/
├── src/computer_use_mcp/
│ ├── _bootstrap.py # runs first: sets GI_TYPELIB_PATH before importing gi
│ ├── server.py # MCP entry + tool registration + --selftest
│ ├── backend/
│ │ ├── base.py # Backend ABC + Rect/UINode/Element/TextBlock dataclasses
│ │ └── linux/
│ │ ├── atspi.py # AT-SPI reads + do_action/set_value
│ │ ├── inject/ # xdotool injection + window geometry
│ │ │ # (apps/base/keyboard/pointer/screens/windows)
│ │ ├── grab.py # screen capture (shared by screenshot and OCR)
│ │ ├── ocr.py # grey-area sensing: tesseract → text blocks with coordinates
│ │ ├── ring.py # visual click ring (feedback for the human)
│ │ └── backend.py # LinuxBackend composite implementation
│ ├── core/
│ │ ├── serializer.py # tree serialization (denoise / save tokens / allocate refs)
│ │ ├── geometry.py # coordinate calibration
│ │ ├── screen_lock.py # screen-exclusive lock (mutual exclusion for injection)
│ │ ├── display/ # isolation sandbox (Xephyr + private AT-SPI bus +
│ │ │ # DISPLAY provisioning + yielding to the user)
│ │ └── coordinator/ # three-tier fallback orchestration
│ ├── tools/ # ui_tree/find/action/layout/screenshot/screen_text/windows/apps
│ └── utils/ # refs/errors/logging/blocks/temps
├── tests/ # unit tests + zenity end-to-end + manual_* stories
├── demo/ # Phase 0 feasibility verification
├── docs/ # deployment guide
├── entry.py # PyInstaller entry point
├── build.sh # local packaging script (onedir + wrapper)
├── packaging/ # distribution: .deb / .mcpb
│ ├── build-in-container.sh # whole pipeline, runs inside ubuntu:22.04
│ ├── assemble.sh # lay out the distributable tree + manifest
│ ├── vendor_libs.py # dependency closure for the 9 bundled binaries, RPATH
│ ├── embed-atspi.sh # libatspi + Atspi typelib closure → _internal/
│ └── deb/ # control files, postinst, setup/doctor scripts, verification
└── pyproject.tomlThe core/display, core/coordinator and backend/linux/inject packages were split out of single
modules by responsibility. Their public namespaces are unchanged (display.MANAGER, inject.XdotoolInjector,
coordinator.Coordinator are re-exported from __init__.py), so every existing import site kept working.
10. Known limitations
Scenes with no accessibility tree (custom-drawn widgets, games, video, remote desktops, DRM): fall back to
get_screen_text. Known limitation: a line of text immediately adjacent to a dark icon can fail to recognise — narrow theregionand retry.Coordinate drift: some GTK dialogs return window-relative coordinates from
get_extents(SCREEN). Thegeometrylayer compensates, and element-leveldo_actionbypasses coordinates entirely.Wayland: the MVP targets X11; injection under Wayland is restricted and would need libei (under evaluation for Phase 2+).
uinput injection: xdotool (XTest) is used today, which suffices for X11; uinput (needs a udev rule) is planned for Phase 2.3.
Non-ASCII input goes through the clipboard:
xdotool typerewrites the global keyboard mapping for CJK input, which makes the user's physical keyboard unresponsive for the duration and races with input-method XKB state. Only pure ASCII usesxdotool type.
Roadmap
✅ Phase 0 Feasibility verification (
demo/)✅ Phase 1 Linux MVP (this project: 17 tools + backend + core + packaging)
✅ Phase 1.5 Isolation sandbox (Xephyr virtual screen by default + private AT-SPI bus +
launch_app)⬜ Phase 2 Precision hardening + token optimization (tree diffing, uinput, focus handling)
⬜ Phase 3 Windows backend (UIA + SendInput)
⬜ Phase 4 Visual parsing + further grey-area fallbacks
License
MIT — Copyright (c) 2026 刘飞 (liufei)
Available Tools
14 toolsact_sequenceA
一次调用批量执行一串动作——省往返的关键工具。⭐ 使用判据:凡是下一步不依赖上一步结果的连续动作,一律一次提交;每多一次单独调用,就多一个「模型思考 + 读结果」的来回(实测每轮 30~70 秒,而工具本身只要零点几秒)。只有需要看结果做分支判断时才拆成多次调用。steps 每项含 op 字段:click{ref?,text?,role?,app?,button?,x?,y?} / type{text,ref?,clear_first?} / key{combo} / wait{title_contains?,window_id?,timeout?} / sleep{seconds} / list_windows{} / screenshot{region?,max_side?}。把 screenshot 放在最后一步,可以在同一次调用里拿到「这一串做完之后长什么样」,省掉单独截图的整个来回(确认类截图占实测截图的三分之二)。stop_on_error=true 时某步失败即停止后续。返回每步 ok/message 的 JSON;点击类步骤的 message 里带落点证据(落在哪扇窗、底下什么字、点后活动窗口),据此判断有没有点偏,不必再截图确认。
Inputs: steps (required), stop_on_error.
clickA
点击一个元素。优先传 ref(来自 get_ui_tree/find_element/get_screen_text);也可传 text(+app) 现场匹配第一个元素。执行采用三级降级:①元素级 do_action(零坐标,首选)→ ②校准坐标点击(兜底)→ ③都失败则提示改用 screenshot。返回结果会标明实际生效的层级。坐标级点击会回报「落点证据」(落在哪扇窗、底下什么字、点后活动窗口)、一张点击前抓的准星小图(红十字 = 你刚才点的那个像素,未缩放、图上 1 像素 = 屏幕 1 像素),以及程序算好的偏差数值:最近文字块离落点多远;若你传了 expect(如 expect="保存"),还会直接给出「未命中,偏差 (+17,-42) 共45px,建议改点其中心 (517,258)」——按建议坐标重试即可,不必再截图估算。另有「点后界面变化」百分比作为命中参考(无变化 = 大概率点空)。preview=false 可关掉这套画面反馈(更省 token,代价是失去位置依据)。灰区应用(无元素树,如 SWT/自绘控件)可传裸坐标 x+y(屏幕绝对坐标,可先用 get_screen_text 读出坐标)直接坐标级点击,无需再借道 shell。
Inputs: x, y, app, ref, +6 more.
element_infoA
按 ref 查看单个元素的详情:角色、名字、值、状态、屏幕矩形、可用 actions。在 click/type 前用它确认元素是否正确、有哪些可执行动作。
Inputs: ref (required). Returns: result
find_elementA
按文本/角色/应用搜索可交互元素,返回候选列表(每个含 [ref])。用 ref 配合 click/type_text 精确操作,避免坐标估算。text 为名字子串(如『保存』『是』),role 为角色子串(如 'push button';注意 GTK 输入框的 role 是 'text' 而非 'entry'),app 限定应用名关键词。至少提供 text 或 role 之一。强烈建议带 app 或 text 限定:全桌面裸搜很慢,且遍历大型应用(浏览器/Electron)时有节点数熔断,结果可能不全。
Inputs: app, role, text, limit, +1 more. Returns: result
get_last_click_imageA
回看最近一次坐标点击的准星小图与结论(index=-1 最近,-2 上上次)。⭐ 什么时候用它:坐标点击之后界面没有预期反应(没弹窗、没跳转、没变化)时,先用它看清「刚才到底点在哪、程序算的偏差是多少、建议改点哪个坐标」,再据此修正重试;不要盲目重复点击(实测那是最容易把界面点花的做法),也不必重新截图重新估算。返回里含:落点坐标、当时程序算出的偏差与建议坐标、点后界面变化百分比。记录只保留本会话内最近 8 次坐标点击;元素级点击零坐标、不产生记录。
Inputs: index.
get_screen_layoutA
获取显示器布局:各屏幕名称、分辨率、在虚拟桌面中的偏移(x,y)、是否主屏,以及虚拟桌面总尺寸。多屏/坐标问题排查时先调它理解几何环境。返回 JSON 文本。
Inputs: no named parameters. Returns: result
get_screen_textA
读取屏幕上的文字及其坐标,返回紧凑文本列表(每行 [ref] 文字 @ (x,y) 宽x高)。灰区应用(无元素树:SWT/Java、自绘控件、游戏、远程桌面)首选感知方式——它比 screenshot 省得多:你读文字就能定位,不需要看图、也不需要从图像里估算像素位置再换算回屏幕坐标(那正是灰区操作慢和点偏的主因)。拿到结果后可直接 click(x=..,y=..),或 click(ref=N) 由服务端代点(避免抄错坐标)。scope='window'(默认)只识别当前活动窗口那块区域,23 秒;scope='screen' 整屏,文字密集时要 810 秒,非必要别用。也可传 region=[x,y,w,h] 自己指定区域。注意:坐标是快照,界面变化(切窗/弹新对话框)后请重新调用,不要复用旧 ref。⚠️ 已知限制:文字紧邻深色图标时那一行会识别失败(实测「确认删除该文件吗?」挨着问号图标时被认成乱码)。若发现某行文字明显不对,把 region 收窄到只含该文字的小块再调一次即可(可先用本次返回的该块 bbox 定位)。
Inputs: scope, region, min_conf, max_items. Returns: result
get_ui_treeA
读取桌面无障碍元素树(结构化文本),这是感知屏幕的首选方式,不要默认用 screenshot。返回紧凑文本树,每个可操作元素带 [ref] 编号,后续用 click(ref)/type_text(ref) 精确操作,无需估算坐标。scope=active_window 最省 token(默认);找特定应用用 scope=app+app=名字关键词;整桌面用 scope=desktop(大,慎用)。interactive_only=true 只列可操作元素,进一步省 token。
Inputs: app, scope, max_nodes, interactive_only. Returns: result
launch_appA
在目标 display 启动应用:isolated 模式即放进 Xephyr 沙箱(把应用放上虚拟屏的唯一正路,宿主桌面不受影响);real 模式即普通启动。command 用 shlex 解析(如 'zenity --entry --title=T'、'gedit'),返回 {pid,command,display} 的 JSON。启动后配合 wait_window 等窗口出现,再 get_ui_tree/click 操作。
Inputs: command (required), settle. Returns: result
list_windowsA
一次列出当前可见窗口:[{id,title,pid,x,y,w,h,area}],按面积降序(面积最大者通常是主窗口;未映射的孤儿窗/隐藏辅助窗已被过滤)。用于甄别同名窗、定位目标窗口 id,替代多次 shell 查询。返回 JSON 文本。
Inputs: limit. Returns: result
press_keyA
发送快捷键(全局键盘注入)。combo 形如 'ctrl+s'、'alt+F4'、'Return'、'Tab'、'ctrl+shift+t'。修饰键用 ctrl/alt/shift/super;常见别名(PageDown/Enter/Esc/方向键等)会自动归一为 xdotool keysym;键名不识别会明确报错而非静默无操作。会回报按键后的活动窗口标题(快捷键生效与否取决于焦点在谁身上)。
Inputs: combo (required). Returns: result
screenshotA
截图(灰区兜底,默认不要主动用)。仅当目标无元素树(自绘控件/游戏/视频/远程桌面)或 get_ui_tree/find_element 无法定位时才使用——它费 token。可传 region=[x,y,w,h] 裁剪感兴趣区域;max_side 控制降采样长边(默认1280,省 token)。inline=false 时不返回图像、只落盘并返回文件路径(更省 token/更快),适合只需存档或后续自行读取的场景。此时可用 save_path 指定落盘位置——已存在的文件不会被覆盖(会直接报错),需要改路径或留空让本工具自动存到临时目录(自动路径只保留最近 5 张、旧的会被回收,要长期留存必须显式传 save_path)。优先用 get_ui_tree + click(ref) 完成任务。
Inputs: inline, region, max_side, save_path.
type_textA
输入文本。若给 ref(文本框/输入区),优先用元素级 set_value(最稳,不受焦点影响);否则/失败时降级为键盘注入(xdotool type)。clear_first=true 先全选删除原内容再输入。建议:先 click(ref) 聚焦目标输入框,再 type_text。键盘注入会回报输入后的活动窗口标题——焦点被别的应用抢走时一眼可见,不必再截图核对。
Inputs: text (required), app, ref, clear_first. Returns: result
wait_windowA
等待窗口标题满足条件(server 内部轮询,单次调用完成)。title_contains 忽略大小写;window_id 指定则只盯该窗口,否则盯活动窗口;超时返回超时提示。用于等页面加载/应用启动,替代外部反复轮询。
Inputs: timeout, window_id, title_contains. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 14 tools
Every tool has a clearly distinct purpose: input (press_key, click, type_text), perception (get_ui_tree, find_element, get_screen_text, screenshot, list_windows, get_screen_layout), automation (act_sequence), waiting (wait_window), launching (launch_app), and inspection (element_info). No two tools appear to do the same thing; even perception tools are differentiated by method and use case.
Most tools follow a consistent verb_noun pattern (press_key, type_text, get_screen_text, list_windows, wait_window, launch_app, get_ui_tree, find_element). A few are single words (click, screenshot) or noun-centric (element_info), but they are still clear and not confusing. Minor deviation from a strict convention.
14 tools is well-scoped for a computer-use server. The set covers perception, input, automation, window management, and app launching without redundancy or bloat. Each tool earns its place for a comprehensive GUI automation domain.
The tool surface covers the core computer-use workflow: perceive (tree, text, screenshot, windows), interact (click, type, key), automate (sequence), wait, and launch. Minor gaps include no explicit scroll or drag-and-drop, but these can be handled via key combos or coordinates. Overall, no dead ends for typical tasks.
Maintenance
Related MCP Connectors
A cloud Linux desktop for your AI: browse, click, type, read the screen and run commands.
Cloud desktops for AI agents: create a Linux computer, see its screen, click, type and run tasks.
Real desktops for AI agents, on machines you choose: screenshot, type, click, hire more agents.
Control a real browser from AI agents: navigate, click, type, screenshot, read the page.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables AI assistants to interact with native Linux desktop applications through AT-SPI2 accessibility interfaces. Provides semantic element targeting, natural language search, and automation capabilities (clicking, typing, keyboard shortcuts) across GTK, Qt, and Electron applications.7MIT
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to control Linux/X11 desktops by providing tools for taking screenshots, clicking, typing, and managing windows via AT-SPI and xdotool.4MIT
- AlicenseAqualityBmaintenanceEnables AI agents to operate a real GNOME Wayland desktop through accessibility-tree widget actions, pointer/keyboard input, OCR, window management, and screen capture, optionally on a private headless session.3328 PyPI11Apache 2.0
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to drive Linux desktop applications through the AT-SPI2 accessibility bus, addressing UI elements by role and name to read text, set values, and invoke actions without relying on pixels or synthetic keystrokes.1MIT