Control a GNOME desktop (Wayland) via screenshots, mouse/keyboard, accessibility tree, OCR, clipboard, and audio transcription, with virtual desktops and multi-client safety.
Screen & input: take screenshots, zoom into regions, click (left/right/middle/double/triple), hover, drag, scroll, type text (pastes non-ASCII), send key combos, with modifiers and coordinate/ref targeting.
Accessibility: read the window tree (refs, roles, boxes), find elements by name or OCR fallback, fill form fields (text, checkboxes, sliders) directly, list windows.
Reading: OCR the screen (tesseract), wait for text/element/screen-idle, get/set clipboard.
App control: open/switch apps via Activities search.
Audio: listen to system output or mic and transcribe locally (Whisper), transcribe audio/video files.
Batch & session: run multiple steps in one round-trip, inspect/end/resume the control session.
Safety: virtual desktops (isolated GNOME Shell), control lease (30s), kill switch, --real for your own session.
Provides desktop automation for a GNOME Shell session on Wayland, including screenshots, mouse and keyboard control, accessibility tree inspection, OCR, clipboard access, app launching, and audio monitoring.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@use-computer-mcp Take a screenshot and tell me what's on the screen"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Lets an agent (Claude, via MCP) see and operate a GNOME desktop on Wayland the way a person does: screenshots, mouse, keyboard, the accessibility tree with clickable refs, OCR, the clipboard, and listening to whatever the computer is playing. Think Claude in Chrome, but for the whole desktop instead of one browser tab.
By default it does not touch your screen. It starts a virtual desktop — a second, isolated GNOME Shell with its own session bus, accessibility bus and virtual monitor — and works there, so you can keep using your machine. It is a real desktop (your extensions, your dock, real apps); nobody is looking at it until you ask to watch:
use-computer watch # a window on your screen showing the agent's desktop
use-computer watch --tty # or draw it in this terminal
use-computer watch --control # take the mouse and keyboard yourselfAdd --real to drive your own session instead.
It ships as three things that share one daemon: an MCP server (use-computer-mcp), a CLI
(use-computer), and a Claude skill (skills/use-computer) that teaches the workflow.
Real limitations, up front:
GNOME Shell on Wayland only. It uses Mutter's private RemoteDesktop/ScreenCast D-Bus API. KDE, Sway, Hyprland and X11 sessions are not supported and nothing here pretends to be.
Single monitor (the primary one) today.
--real drives your actual session. The pointer moves; typing goes to the focused
window. Don't use the machine at the same time. The default virtual desktop has neither
problem, but it is also not your screen: an agent cannot see or fix what is on yours unless
you ask for --real.
A virtual desktop costs a GNOME Shell (~300 MB). It starts on first use and is reaped
when the daemon goes idle — unless its keyring is unlocked, in which case it stays until
vd stop.
An unlocked keyring is readable by every agent driving that desktop. That is the point
(apps come up logged in), but secret-tool in that desktop's environment reads any secret.
Accessibility coverage depends on the app. GTK/GNOME apps and Firefox are good;
Chromium/Electron apps expose little unless started with --force-renderer-accessibility;
games and canvases are pixels only.
Audio transcription is CPU-only here: about real time on long audio, ~15 s minimum per call.
Capability | How | Measured on GNOME 50 / i7-1265U |
Screenshot | Mutter ScreenCast → PipeWire → GStreamer, downscaled to 1280 px wide WebP | ~0.3 s first, ~15 KB |
Zoom | full-resolution crop of a region | ~0.06 s |
Click / double / triple / right / middle, hover, drag, scroll | Mutter RemoteDesktop absolute pointer events | verified against a GTK4 test app's own event log |
Key combos | X keysyms ( | verified, incl. |
Typing any text | ASCII as keys; everything else pasted via Mutter's clipboard, old clipboard restored |
|
Typed-text verification | reads the focused field back through AT-SPI |
|
Accessibility tree + refs | AT-SPI; refs clickable via the element's own action | 10–20 ms for a small window |
Screen positions for refs | window-relative extents + window origin (see below) | exact for X11 apps; matched for Wayland apps |
| EditableText / Value / toggle action, mouse fallback | |
OCR | tesseract, 2× upscale, sparse-text mode | ~0.7 s full 1080p screen |
Wait for text / element / idle screen | OCR polling, AT-SPI polling, damage-driven frame timing | |
Open app | Activities search (Super, type, Enter) | |
Clipboard get/set | Mutter selection API | round-trips Unicode |
Listen / transcribe |
| 62 s speech in 62 s, 2.1 GB RAM |
Kill switch | stopping screen sharing from the top bar revokes control until resumed | |
Multi-agent safety | one daemon, a 30 s control lease per client | |
Virtual desktops | isolated headless GNOME Shell per name; own D-Bus, a11y bus and runtime dir | ~4 s to start, GPU-accelerated where a DRM device exists |
Watching one | window (GTK4) or terminal (chafa: kitty/sixel/symbols) | symbols mode ≈ 1.7 KB/frame, so it works over SSH |
Taking over |
| |
Headless hosts | works on a box with no desktop session at all | verified on an Arch server at |
Real keyring in a virtual desktop | locked gnome-keyring-daemon on the desktop's bus; | Chrome in the VD opened Gmail logged in; wrong password leaves |
Logins without the agent seeing them |
| refuses unless an app's password field has focus |
Related MCP server: astra-linux-agent
Multi-monitor support.
Sound inside a virtual desktop: it shares the host's PipeWire, so listen hears the real
machine's output, not the virtual desktop's apps.
Window management beyond "open/switch app via search" (no move/resize/list-geometry API —
GNOME blocks Shell.Introspect for outside callers).
A way to see the mouse cursor in screenshots (it's hidden on purpose).
Non-GNOME compositors.
HTTPS for the secret page (it relies on Tailscale's encryption; --host on a plain LAN
sends the secret in clear).
Unlocking keyrings other than login.
Any testing of your real session and a virtual desktop both holding login.keyring open.
On this box no real session runs, so it never came up; elsewhere, prefer one at a time.
With ownbox:
ownbox install use-computerSetup (ownbox.sh setup) does the following, and ownbox uninstall undoes the non-package parts:
Installs missing system packages with pacman (asks sudo): python-gobject,
gst-plugin-pipewire, at-spi2-core, wl-clipboard, tesseract, tesseract-data-eng,
tesseract-data-osd. More OCR languages: USE_COMPUTER_TESSERACT_LANGS="eng osd srp srp_latn".
Creates .venv with system site-packages (PyGObject comes from the distro).
Sets org.gnome.desktop.interface toolkit-accessibility true (restored on uninstall if it
was false). Apps started after this expose their accessibility tree.
Registers the MCP server in ~/.claude.json (user scope, the same entry
claude mcp add --scope user would write). Start a new Claude session afterwards.
Runs use-computer doctor.
Audio is opt-in because the model is 1.6 GB:
use-computer audio setupManual install:
git clone https://github.com/BogdanStamenovic/use-computer && cd use-computer
./ownbox.sh setupFrom Claude: just ask for something that needs the desktop; the skill loads and the
use-computer MCP tools do the work.
From a shell:
use-computer doctor
use-computer screenshot # prints a PNG path (of the virtual desktop)
use-computer --real screenshot # ...of your actual screen
use-computer vd list # virtual desktops
use-computer watch # watch the agent work
use-computer tree # accessibility tree of the focused window
use-computer find "Save"
use-computer click --ref ref_12
use-computer click --at 640,360 # screenshot pixels; add --screen for real pixels
use-computer type "Zdravo, svete"
use-computer key "ctrl+s"
use-computer ocr --region 0,0,640,360
use-computer audio listen 20 # what the computer plays, for 20 s
use-computer stop # end the session, indicator disappearsCommand | What it does |
| capture to a file |
|
|
| pointer |
| keyboard |
| accessibility |
| reading and waiting |
| desktop |
| JSON list of |
| hearing |
| virtual desktops |
| see one |
| the real login keyring inside a virtual desktop |
| the user types a password/2FA code into the focused field |
| plumbing |
--desktop NAME picks which virtual desktop to drive (default agent); --real drives your
own session.
--json prints raw results. Exit codes: 0 ok, 1 failed, 2 usage error, 3 control revoked by the user.
A virtual desktop has its own session bus, so it cannot see any other session's keyring,
and Chrome on Linux keeps the key to its cookie store there ("Chrome Safe Storage"): without
it every site looks logged out. Each virtual desktop therefore runs a gnome-keyring-daemon
over your real ~/.local/share/keyrings, starting locked.
use-computer keyring unlock --qr # prints a URL (+ QR); open it on your phone, type the password
use-computer keyring status # {"running": true, "locked": false}
use-computer handoff --prompt "GitHub password for BogdanStamenovic" # agent focused the field firstBoth commands serve the same page (secretpage.py):
Property | Value |
Bound to | the tailnet IPv4 ( |
Access | 192-bit random token in the path; anything else is a bare 404 |
Lifetime | one success, 5 failed tries, or |
Transport | plain HTTP. Tailnet traffic is WireGuard-encrypted; over a LAN it would not be |
Where the secret goes | keyring: gnome-keyring's |
Odd-looking choices, explained:
Unlocking uses an "InternalUnsupportedGuiltRidden" D-Bus interface (what Seahorse uses).
gnome-keyring-daemon --start --unlock is refused by the daemon, and --replace --unlock
starts a new daemon outside the desktop's process group, which would outlive vd stop.
keyring unlock refuses when login.keyring does not exist, because unlocking a missing
keyring makes gnome-keyring create a new one with whatever was typed.
Secret typing checks focus first. A test handoff on a fresh desktop typed the "password"
into the overview's search bar in plain text. type(secret=True) now refuses unless an
app's password text element (or a text field, with --visible) has focus, wipes the
clipboard if a non-ASCII paste would leave the secret there, and reports nothing about what
was typed.
handoff runs under the caller's control lease. A shell-run handoff from an MCP agent would otherwise be locked out by that agent's own 30 s lease; the MCP tool runs the page inside the MCP server for that reason.
Claude ──stdio──► use-computer-mcp ─┐
Bash ─────────► use-computer (CLI)├─ unix socket ─► daemon (one per D-Bus session)
┘ │
┌─────────────────────────────────────────────┼──────────────────────────┐
│ Mutter RemoteDesktop session: pointer, keysyms, clipboard │
│ Mutter ScreenCast stream ─► PipeWire ─► GStreamer appsink (latest frame)│
│ AT-SPI (accessibility bus): tree, refs, actions, text │
│ tesseract (subprocess) │
└────────────────────────────────────────────────────────────────────────┘
listen/transcribe run in the calling process (pw-record + faster-whisper), not the daemon.Why a daemon. A Mutter session dies with the D-Bus connection that created it, CLI calls are separate processes, and several Claude sessions must not fight over one pointer. The daemon is single-threaded (it pumps GLib between requests), holds the session and the newest frame, stops the session after 90 s idle (the sharing indicator disappears) and exits after 30 min.
Why Mutter's API and not the portal. No consent dialog, no restore tokens, absolute coordinates. The cost is GNOME-only.
Why typing pastes non-ASCII. Mutter turns a keysym into a key press only if the active
layout has it, and drops it silently otherwise. Switching layouts from a remote session is
refused unless the session installed the keymap itself. Pasting is the only path that works
for all text; the clipboard is restored afterwards, and handed to wl-copy when the session
ends so it isn't lost.
Why screen positions need a trick. Wayland apps don't know where their window is, so
AT-SPI gives window-relative boxes and bogus screen boxes. GNOME Shell's own accessibility tree
does expose every window actor's screen rectangle, but that includes client-side shadows of
unknown per-side size. locate.py finds the content rectangle inside the actor by scoring every
possible offset on edge strength along its perimeter (shadows are smooth, window borders are
sharp). X11 apps report real screen coordinates and skip this.
Screenshots settle. The screencast is damage-driven, so "no new frame for 120 ms" means the screen stopped changing; screenshots wait for that (max 1 s).
.venv/bin/pytest -q # unit tests, no desktop neededEnd-to-end tests run against a separate headless GNOME Shell so they never touch your desktop:
tests/e2e/headless-shell.sh /tmp/uc.env & # isolated shell, bus, runtime dir
. /tmp/uc.env
python tests/e2e/harness.py /tmp/h.log & # GTK4 app that logs every event it receives
python tests/e2e/mcp_smoke.py /tmp # drives the MCP server over stdio
tests/e2e/headless-stop.shRead the comments in headless-shell.sh before changing it. Running a second shell with the
real XDG_RUNTIME_DIR clobbers the desktop's accessibility bus socket and document portal
mount (this happened; see docs/development-log.md). The headless shell also doesn't register
itself with AT-SPI, so window-position resolution can only be tested on a real session.
MIT
Read or replace the clipboard text.
Inputs: action (required), text.
Mouse, keyboard and screen on the user's GNOME desktop.
Coordinates are pixels in the most recent screenshot (it is downscaled, e.g. 1280x720 for a 1920x1080 screen; the tool maps them back). A screenshot waits briefly for the screen to stop changing. Text outside ASCII is pasted via the clipboard, which is restored afterwards. Actions other than screenshot/zoom return a short confirmation, not an image: take a screenshot when you need to see the outcome.
Inputs: action (required), ref, text, mouse, +8 more.
Run several steps in one round trip, e.g. click a field, type, press Return, wait, screenshot. Returns every step's text and any images, in order.
Inputs: actions (required).
Find elements by name. Returns refs with boxes (accessibility) or, if nothing accessible matches, OCR text boxes with a click_at point in screenshot pixels.
Inputs: query (required), ocr, scope.
Set a form control's value directly through accessibility, without typing.
Inputs: ref (required), value (required).
Record audio for a fixed time and transcribe it with Whisper (local, CPU). The recording is deleted afterwards. Start playback before calling. Transcription takes about as long as the audio (minimum ~15 s, Whisper works in 30-second windows).
Inputs: seconds (required), source, language.
Read text from the screen with tesseract: lines with confidence and boxes in screenshot pixels. Use when accessibility has nothing (images, canvases, terminals, remote desktops). ~0.7 s for a full 1080p screen.
Inputs: lang, region.
Open or switch to an app through the Activities search (Super, type, Enter). Confirm with a screenshot afterwards.
Inputs: name (required).
Accessibility tree of windows: one line per element with [ref], role, name, @(x,y,w,h) box in screenshot pixels, value, states and actions.
Prefer refs over pixel guessing. Apps that do not expose accessibility (games, canvases, some Electron apps) show little or nothing; use screenshots and ocr there. 'positions=unresolved' means refs still work for actions but boxes are unknown.
Inputs: scope, filter, ref_id, max_chars.
Inspect or end the control session.
Inputs: action (required).
Transcribe an audio/video file with Whisper (local, CPU).
Inputs: path (required), language.
Wait for something instead of sleeping. Returns met=true/false.
Inputs: gone, name, role, text, +2 more.
List open windows (app, title, active, ref) from accessibility.
Inputs: positions.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 13 tools
Most tools have distinct roles: read_screen, find, and ocr separate accessibility-tree reading from element lookup from text extraction, while listen and transcribe split recording from file transcription. The main ambiguity is between computer and computer_batch, but the batch tool is clearly described as a multi-step version of the same action set.
Names are readable and consistently lowercase snake_case, but conventions are mixed: some are imperative verb phrases (read_screen, open_app, wait_for, form_input) while others are bare nouns (computer, ocr, windows, clipboard, session) or awkward compounds (computer_batch). This is mildly inconsistent but not chaotic.
13 tools is well-scoped for a desktop-control server; input, screen perception, accessibility, OCR, window management, waiting, clipboard, batch execution, audio transcription, and session control each earn a place. There is no obvious bloat or redundancy at the tool-count level.
The surface covers core desktop-automation workflows well: mouse/keyboard input, screenshots, accessibility trees, OCR, app/window management, form filling, waiting, clipboard, and session control. Minor gaps exist, such as no dedicated audio-output tool and relying on computer for primitive actions like screenshot or typing, but these are workable.
A cloud Linux desktop for your AI: browse, click, type, read the screen and run commands.
Operate Linux, macOS and Windows from your LLM. Every action runs through an auditable allowlist.
Cloud desktops for AI agents: create a Linux computer, see its screen, click, type and run tasks.
Control a real browser from AI agents: navigate, click, type, screenshot, read the page.
