# use-computer

Lets an agent (Claude, via MCP) see and operate a GNOME desktop on Wayland the way a person
does: screenshots, mouse, keyboard, the accessibility tree with clickable refs, OCR, the
clipboard, and listening to whatever the computer is playing. Think Claude in Chrome, but for
the whole desktop instead of one browser tab.

By default it does **not** touch your screen. It starts a *virtual desktop* — a second,
isolated GNOME Shell with its own session bus, accessibility bus and virtual monitor — and
works there, so you can keep using your machine. It is a real desktop (your extensions, your
dock, real apps); nobody is looking at it until you ask to watch:

```bash
use-computer watch              # a window on your screen showing the agent's desktop
use-computer watch --tty        # or draw it in this terminal
use-computer watch --control    # take the mouse and keyboard yourself
```

Add `--real` to drive your own session instead.

It ships as three things that share one daemon: an MCP server (`use-computer-mcp`), a CLI
(`use-computer`), and a Claude skill (`skills/use-computer`) that teaches the workflow.

**Real limitations, up front:**

- **GNOME Shell on Wayland only.** It uses Mutter's private RemoteDesktop/ScreenCast D-Bus
  API. KDE, Sway, Hyprland and X11 sessions are not supported and nothing here pretends to be.
- **Single monitor** (the primary one) today.
- **`--real` drives your actual session.** The pointer moves; typing goes to the focused
  window. Don't use the machine at the same time. The default virtual desktop has neither
  problem, but it is also not your screen: an agent cannot see or fix what is on yours unless
  you ask for `--real`.
- **A virtual desktop costs a GNOME Shell** (~300 MB). It starts on first use and is reaped
  when the daemon goes idle.
- Accessibility coverage depends on the app. GTK/GNOME apps and Firefox are good;
  Chromium/Electron apps expose little unless started with `--force-renderer-accessibility`;
  games and canvases are pixels only.
- Audio transcription is CPU-only here: about real time on long audio, ~15 s minimum per call.

## What works today

| Capability | How | Measured on GNOME 50 / i7-1265U |
|---|---|---|
| Screenshot | Mutter ScreenCast → PipeWire → GStreamer, downscaled to 1280 px wide WebP | ~0.3 s first, ~15 KB |
| Zoom | full-resolution crop of a region | ~0.06 s |
| Click / double / triple / right / middle, hover, drag, scroll | Mutter RemoteDesktop absolute pointer events | verified against a GTK4 test app's own event log |
| Key combos | X keysyms (`ctrl+shift+t`, `Tab Return`, `F5`) | verified, incl. `shift+Tab` → `ISO_Left_Tab` |
| Typing any text | ASCII as keys; everything else pasted via Mutter's clipboard, old clipboard restored | `Hi šđčćž Ћирилица + done` arrives intact |
| Typed-text verification | reads the focused field back through AT-SPI | `verified: true/false/null` |
| Accessibility tree + refs | AT-SPI; refs clickable via the element's own action | 10–20 ms for a small window |
| Screen positions for refs | window-relative extents + window origin (see below) | exact for X11 apps; matched for Wayland apps |
| `form_input` | EditableText / Value / toggle action, mouse fallback | |
| OCR | tesseract, 2× upscale, sparse-text mode | ~0.7 s full 1080p screen |
| Wait for text / element / idle screen | OCR polling, AT-SPI polling, damage-driven frame timing | |
| Open app | Activities search (Super, type, Enter) | |
| Clipboard get/set | Mutter selection API | round-trips Unicode |
| Listen / transcribe | `pw-record` of the default output's monitor + faster-whisper large-v3-turbo int8 | 62 s speech in 62 s, 2.1 GB RAM |
| Kill switch | stopping screen sharing from the top bar revokes control until resumed | |
| Multi-agent safety | one daemon, a 30 s control lease per client | |
| Virtual desktops | isolated headless GNOME Shell per name; own D-Bus, a11y bus and runtime dir | ~4 s to start, GPU-accelerated where a DRM device exists |
| Watching one | window (GTK4) or terminal (chafa: kitty/sixel/symbols) | symbols mode ≈ 1.7 KB/frame, so it works over SSH |
| Taking over | `watch --control` forwards your mouse, keys and scroll into it | |
| Headless hosts | works on a box with no desktop session at all | verified on an Arch server at `multi-user.target` |

## What does not exist yet

- Multi-monitor support.
- Sound *inside* a virtual desktop: it shares the host's PipeWire, so `listen` hears the real
  machine's output, not the virtual desktop's apps.
- Window management beyond "open/switch app via search" (no move/resize/list-geometry API —
  GNOME blocks `Shell.Introspect` for outside callers).
- A way to see the mouse cursor in screenshots (it's hidden on purpose).
- Non-GNOME compositors.

## Install

With [ownbox](https://github.com/BogdanStamenovic/ownbox):

```bash
ownbox install use-computer
```

Setup (`ownbox.sh setup`) does the following, and `ownbox uninstall` undoes the non-package parts:

1. Installs missing system packages with pacman (asks sudo): `python-gobject`,
   `gst-plugin-pipewire`, `at-spi2-core`, `wl-clipboard`, `tesseract`, `tesseract-data-eng`,
   `tesseract-data-osd`. More OCR languages: `USE_COMPUTER_TESSERACT_LANGS="eng osd srp srp_latn"`.
2. Creates `.venv` **with system site-packages** (PyGObject comes from the distro).
3. Sets `org.gnome.desktop.interface toolkit-accessibility true` (restored on uninstall if it
   was false). Apps started *after* this expose their accessibility tree.
4. Registers the MCP server in `~/.claude.json` (user scope, the same entry
   `claude mcp add --scope user` would write). Start a new Claude session afterwards.
5. Runs `use-computer doctor`.

Audio is opt-in because the model is 1.6 GB:

```bash
use-computer audio setup
```

Manual install:

```bash
git clone https://github.com/BogdanStamenovic/use-computer && cd use-computer
./ownbox.sh setup
```

## Usage

From Claude: just ask for something that needs the desktop; the skill loads and the
`use-computer` MCP tools do the work.

From a shell:

```bash
use-computer doctor
use-computer screenshot                 # prints a PNG path (of the virtual desktop)
use-computer --real screenshot          # ...of your actual screen
use-computer vd list                    # virtual desktops
use-computer watch                      # watch the agent work
use-computer tree                       # accessibility tree of the focused window
use-computer find "Save"
use-computer click --ref ref_12
use-computer click --at 640,360         # screenshot pixels; add --screen for real pixels
use-computer type "Zdravo, svete"
use-computer key "ctrl+s"
use-computer ocr --region 0,0,640,360
use-computer audio listen 20            # what the computer plays, for 20 s
use-computer stop                       # end the session, indicator disappears
```

| Command | What it does |
|---|---|
| `screenshot`, `zoom X0,Y0,X1,Y1` | capture to a file |
| `click`, `double-click`, `triple-click`, `right-click`, `middle-click` | `--at X,Y` or `--ref`; `--modifiers ctrl`; `--mouse` forces a real click on refs |
| `move`, `drag FROM TO`, `scroll [--at] DIR [N]` | pointer |
| `key KEYS [--repeat N]`, `type TEXT [--ref]` | keyboard |
| `tree [--scope all\|APP] [--all] [--ref]`, `find QUERY`, `form-input REF VALUE`, `action REF NAME` | accessibility |
| `ocr [--region] [--lang]`, `wait-for --text/--name/--idle [--gone]` | reading and waiting |
| `windows`, `open-app NAME`, `clipboard get\|set` | desktop |
| `batch FILE\|-` | JSON list of `{"op", "args"}` |
| `audio setup\|status\|listen SECONDS\|transcribe PATH` | hearing |
| `vd start [NAME] [--size WxH]`, `vd stop [NAME\|--all]`, `vd list` | virtual desktops |
| `watch [NAME] [--tty] [--control] [--fullscreen] [--fps N] [--format kitty\|sixels\|symbols]` | see one |
| `status`, `stop`, `resume`, `shutdown`, `daemon`, `doctor`, `mcp-register`, `mcp-unregister` | plumbing |

`--desktop NAME` picks which virtual desktop to drive (default `agent`); `--real` drives your
own session.

`--json` prints raw results. Exit codes: 0 ok, 1 failed, 2 usage error, 3 control revoked by the user.

## How it works

```
Claude ──stdio──► use-computer-mcp ─┐
Bash   ─────────► use-computer (CLI)├─ unix socket ─► daemon (one per D-Bus session)
                                    ┘                   │
          ┌─────────────────────────────────────────────┼──────────────────────────┐
          │ Mutter RemoteDesktop session: pointer, keysyms, clipboard              │
          │ Mutter ScreenCast stream ─► PipeWire ─► GStreamer appsink (latest frame)│
          │ AT-SPI (accessibility bus): tree, refs, actions, text                  │
          │ tesseract (subprocess)                                                 │
          └────────────────────────────────────────────────────────────────────────┘
listen/transcribe run in the calling process (pw-record + faster-whisper), not the daemon.
```

**Why a daemon.** A Mutter session dies with the D-Bus connection that created it, CLI calls
are separate processes, and several Claude sessions must not fight over one pointer. The daemon
is single-threaded (it pumps GLib between requests), holds the session and the newest frame,
stops the session after 90 s idle (the sharing indicator disappears) and exits after 30 min.

**Why Mutter's API and not the portal.** No consent dialog, no restore tokens, absolute
coordinates. The cost is GNOME-only.

**Why typing pastes non-ASCII.** Mutter turns a keysym into a key press only if the active
layout has it, and drops it silently otherwise. Switching layouts from a remote session is
refused unless the session installed the keymap itself. Pasting is the only path that works
for all text; the clipboard is restored afterwards, and handed to `wl-copy` when the session
ends so it isn't lost.

**Why screen positions need a trick.** Wayland apps don't know where their window is, so
AT-SPI gives window-relative boxes and bogus screen boxes. GNOME Shell's own accessibility tree
does expose every window actor's screen rectangle, but that includes client-side shadows of
unknown per-side size. `locate.py` finds the content rectangle inside the actor by scoring every
possible offset on edge strength along its perimeter (shadows are smooth, window borders are
sharp). X11 apps report real screen coordinates and skip this.

**Screenshots settle.** The screencast is damage-driven, so "no new frame for 120 ms" means the
screen stopped changing; screenshots wait for that (max 1 s).

## Development

```bash
.venv/bin/pytest -q                    # unit tests, no desktop needed
```

End-to-end tests run against a **separate headless GNOME Shell** so they never touch your
desktop:

```bash
tests/e2e/headless-shell.sh /tmp/uc.env &      # isolated shell, bus, runtime dir
. /tmp/uc.env
python tests/e2e/harness.py /tmp/h.log &       # GTK4 app that logs every event it receives
python tests/e2e/mcp_smoke.py /tmp             # drives the MCP server over stdio
tests/e2e/headless-stop.sh
```

Read the comments in `headless-shell.sh` before changing it. Running a second shell with the
real `XDG_RUNTIME_DIR` clobbers the desktop's accessibility bus socket and document portal
mount (this happened; see `docs/development-log.md`). The headless shell also doesn't register
itself with AT-SPI, so window-position resolution can only be tested on a real session.

## License

MIT