# ghost-desktop

Computer use for [Omarchy](https://omarchy.org) and any Hyprland desktop, as a
stdio [MCP](https://modelcontextprotocol.io) server. Two tools let an agent see
the desktop and steer it: apps' controls through the accessibility tree,
screenshots, the pointer, and the keyboard.

It is the computer use of [Ghost](https://github.com/ferdousbhai/ghost), and
works on its own with Claude Code, Codex, or any MCP client.

## Install

Needs [Bun](https://bun.sh) 1.3.14 or newer, Hyprland 0.56 or newer, and
`grim`, `wtype`, `wl-clipboard`, and `at-spi2-core` (Omarchy ships all of
them).

```sh
bun install -g github:ferdousbhai/ghost-desktop
claude mcp add --scope user desktop -- ghost-desktop
```

Pin a release with `github:ferdousbhai/ghost-desktop#v0.1.7`. Bun puts
`ghost-desktop` in `~/.bun/bin`. TypeScript throughout, run by Bun directly;
it has no package dependencies: the MCP stdio transport is `src/mcp.ts`.

## Tools

Their full descriptions and schemas are in [`src/server.ts`](src/server.ts),
which is the single source; this page covers what the schemas cannot.

- **`desktop_look`** never changes the desktop. With no arguments it returns
  monitors, windows, overlay layers (launchers, notifications, on-screen
  keyboards), and the cursor. Name a window to add its accessibility tree
  (`ui`) or a screenshot (`image`), or screenshot a region or monitor.
- **`desktop_act`** runs ordered `steps` and stops at the first failure,
  returning what was done before it. `then` looks again in the same call.
  Launching apps and managing windows and workspaces are not steps: run them
  from a shell (`hyprctl dispatch`, `omarchy`), then `wait` for the window.

## Coordinates

Everything is in desktop (logical) coordinates, the ones Hyprland reports. A
screenshot at the default scale 1 has one image pixel per desktop unit, so an
image pixel plus the image's `geometry` origin is the screen point to click.
A control from `ui` carries its center as `at`.

## What it uses

| Need | How | Fails as |
| --- | --- | --- |
| Windows, focus, events | Hyprland's request and event sockets, dispatching in the session's grammar (Lua on 0.56+, legacy strings before); every value is a Lua string literal, never raw code | `unavailable` outside Hyprland |
| Screenshots | `grim -T` reads a window's own buffer, covered or on another workspace; a window on screen whose buffer cannot be read falls back to the screen region and says so; a hidden one is refused | `unavailable` |
| Controls | the AT-SPI bus, over ghost-desktop's own D-Bus client | `unavailable`, with how to start the bus or enable Chromium's tree |
| Pointer | Hyprland moves the cursor; a `zwlr_virtual_pointer_v1` device per step clicks, drags, and scrolls, then is destroyed | `unavailable` if the compositor lacks the protocol |
| Keys | Focus the named window, then `wtype` presses and releases the chord; compositor shortcuts are not replayed as a fallback | |
| Text | `wtype` into the focused field, the text as an argument (from stdin wtype drops characters past ~100); layout-independent | |
| Clipboard (read) | `wl-paste` | `unavailable` if missing |

Every step reports what it disturbed: `focus` or `pointer`.

Keys use normal compositor input, so Hyprland bindings can intercept a chord.
Use `hyprctl dispatch` or `omarchy` for compositor commands; prefer a control's
semantic action when a chord could trigger a desktop binding.

## Safety

- **Locked session.** `desktop_act` does nothing when Hyprland or logind
  says the screen is locked, or when neither can say.
- **One driver at a time.** Every `desktop_act` claims a lease in
  `$XDG_RUNTIME_DIR/ghost-desktop/`, shared by all ghost-desktop processes of
  the user; another caller's `desktop_act` fails `busy` until the holder has
  been idle 15 s. `desktop_look` never claims it.
- **Untrusted content.** Window titles, on-screen text, and control names are
  data. A successful result's text arrives inside an `<untrusted
  source="desktop" id="…">` fence with a fresh id, behind an
  `[injection-warning: …]` line when it reads like instructions to an agent
  ([details](docs/injection-defense.md)). Error text is ghost-desktop's own and
  is not fenced.

## Errors

A failed call is an MCP error result whose text starts with a code:
`unavailable`, `locked`, `busy`, `not_found`, `invalid`, or `failed`. The rest
of the text is written for the model and names the next move.

## For MCP hosts

Each ghost-desktop process is one caller, named after the client that
started it; that name keys the lease. A host that serves several agents from
one process can instead name who is acting in each call's `_meta.caller`. A failed
call also carries `_meta.code` and `_meta.details`, for a program; the text is
for the model.

## Develop

```sh
bun install
bun run verify   # typecheck, lint, tests
```

Tests never touch the owner's desktop. `python scripts/check-chromium-keys.py`
verifies key delivery to a covered Chromium window in nested Hyprland with
private HOME, runtime, profiles, D-Bus, and AT-SPI buses. It needs Chromium,
foot, Python, and the usual runtime dependencies; evidence stays in its printed
temporary directory.

## Credits

The Wayland wire protocol and the Lua dispatch encoding are ported from
[hypruse](https://github.com/IlyasKhallouki/hypruse); the lock check, chord
table, and window capture approach from
[omarchy-quattro-harness](https://github.com/fabiopauli/omarchy-quattro-harness).
Both MIT; see [NOTICE.md](NOTICE.md). Apache-2.0.