hypr-computer

It sees the screen, moves and clicks the mouse, types on a real keyboard layout, swipes on a touchpad, manages windows, reads text with OCR and operates widgets through the accessibility tree.

By default every agent works on its own desktop: a nested Hyprland with its own cursor, focus, clipboard and apps. You watch it live with Super+N and keep working undisturbed on yours.

• Built for Hyprland 0.56+ with a Lua config

• Works with the Phantomat infinite-canvas plugin (canvas coordinates are mapped to the screen)

• Pairs with agentControl, which streams each agent's desktop next to its Claude Code terminal

Quick start

1. System packages. PyGObject, evdev and Pillow come from the system Python, which the venv shares.

You also need Python 3.12+ and uv.

2. Install and register with Claude Code.

git clone https://github.com/zinoos/hypr-computer.git ~/projects/hypr-computer
cd ~/projects/hypr-computer
uv venv --python /usr/bin/python3 --system-site-packages .venv
uv pip install --python .venv/bin/python -e .
claude mcp add --scope user hypr-computer -- "$PWD/.venv/bin/python" -m hypr_computer

Other MCP clients: run <checkout>/.venv/bin/python -m hypr_computer as a stdio server.

3. Try it. Start a new Claude Code session and ask for something visual:

Open Firefox on your desktop, search for "Hyprland wiki" and tell me the first heading.

On first use an agent desktop starts on place 4 (or the next free one). Press Super+4 to watch.

Optional: tidy viewers. With contrib/agent-places.lua each viewer fills its place without splitting the one you're on or taking focus (recommended with Phantomat):

cp contrib/agent-places.lua ~/.config/hypr/
echo 'dofile(os.getenv("HOME") .. "/.config/hypr/agent-places.lua")' >> ~/.config/hypr/hyprland.lua
hyprctl reload

Related MCP server: hyprcu

What it looks like

your Hyprland (one seat: your mouse + keyboard)
 ├─ places 1-3: yours (Super+1..3)
 ├─ place 4: viewer window  ◄── mirrors ──  agent desktop "desktop-4" (nested Hyprland)
 │                                             ├─ headless output AGENT-1 (the agent's screen)
 │                                             ├─ virtual pointer + keyboard (Wayland protocols, its own seat)
 │                                             └─ private D-Bus + AT-SPI, own clipboard, own apps
 └─ place 5…0: more agents

The agent desktop renders to a headless output, so it keeps working whether or not anyone watches, and nothing it does touches your pointer, focus or windows.

• Places 4-9 and 0 belong to agents, 1-3 stay yours. Your own windows on 4-0 aren't touched, but each agent's viewer opens there. Pick a place with agent_desktop(action="start", place=N).

• Viewers only show. The agent's cursor is software-rendered, so you see it 1:1. Your input never reaches the agent desktop.

• Viewer modes: full place (default), pip (small, pinned over everything), window, tile, hidden.

• Closing the viewer stops that agent desktop (a small watchdog notices).

• Browsers get their own profile under ~/.local/share/hypr-computer/agents/<name>/, otherwise they'd open inside your running instance. Your logins aren't there.

Tools

Cheapest first. Reading the UI as text (a11y_tree + a11y_act, ocr) is exact and costs no image tokens. wait(until_text=…) beats fixed sleeps. Action tools take screenshot_after=true for a small check image (long edge 900 px); call screenshot when you need detail.

Targets and coordinates

Every screenshot/zoom returns a frame id (f1, f2, …). Unless you say otherwise, x/y refer to the pixels of the latest frame, so the image the model sees is the space it clicks in. Images stay under Claude's resize limits (1568 px long edge, 1.15 MP) so they aren't rescaled again on the way in. Instead of pixels you can also pass:

Your real desktop

agent_desktop(action="use_host") switches the tools to your real desktop, read-only: screenshots, OCR, window list. HYPR_COMPUTER_TARGET=host makes that the default.

Input on your real desktop (shared mouse and keyboard via uinput) is off unless you set HYPR_COMPUTER_ALLOW_HOST_INPUT=1, and it is refused while the screen is locked. It needs write access to /dev/uinput, for example:

echo 'KERNEL=="uinput", GROUP="input", MODE="0660", OPTIONS+="static_node=uinput"' | sudo tee /etc/udev/rules.d/60-uinput.rules
sudo usermod -aG input "$USER"   # then log out and back in
sudo udevadm control --reload && sudo udevadm trigger

Configuration

Environment variables for the server process:

With Claude Code, pass them on registration, e.g. claude mcp add --scope user -e HYPR_COMPUTER_OCR_LANGS=eng+deu hypr-computer -- ….

Troubleshooting

How input works

On an agent desktop: a pure-Python Wayland client creates zwlr_virtual_pointer_v1 (absolute motion, buttons, wheel) and zwp_virtual_keyboard_v1 (keymap uploaded from libxkbcommon, modifiers tracked with an xkb state) on the nested compositor. No touchpad gestures there.

On the host (use_host, only with HYPR_COMPUTER_ALLOW_HOST_INPUT=1):

• Three virtual devices on /dev/uinput: a keyboard, a pointer and a touchpad. They are created on first use and live as long as the server. Hyprland refocuses when a device is plugged in, so the server puts focus back afterwards.

• Keys are real evdev keycodes. The text→key map comes from libxkbcommon using the layout Hyprland reports (e.g. German: z = KEY_Y, @ = AltGr+Q). Characters the layout can't produce (emoji, dead-key-only characters like ~) go through wtype.

• The pointer moves with real relative motion, so hover and drag targets see the path. Hyprland sets this device to flat acceleration. A final hl.dsp.cursor.move warp fixes any leftover error, e.g. from gaps between monitors.

release_all runs on every error and at exit, so modifier keys can't get stuck.

Development

src/hypr_computer/
  server.py      MCP server: all tools and the instructions sent to the model
  agentdesk.py   agent desktops: nested Hyprland, places, viewer, watchdog
  viewer.py      the watch-only viewer window
  wlvirt.py      Wayland virtual pointer/keyboard client (agent desktops)
  agentinput.py  input on agent desktops
  vinput.py      uinput keyboard/pointer/touchpad (host)
  xkb.py         text → keys via libxkbcommon
  screen.py      screenshots, frames, zoom, OCR
  a11y.py        accessibility tree (AT-SPI)
  hypr.py        Hyprland IPC (Lua eval, dispatch, queries, Phantomat canvas projection)
  target.py      which desktop the tools act on
contrib/agent-places.lua   optional Hyprland helper for viewer placement

Tests talk to the server over stdio; images go to $OUT:

OUT=/tmp/hc .venv/bin/python tests/agent_live.py   # full input test on a fresh agent desktop, safe while you work
OUT=/tmp/hc .venv/bin/python tests/smoke.py        # read-only tools
OUT=/tmp/hc .venv/bin/python tests/live.py         # same as agent_live on your REAL desktop: injects input, run only when you're away

License

MIT