handy

Real-time native mouse, keyboard, window, clipboard, and coordinate-mapped screen control for LLMs and agents.

Handy lets LLMs see your computer screen (or a single desktop window) in real-time with dual-contrast mapped (x, y) coordinate rulers, active window telemetry, pixel hex under the cursor, and sub-2ms frame-to-frame visual diffs — and control the mouse, keyboard, clipboard, and desktop windows natively with chained multi-step actions (focus_window -> click -> paste -> wait_for_settled in a single call). It installs cleanly on any PC with zero C++ addon compilation (node-gyp free) and ships prepackaged as a stdio MCP server and agent plugin with operational skills, a local SSE-powered live web Control Panel (handy ui), a physical (0, 0) corner-slam failsafe, and an instant on/off permission toggle.

Highlights

• Set-of-Mark Vision + Click-by-Mark (handy_see marks: true): Instead of guessing pixel coordinates from a screenshot, Handy detects and numbers the interactive elements on screen — control (buttons, fields, checkboxes), text, icon, and region — and outlines each with a small numbered badge. Every pointer step (move, click, double_click, triple_click, right_click, drag, scroll) then accepts { "mark": 5 } in place of { "x": …, "y": … }, and the text chain DSL understands click mark 5 / click #5. Marks survive layout shift and HiDPI scaling because they resolve to the element's own centre at execution time, and they work identically in the CLI (handy see --marks) and the Control Panel, where clicking an element on the screenshot inserts its mark reference instead of brittle coordinates.

• Human-Like Pointer Motion (motion: "human"): Pointer steps can travel along a Fitts's-law-paced Bézier arc rather than a teleporting straight line, with the duration derived automatically from travel distance and target size when omitted. Useful where synthetic instant motion is filtered or where a visible cursor path helps a human follow along. motion: "linear" and "ease" are also available.

• Chained Real-Time Controls (handy_act): Execute up to 100 sequential steps (focus_window, move, click, double_click, triple_click, right_click, drag, scroll, paste, type, key, mouse_down/up, key_down/up, wait, wait_for_change, wait_for_settled, wait_for_color, screenshot) in one atomic call, supporting global screen, normalized (0..1000), and window-relative coordinates, plus screenshotAfter: true with numbered amber action-trail markers (#1, #2, drag vectors).

• Coordinate-Mapped Vision + Window Zoom + Smart Diffing (handy_see): Renders exact OS screen coordinate rulers (dual-contrast so lines are legible on both #000 dark and #fff light apps), (x, y) intersection pills, and a high-contrast MOUSE (x,y) reticle — plus pixelUnderMouse (#rrggbb), activeWindow, screenDiff (changedPercent, changedBounds), and highlightDiff: true (CHANGED outline). Supports window or region zoom at 1:1 resolution while preserving global screen coordinates. Downscales with area-average box filtering (so 1px text survives instead of being sampled away) and encodes with per-row filter selection, producing captures that are ~26% smaller and noticeably sharper than nearest-neighbour bilinear output.

• Window & Clipboard Awareness (handy_windows, handy_clipboard): Inspect open desktop windows with { x, y, width, height } bounds, bring target apps to the foreground cleanly, and paste multi-line Unicode text instantaneously while preserving the user's prior clipboard.

• Instant On/Off Toggle + (0, 0) Failsafe + Live SSE UI: Toggle LLM access live at any time via CLI (handy off / handy on / handy toggle), plugin slash commands (/handy:off / /handy:on / /handy:toggle / /handy:status / /handy:doctor / /handy:windows), --view-only mode, --ttl timers, or the loopback web Control Panel (handy ui with live Server-Sent Events push) — or slam your physical mouse to the top-left (0, 0) corner to abort an active chain immediately.

• Works on Any PC Out of the Box:

  • Windows 10/11: Persistent Win32 SendInput (KEYEVENTF_UNICODE, MOUSEEVENTF_VIRTUALDESK), EnumWindows/SetForegroundWindow, MTA-safe Win32 OpenClipboard, and GDI CopyFromScreen with Per-Monitor V2 DPI awareness and GetDpiForMonitor scale telemetry.

  • macOS (Apple Silicon & Intel): Persistent CoreGraphics CGEvent + NSWorkspace + NSPasteboard + in-memory CGWindowListCreateImage PNG capture (with screencapture fallback) and Retina 2x logical-point mapping.

  • Linux (X11 & Wayland): Pure native X11 wire-protocol client (XTEST + GetImage + BIG-REQUESTS + RANDR + EWMH _NET_CLIENT_LIST/_NET_ACTIVE_WINDOW, zero external packages required) plus Wayland/CLI support (grim, maim, ydotool, wtype, xdotool, wl-clipboard, xclip, xsel).

  • Virtual Desktop (HANDY_BACKEND=virtual): Built-in 1280x800 multi-window interactive desktop workbench for headless CI and safe agent sandboxing.

Related MCP server: Open Computer Use

Install on Any PC

Requires Node.js 20+.

1. One-Command Auto-Setup (Claude Desktop, Cursor, Windsurf, Claude Code)

npx -y @miisodev/handy install

Preview changes first with --dry-run, target a single host (handy install cursor), or remove cleanly (handy uninstall). Backups (*.handy.bak) are created automatically before modifying existing config files.

2. Prepackaged Plugin (Claude Code / Agent Marketplaces)

Installs the stdio MCP server, the handy operational skill (skills/handy/SKILL.md), and the /handy:on, /handy:off, /handy:toggle, /handy:status, /handy:doctor, and /handy:windows commands together:

claude plugin marketplace add miisodev/handy
claude plugin install handy

3. Global CLI + Live Control Panel

npm install -g @miisodev/handy
handy doctor
handy ui                  # Launch local Control Panel at http://127.0.0.1:7788

4. Manual Stdio MCP Configuration

{
  "mcpServers": {
    "handy": {
      "command": "npx",
      "args": ["-y", "@miisodev/handy", "mcp"]
    }
  }
}

Permission Controls & Human Failsafe

Handy stores its permission toggle at ~/.handy/permissions.json (0600) and checks it synchronously before every handy_see screen capture, handy_windows/handy_clipboard call, and before every individual step of a handy_act chain.

handy off                 # Immediately block all LLM screen viewing & input control
handy on                  # Enable full screen vision & mouse/keyboard control
handy toggle              # Flip between ON and OFF
handy on --view-only      # Allow screen viewing (handy_see), block mouse & keyboard
handy on --ttl 30m        # Auto-disable after 30 minutes
handy ui                  # Open live browser switchboard + coordinate inspector
handy status              # Check current toggle state, backend, windows & displays

• Physical Failsafe: Slamming your mouse cursor into the top-left (0, 0) corner aborts any in-flight chain and releases held keys/buttons immediately.

• Prompt-Injection Guard: When turned OFF by the user, an MCP client cannot re-enable Handy on its own, while calling handy_permission({ "action": "off" }) is always permitted as a one-way stop switch.

MCP Tools

Example: Chained Window Focus + Instant Paste + Smart Visual Settle (handy_act)

{
  "actions": [
    { "type": "focus_window", "title": "Workbench" },
    { "type": "click", "x": 250, "y": 172 },
    { "type": "paste", "text": "Sam — ready for review 🚀", "submit": true },
    { "type": "wait_for_settled", "quietMs": 200 }
  ],
  "screenshotAfter": true
}

Example: See the Screen as Numbered Elements, Then Click One (marks)

Two calls, no coordinate arithmetic. Ask for a marked capture, pick the element you want by index, and act on it:

// handy_see
{ "marks": true, "maxMarks": 30 }

// handy_act — mark 13 was the "Submit" button in the capture above
{
  "actions": [
    { "type": "click", "mark": 5 },
    { "type": "paste", "text": "Sam — ready for review 🚀" },
    { "type": "click", "mark": 13 },
    { "type": "wait_for_settled", "quietMs": 200 }
  ],
  "verifyChange": true
}

handy_see returns each element's index, kind, bounds, centerX/centerY, and a score, and the marks stay valid for subsequent handy_act calls until the next marked capture. In the CLI, where every command is a separate process, a mark reference detects elements from the live screen on demand instead — detection is deterministic for an unchanged screen, so see --marks and a later act agree on the numbering. To guarantee the indices cannot drift (if the screen may change between commands), do both in one chain: handy act '[{"type":"screenshot","marks":true},{"type":"click","mark":5}]'. verifyChange: true adds a before/after frame diff so the receipt reports whether the screen actually changed — a cheap post-condition check that catches clicks which landed but did nothing.

The equivalent chain-DSL form works everywhere a chain is accepted:

handy see --marks --out marked.png
handy act 'click mark 5 -> paste "Sam" -> click mark 13 -> wait_for_settled'

Standalone CLI Usage

# Capture coordinate-mapped screen image — or zoom into a single window
handy see --out screen.png
handy see --window "Scratchpad" --highlight-diff --out scratchpad.png

# Detect & number interactive UI elements, printing each mark's centre, bounds, and score
handy see --marks --out marked.png
handy see --marks --max-marks 20 --json

# Inspect & focus desktop windows
handy windows
handy focus "Visual Studio Code"

# Read or write system clipboard
handy clipboard get
handy clipboard set "Copied from terminal"

# Execute a chained control sequence using human shorthand or JSON
handy act 'focus "Scratchpad" -> click 800 420 -> paste "Hello 🌍" -> wait_for_settled' --screenshot-after after.png

# Click a detected element by mark instead of by coordinate
handy act 'click mark 5 -> type "Sam" -> click #13' --json

License

MIT © Kevin Miiso Novo