Skip to main content
Glama

Three AI agents fill in three apps hidden behind your note at the same time, while you keep typing in it

Sidehand lets Claude, Codex, Cursor or any MCP client read and operate Windows apps behind the window you are using. It types, clicks, picks from lists and reads the screen as text without moving your mouse or taking your keyboard.

Why Sidehand

  • In the benchmark task your note kept the keyboard the whole time: 0 s lost, no window came forward, the cursor never moved.

  • It works in windows covered by yours, so you can keep using the one in front.

  • One short line per control: the task took 277 tokens, against 2,488 to 13,425 for the other tools. Reading a window takes about 4 to 5 times fewer tokens than agent-desktop on typical apps, and about 20 times fewer on an Edge page.

  • It finished the task in 2.0 s. Windows-MCP and Terminator, the other tools that finished it, took 65.5 s and 68.0 s.

Related MCP server: com-use-mcp

Quick start

You need Windows 10 or 11 and Node 18.3+. Add the server to your MCP client:

{
  "mcpServers": {
    "sidehand": { "command": "npx", "args": ["-y", "sidehand", "mcp"] }
  }
}

Client

Where

Claude Code

claude mcp add sidehand -- npx -y sidehand mcp

Cursor

~/.cursor/mcp.json

Codex

codex mcp add sidehand -- npx -y sidehand mcp

Claude Desktop

Settings › Developer › Edit Config

pi

pi install git:github.com/Blue-B/sidehand

On native Windows, Claude Desktop, Cursor and Codex need cmd /c npx. That, the Codex TOML, the CLI, the Claude Code guard hook and SIDEHAND_ALLOW are in docs/setup.md.

Or paste this to your AI agent and it sets itself up:

Install Sidehand (https://github.com/Blue-B/sidehand) for yourself: add an MCP server named "sidehand"
with command "npx" and args ["-y", "sidehand", "mcp"] to your own MCP settings.
If you are Claude Code, also run `npm install -g sidehand` and add the `sidehand guard`
PreToolUse hook from docs/setup.md in that repo. Then tell me whether I need to restart you.

Run sidehand doctor (or npx -y sidehand doctor) to check your setup.

How it compares

Sidehand

Claude computer use

ChatGPT computer use

agent-desktop

You keep typing while it works

✅

❌ hides your other windows on Windows

❌ moves the pointer and takes the foreground on Windows

❌ 4 windows came forward

Works on covered windows

✅

❌ on Windows

❌ on Windows

✅

Reads apps as text

✅

screenshots

screenshots

✅

Sidehand and agent-desktop were measured (docs/benchmark.md); the computer use cells come from the linked vendor pages. More tools and all sources: docs/comparison.md.

Usage

$ sidehand tree "Contoso Orders"
window: Contoso Orders  (18 elements)
  e4 Edit "Customer"
  e12 Button "Submit order"

$ sidehand act e4 type "Blue-B Studio"
$ sidehand act e12 click

Command

What it does

tree <title>

the window's controls as text, each with a ref (e4)

act <ref> <action> [value]

type, click, select, check, press, wheel, drag and more

ocr <title>

text of custom-drawn windows (games, Qt, canvas) via Windows OCR, with refs (t1) that act can click

wait <title> <text>

wait until a control with that text appears (or --gone)

window, launch

move, resize, restore or close a window; start an app without taking your focus

capture, mask

screenshot a covered window; black out, pixelate or blur regions

doctor

check your setup

MCP and pi expose the same operations as tools (ui_tree, ui_ocr, ui_act, ...). Every command and option: docs/usage.md.

Hide private info in screenshots

Blur emails, API keys and faces before you paste a screenshot into a blog post, issue or chat. Your agent can do it for you: ask "hide the email in this screenshot".

A fake account page, original and masked with black, pixelate and blur

$ sidehand mask shot.png "170,185,180,26;170,331,256,22" --style pixelate

What works in the background

Win32, WinForms

WPF

Chrome, Edge

VS Code

Type, click, check, select, expand

✅

✅

✅ except <select>

✅ click

Wheel, double-click

✅

✅ through UI Automation

✅

-

Right-click, hover, drag

✅

❌

✅

-

Keys and shortcuts

✅

Enter on a button or link; others with consent

keys in text fields, Enter on a button or link; others with consent

with consent

"With consent" means --allow-foreground, your permission to take the foreground. Games that read raw mouse input ignore posted clicks, but capture and ocr still work. Apps running as administrator need an agent running as administrator. Details: docs/usage.md.

Troubleshooting

Run sidehand doctor first. It checks Node, WSL, PowerShell, the execution policy, the engine and OCR languages, and touches no window.

  • spawn npx ENOENT on native Windows: start the server with cmd /c npx (docs/setup.md).

  • Codex startup timeout: add startup_timeout_sec = 60 under [mcp_servers.sidehand].

  • "running scripts is disabled on this system": Group Policy sets the execution policy; only your administrator can change it.

  • ocr says no language is installed: add a Windows OCR language pack.

  • An app running as administrator does not respond: run your agent as administrator too.

More: docs/troubleshooting.md.

Security

Text Sidehand reads from other windows goes to your agent, so treat it like web content: a page or message can try to instruct the agent. SIDEHAND_ALLOW limits which windows it sees, and every string it passes to PowerShell is quoted or refused.

License

PolyForm Shield 1.0.0. Free for any use, including at work, except building or offering a product that competes with Sidehand or with products Blue-B provides using it, free or paid. It includes a patent license that ends for anyone who claims Sidehand infringes a patent. For a commercial license, open an issue.

Available Tools

12 tools
captureA
Read-only

Screenshot: full, monitor, region, active or window (PrintWindow, works while covered). Returns the image. Capturing a minimized window restores it, so it needs bring_to_front after the user agreed.

Inputs: mode (required), h, w, x, +4 more.

clipboardA
Destructive

Read the clipboard text, or replace it with text (that overwrites what the user copied; prefer ui_act type).

Inputs: text.

cropB
Idempotent

Crop a PNG to a rectangle in image pixels. Returns the image.

Inputs: h (required), w (required), x (required), y (required), +1 more.

launchA

Start an app, document or URL without it taking the user's focus.

Inputs: app (required), args.

list_monitorsB
Read-only

List monitors with their virtual-desktop coordinates.

Inputs: no named parameters.

list_windowsA
Read-only

List visible top-level windows (handle, size, title).

Inputs: no named parameters.

maskB
Idempotent

Hide private information in a PNG before sharing it: black, pixelate or blur rectangles. Returns the image.

Inputs: in (required), regions (required), style.

ui_actA
Destructive

Act on a ref from the latest ui_tree/ui_wait/ui_ocr without moving the mouse or the user's focus. click | type (replace text) | check | uncheck | select (value= inside a list/combo) | expand | collapse | scroll | right-click | double-click | hover | wheel (value 'down 3') | drag (value= or 'dx,dy') | press (value 'Enter', 'ctrl+a', 'Down Down Enter'). press works in the background on classic controls and on a visible text field of a browser or Electron window. focus, and press on other WPF/browser controls (except Enter on a button or link), take the user's foreground: only with allow_foreground after the user agreed; press hands the foreground back once the keys are in, focus keeps it. If an action fails, never fall back to moving the real mouse or bringing the window forward yourself; tell the user. Refs expire when a window is read again.

Inputs: ref (required), action (required), value, allow_foreground.

ui_ocrA
Read-only

Only when ui_tree shows almost nothing (games, Qt, Sciter, canvas UIs): the window's text read by Windows OCR, one line per text with a ref (t3) and screen rect. ui_act takes a t-ref for click, double-click, right-click, hover, wheel and drag, posted at the text's center as mouse messages (no real cursor, no focus change). About 0.5 to 2 s; works while covered. t-refs expire after 60 s.

Inputs: title (required).

ui_treeA
Read-only

A window's controls as compact text, one line per control with a ref (e12). Long lists show 25 items; filter prints matching lines only (refs still cover the window); rects adds screen rectangles. If it shows almost nothing (games, Qt, canvas apps), use ui_ocr.

Inputs: title (required), rects, filter, max_nodes.

ui_waitA
Read-only

Poll a window until a control containing text appears (or, with gone, disappears). Use instead of sleeping.

Inputs: text (required), title (required), gone, timeout_ms.

windowA
Destructive

Move ('x,y'), resize ('w,h'), minimize, restore (back behind the user's windows) or close a window without activating it.

Inputs: title (required), action (required), value.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.8/5.0

    Scored across 12 tools

    Disambiguation5/5

    Each tool targets a distinct resource or action: image ops (capture/crop/mask), enumeration (list_windows/list_monitors), UI inspection (ui_tree/ui_ocr), and interaction (ui_act/ui_wait/window/launch). The ui_tree vs ui_ocr boundary is explicitly defined, and ui_act vs window is clearly separated (control-level vs window-level manipulation). The remaining overlap (capture/crop/mask) is a natural pipeline where each step is unambiguous.

    Naming Consistency4/5

    All names are lowercase snake_case with predictable families: list_* for enumeration, ui_* for UI automation, and bare verbs for image/clipboard actions. Minor inconsistency in that some tools are bare nouns (clipboard, window) while siblings are verbs (capture, crop, launch), but the grouping remains readable and predictable.

    Tool Count5/5

    Twelve tools is well within the ideal band and each one covers a needed capability of desktop automation (capture, inspect, act, window control, launch). No tool appears redundant or vestigial, and the surface is not bloated.

    Completeness4/5

    The lifecycle is broadly covered: discover windows/monitors, inspect via UI tree or OCR, act, wait, and manage window state. Gaps are minor: clipboard is text-only, there is no explicit window-activate tool (handled indirectly via ui_act focus), and no process-level launch/terminate or wait-for-window-primitive beyond ui_wait on controls.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI assistants to autonomously control Windows 11 and 10 desktops via sub-10ms screen capture, native UI Automation element inspection, and zero-lag keyboard and mouse input. Combines a visual plane with a semantic plane and stall detection so agents can operate real applications reliably without vision-only guessing.
      262 npm
      171 PyPI
      7
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI models to control a local Windows desktop by moving the cursor, clicking, scrolling, typing Unicode text, sending key chords, and capturing screenshots. Runs natively on Windows with minimal latency and no heavy dependencies such as Python, PyAutoGUI, or Docker.
      151 npm
      1
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to read what is on a Windows desktop as text — open windows, their contents, and what changed — using Win32 and UI Automation with optional NPU vision. Secrets, password fields, and private windows are withheld and personal data scrubbed on the user's PC before anything reaches the agent.
      AGPL 3.0