Sidehand

Sidehand lets Claude, Codex, Cursor or any MCP client read and operate Windows apps behind the window you are using. It types, clicks, picks from lists and reads the screen as text without moving your mouse or taking your keyboard.
Why Sidehand
In the benchmark task your note kept the keyboard the whole time: 0 s lost, no window came forward, the cursor never moved.
It works in windows covered by yours, so you can keep using the one in front.
One short line per control: the task took 277 tokens, against 2,488 to 13,425 for the other tools. Reading a window takes about 4 to 5 times fewer tokens than agent-desktop on typical apps, and about 20 times fewer on an Edge page.
It finished the task in 2.0 s. Windows-MCP and Terminator, the other tools that finished it, took 65.5 s and 68.0 s.
Related MCP server: com-use-mcp
Quick start
You need Windows 10 or 11 and Node 18.3+. Add the server to your MCP client:
{
"mcpServers": {
"sidehand": { "command": "npx", "args": ["-y", "sidehand", "mcp"] }
}
}Client | Where |
Claude Code |
|
Cursor |
|
Codex |
|
Claude Desktop | Settings › Developer › Edit Config |
pi |
|
On native Windows, Claude Desktop, Cursor and Codex need cmd /c npx. That, the Codex TOML, the CLI, the Claude Code guard hook and SIDEHAND_ALLOW are in docs/setup.md.
Or paste this to your AI agent and it sets itself up:
Install Sidehand (https://github.com/Blue-B/sidehand) for yourself: add an MCP server named "sidehand"
with command "npx" and args ["-y", "sidehand", "mcp"] to your own MCP settings.
If you are Claude Code, also run `npm install -g sidehand` and add the `sidehand guard`
PreToolUse hook from docs/setup.md in that repo. Then tell me whether I need to restart you.Run sidehand doctor (or npx -y sidehand doctor) to check your setup.
How it compares
Sidehand | ||||
You keep typing while it works | ✅ | ❌ hides your other windows on Windows | ❌ moves the pointer and takes the foreground on Windows | ❌ 4 windows came forward |
Works on covered windows | ✅ | ❌ on Windows | ❌ on Windows | ✅ |
Reads apps as text | ✅ | screenshots | screenshots | ✅ |
Sidehand and agent-desktop were measured (docs/benchmark.md); the computer use cells come from the linked vendor pages. More tools and all sources: docs/comparison.md.
Usage
$ sidehand tree "Contoso Orders"
window: Contoso Orders (18 elements)
e4 Edit "Customer"
e12 Button "Submit order"
$ sidehand act e4 type "Blue-B Studio"
$ sidehand act e12 clickCommand | What it does |
| the window's controls as text, each with a ref ( |
|
|
| text of custom-drawn windows (games, Qt, canvas) via Windows OCR, with refs ( |
| wait until a control with that text appears (or |
| move, resize, restore or close a window; start an app without taking your focus |
| screenshot a covered window; black out, pixelate or blur regions |
| check your setup |
MCP and pi expose the same operations as tools (ui_tree, ui_ocr, ui_act, ...). Every command and option: docs/usage.md.
Hide private info in screenshots
Blur emails, API keys and faces before you paste a screenshot into a blog post, issue or chat. Your agent can do it for you: ask "hide the email in this screenshot".

$ sidehand mask shot.png "170,185,180,26;170,331,256,22" --style pixelateWhat works in the background
Win32, WinForms | WPF | Chrome, Edge | VS Code | |
Type, click, check, select, expand | ✅ | ✅ | ✅ except | ✅ click |
Wheel, double-click | ✅ | ✅ through UI Automation | ✅ | - |
Right-click, hover, drag | ✅ | ❌ | ✅ | - |
Keys and shortcuts | ✅ | Enter on a button or link; others with consent | keys in text fields, Enter on a button or link; others with consent | with consent |
"With consent" means --allow-foreground, your permission to take the foreground.
Games that read raw mouse input ignore posted clicks, but capture and ocr still work.
Apps running as administrator need an agent running as administrator. Details: docs/usage.md.
Troubleshooting
Run sidehand doctor first. It checks Node, WSL, PowerShell, the execution policy, the engine and OCR languages, and touches no window.
spawn npx ENOENTon native Windows: start the server withcmd /c npx(docs/setup.md).Codex startup timeout: add
startup_timeout_sec = 60under[mcp_servers.sidehand]."running scripts is disabled on this system": Group Policy sets the execution policy; only your administrator can change it.
ocrsays no language is installed: add a Windows OCR language pack.An app running as administrator does not respond: run your agent as administrator too.
More: docs/troubleshooting.md.
Security
Text Sidehand reads from other windows goes to your agent, so treat it like web content: a page or message can try to instruct the agent.
SIDEHAND_ALLOW limits which windows it sees, and every string it passes to PowerShell is quoted or refused.
License
PolyForm Shield 1.0.0. Free for any use, including at work, except building or offering a product that competes with Sidehand or with products Blue-B provides using it, free or paid. It includes a patent license that ends for anyone who claims Sidehand infringes a patent. For a commercial license, open an issue.
Available Tools
12 toolscaptureARead-only
Screenshot: full, monitor, region, active or window (PrintWindow, works while covered). Returns the image. Capturing a minimized window restores it, so it needs bring_to_front after the user agreed.
Inputs: mode (required), h, w, x, +4 more.
clipboardADestructive
Read the clipboard text, or replace it with text (that overwrites what the user copied; prefer ui_act type).
Inputs: text.
cropBIdempotent
Crop a PNG to a rectangle in image pixels. Returns the image.
Inputs: h (required), w (required), x (required), y (required), +1 more.
launchA
Start an app, document or URL without it taking the user's focus.
Inputs: app (required), args.
list_monitorsBRead-only
List monitors with their virtual-desktop coordinates.
Inputs: no named parameters.
list_windowsARead-only
List visible top-level windows (handle, size, title).
Inputs: no named parameters.
maskBIdempotent
Hide private information in a PNG before sharing it: black, pixelate or blur rectangles. Returns the image.
Inputs: in (required), regions (required), style.
ui_actADestructive
Act on a ref from the latest ui_tree/ui_wait/ui_ocr without moving the mouse or the user's focus. click | type (replace text) | check | uncheck | select (value= inside a list/combo) | expand | collapse | scroll | right-click | double-click | hover | wheel (value 'down 3') | drag (value= or 'dx,dy') | press (value 'Enter', 'ctrl+a', 'Down Down Enter'). press works in the background on classic controls and on a visible text field of a browser or Electron window. focus, and press on other WPF/browser controls (except Enter on a button or link), take the user's foreground: only with allow_foreground after the user agreed; press hands the foreground back once the keys are in, focus keeps it. If an action fails, never fall back to moving the real mouse or bringing the window forward yourself; tell the user. Refs expire when a window is read again.
Inputs: ref (required), action (required), value, allow_foreground.
ui_ocrARead-only
Only when ui_tree shows almost nothing (games, Qt, Sciter, canvas UIs): the window's text read by Windows OCR, one line per text with a ref (t3) and screen rect. ui_act takes a t-ref for click, double-click, right-click, hover, wheel and drag, posted at the text's center as mouse messages (no real cursor, no focus change). About 0.5 to 2 s; works while covered. t-refs expire after 60 s.
Inputs: title (required).
ui_treeARead-only
A window's controls as compact text, one line per control with a ref (e12). Long lists show 25 items; filter prints matching lines only (refs still cover the window); rects adds screen rectangles. If it shows almost nothing (games, Qt, canvas apps), use ui_ocr.
Inputs: title (required), rects, filter, max_nodes.
ui_waitARead-only
Poll a window until a control containing text appears (or, with gone, disappears). Use instead of sleeping.
Inputs: text (required), title (required), gone, timeout_ms.
windowADestructive
Move ('x,y'), resize ('w,h'), minimize, restore (back behind the user's windows) or close a window without activating it.
Inputs: title (required), action (required), value.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 12 tools
Each tool targets a distinct resource or action: image ops (capture/crop/mask), enumeration (list_windows/list_monitors), UI inspection (ui_tree/ui_ocr), and interaction (ui_act/ui_wait/window/launch). The ui_tree vs ui_ocr boundary is explicitly defined, and ui_act vs window is clearly separated (control-level vs window-level manipulation). The remaining overlap (capture/crop/mask) is a natural pipeline where each step is unambiguous.
All names are lowercase snake_case with predictable families: list_* for enumeration, ui_* for UI automation, and bare verbs for image/clipboard actions. Minor inconsistency in that some tools are bare nouns (clipboard, window) while siblings are verbs (capture, crop, launch), but the grouping remains readable and predictable.
Twelve tools is well within the ideal band and each one covers a needed capability of desktop automation (capture, inspect, act, window control, launch). No tool appears redundant or vestigial, and the surface is not bloated.
The lifecycle is broadly covered: discover windows/monitors, inspect via UI tree or OCR, act, wait, and manage window state. Gaps are minor: clipboard is text-only, there is no explicit window-activate tool (handled indirectly via ui_act focus), and no process-level launch/terminate or wait-for-window-primitive beyond ui_wait on controls.
Maintenance
Related MCP Connectors
Connect ChatGPT, Claude, Cursor or VS Code to your Windows PC: files, terminal, browser, apps.
Connect AI assistants to your approved computer for files, commands, browser and desktop tasks.
Give your AI hands on every computer you own. Remote control of your Windows, Mac and Linux machines from one ChatGPT, Claude or Cursor chat — even from your phone. Ask it to find out why the office server is crawling, update every PC in the house, sort out your mom's laptop from another city, or ship the site from your work machine while you're on a train. It runs commands, reads files and takes a screenshot when it needs to see the screen. You decide what each computer allows, and you can cut any of them off in a second.
Eyes and hands on real Windows PCs — observe, click, type via Glasswarp API.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceEnables AI assistants to autonomously control Windows 11 and 10 desktops via sub-10ms screen capture, native UI Automation element inspection, and zero-lag keyboard and mouse input. Combines a visual plane with a semantic plane and stall detection so agents can operate real applications reliably without vision-only guessing.262 npm171 PyPI7MIT
- FlicenseNot gradedqualityCmaintenanceEnables AI agents to see and operate a Windows PC through mouse, keyboard, screenshots, OCR, window control, and shell commands.-
- AlicenseNot gradedqualityCmaintenanceEnables AI models to control a local Windows desktop by moving the cursor, clicking, scrolling, typing Unicode text, sending key chords, and capturing screenshots. Runs natively on Windows with minimal latency and no heavy dependencies such as Python, PyAutoGUI, or Docker.151 npm1MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to read what is on a Windows desktop as text — open windows, their contents, and what changed — using Win32 and UI Automation with optional NPU vision. Secrets, password fields, and private windows are withheld and personal data scrubbed on the user's PC before anything reaches the agent.AGPL 3.0