Hands

Computer use for AI agents on Windows that works beside you, not instead of you. Your agent drives desktop apps and Chrome in the background while you keep your mouse, your keyboard and your window. You can always see what it is doing, and one keypress stops it.

⚠️ Read this before installing

Hands lets an AI agent operate the apps on your computer and, with the Chrome extension, the web pages it opens. The extension uses Chrome's debugger permission, which gives it full control of the tabs it opens (it never attaches to your own tabs, but the permission itself is powerful).

• Install it only on a machine you control, for an agent you trust.

• Review the security model and the deny list before first use.

• Do not leave sensitive work (banking, email, admin consoles) open and unattended while an agent holds the hands.

• Hands is provided as is, without warranty of any kind (see LICENSE). You are responsible for what you allow an agent to do on your computer.

Related MCP server: jenny-mcp-server

What it does

• Sees any window, even a covered one: a screenshot (full window content, JPEG ≤1568 px) plus an indexed accessibility tree from one cached UI Automation round trip (typically 25–80 ms).

• Acts in the background first: clicking an element goes through UI Automation patterns (Invoke, Toggle, Select, Expand, default action), values through ValuePattern, typing into classic edit boxes without focus. Your cursor and focus are not touched.

• Real input only when unavoidable, politely: it waits until your hands have rested ≥0.7 s, acts, returns your cursor at once and gives your window back. If you move during it, the batch stops.

• Always visible: a soft glow round every screen while the agent works, a brighter border and a pill (" is working here · Esc Esc to stop · Stop") round the window it is in, and its own cursor that travels to each target before the action lands. None of it appears in screenshots (WDA_EXCLUDEFROMCAPTURE). The name, initial and colours are configurable.

• Asks first: the first use of each app or site shows a card (No / Allow once / Always).

• Cleans up: windows and tabs the agent opened close when it is done; anything with unsaved changes is left open and reported, never discarded. A window that holds your documents is never treated as the agent's.

• One agent at a time: several agent sessions can each run Hands; a lease (the "bouncer") lets one hold the hands, tells the others exactly who holds them, keeps a line, and nudges the next one when they free.

• Chrome in background tabs: the agent opens its own tabs, in its own tab group, and drives them through the debugger with focus emulation; your active tab, mouse and keyboard stay yours.

Windows 10/11 only. Built clean-room after studying how other agents' computer use behaves; it uses no code or binaries from any other computer-use product.

Install

Requirements: Windows 10 or 11, Python 3.11+, Google Chrome (for the browser part), and the .NET Framework 4.x compiler that ships with Windows (for the overlay).

git clone <this repository> hands
cd hands
pip install -r requirements.txt
overlay\build.cmd

Register the MCP server with your agent host. For Claude Code:

claude mcp add hands -- python -X utf8 C:\path\to\hands\mcp_server.py

Any MCP client that speaks stdio works the same way: command python, arguments -X utf8 <repo>\mcp_server.py.

Chrome (optional)

1. python -X utf8 chrome\install_host.py registers the native messaging host for the current user (--status shows it, --uninstall removes it).

2. In Chrome: chrome://extensions → Developer mode → Load unpacked → the chrome\extension folder. The id is pinned by the manifest's key, so it matches what the host allows.

3. Restart the MCP server. The host and the server must run from the same folder (they share state\chrome.key).

Electron apps

Some Electron and Chromium apps only expose their accessibility tree while Windows' screen-reader flag is on. That flag is system-wide, so Hands leaves it alone unless you opt in with "screen_reader_flag": true in state\config.json; it is then set each time Hands starts (apps read it when they launch). It resets at sign-out.

Use

The loop an agent follows:

1. hands_windows (list) or hands_launch (open an app; returns its new window)

2. hands_look(window): screenshot + tree, e.g. [12] button "Save" #saveButton (invoke) @410,288

3. hands_act(window, look, actions): a batch of actions, then a fresh look in the same call

4. verify the outcome

5. hands_done: closes what the agent opened, hands the hands back

Browser: browser_tabs, browser_open, browser_look, browser_act, browser_close. Whole screen: hands_screen. Who holds the hands: hands_who.

The full guide for agents (every action, consent, cleanup, the bouncer, troubleshooting) is skills/hands/SKILL.md. Drop it into your agent's skills so it loads before the tools are used.

Configuration

Everything is optional. state\config.json:

{
  "identity": {
    "name": "Nova",
    "initial": "N",
    "color": "#00A88F",
    "color_hot": "#5CFFE0",
    "session": "nova"
  },
  "screen_reader_flag": false,
  "notify_cmd": ["C:\\Python311\\python.exe", "C:\\path\\to\\notify.py"],
  "nudge_cmd": ["C:\\Python311\\python.exe", "C:\\path\\to\\nudge.py"]
}

• identity: how the agent looks and is named on screen. Invalid values are ignored. chrome_host and chrome_pipe can be changed too, but the host name must then also be changed in chrome\extension\sw.js.

• notify_cmd: run with one message argument whenever the host-app door (below) is used. With none configured, that door stays shut.

• nudge_cmd: run with one JSON argument {"house", "id", "message"} when the hands free for a waiting session. Route it to that session however your agent host receives messages.

• Both commands must start with the full path of an existing .exe that is not a shell (cmd.exe, PowerShell, script hosts are refused) and a .bat / .cmd never runs: the nudge's argument carries text another session chose, so it must arrive at your program as one plain argument. Treat it as data in your script too.

• Environment: HANDS_HOUSE (which kind of host, e.g. cc) and HANDS_SESSION (that host's session or thread id) name the session exactly for the bouncer. Without them a session is named by its parent process id.

The bouncer

Every screen-touching tool takes or renews a lease shared by all Hands servers on the machine (a named mutex and state\bouncer.json). A refused call reads like:

Hold up: CC session 7f3a… is holding the hands since 21:25 (last move 4 s ago, hands_act). You're #1 in line; I'll nudge you the moment they're free (a silent holder lets go after 3 min).

The lease frees on hands_done, when the holder's server exits or its process dies, after 3 minutes of silence, or when you press Stop (which nudges nobody). The first in line is nudged by its exact id and its place is held for 60 s. Listing tools and hands_who never need the lease.

Security model

• Background first, visible always, stoppable always. Esc Esc, the pill's Stop, or Cancel on Chrome's debugging bar stops both desktop and browser; the agent resumes only by looking again.

• Consent per app and per site, stored in state\consent.json ("Always") or kept for the session ("Once").

• Hard deny list (no consent can open it): terminals and shells, the Run dialog, UAC / credential / lock screens, Windows Security, password managers (by host in the browser), registry and management consoles, other agent hosts, and browser-internal pages. The Windows key is never pressed.

• Host-app door: the one exception to the agent host being denied is a narrow, optional door (hands_claude_app) that can only turn a session's Remote Control switch on or press New, and reports every use through notify_cmd.

• Chrome: the agent drives only tabs it opened (a tab your page opens is never its), the page and every frame pass the deny list and consent, frames it may not read are painted over in screenshots (or the screenshot is withheld), only trusted presses confirm a click, and a tab that lands on a denied page is blanked or closed.

• Local only: the extension talks to a native host, the host to the engine over a named pipe with a shared key challenge both ways (state\chrome.key, never printed or logged). No TCP port is opened.

• Never type passwords, payment details or identity numbers through Hands; sign-ins are yours to do.

Layout

Tests

python -X utf8 tests\run.py runs every asserting test and exits non-zero on any failure. Desktop tests open their own Calculator, console and scratch windows; browser tests start their own Chrome with a temporary profile and their own pipe. They do appear on screen: run them when you are not using the machine. One criterion (real-input fallback while you are typing) needs a person's hands and is reported SKIPPED-MANUAL.

Test switches only work together with HANDS_TEST_HOOKS=1: HANDS_TEST_ALLOW=<key;key> lets a test drive what it opens, HANDS_PROOF=1 makes the overlay visible to screenshots. Never set them in normal use.

License

Apache License 2.0. See LICENSE and NOTICE. Copyright 2026 Code & Chaos.