🖐️ Grasp

Give Claude eyes and hands — full desktop control over MCP.

Grasp is a Model Context Protocol server that lets Claude see your screen and drive your mouse, keyboard and shell. Install it once and Claude Code — or any MCP host — can operate your Windows PC the way a person would: look, point, click, type, run commands.

What "eyes and hands" means. Eyes — Grasp captures the screen, resizes it to a vision-friendly resolution and draws a coordinate grid on top, so Claude can read exact positions instead of guessing. It can also read the screen as text with zero API key using Windows UI Automation + built-in OCR. Hands — Grasp injects real, DPI-correct mouse and keyboard input at the OS level (a Rust backend), and runs shell commands directly. Together they close the loop: Claude looks, decides, acts, looks again — and a crash-proof supervisor keeps the tools alive across the whole session.

Why Grasp

Most "computer use" setups are either a cloud VM you don't control, or a browser-only automation that can't touch the rest of your machine. Grasp runs locally, drives the real desktop, and plugs into the tools you already use through a single mcp add command.

• Pixel-perfect on scaled displays. The Rust input backend respects Windows display scaling (125% / 150% / 200%) — clicks land exactly where Claude looked, not two centimetres off.

• Vision tuned for accuracy. Screenshots are cropped/resized to ~1280 px with a 100-px coordinate grid overlay, the single biggest lever on click precision.

• Key-free text vision. describe_screen works with no API key and no network — Windows UI Automation reads real interactive elements and built-in OCR reads on-screen text, both with screen coordinates. A cheap cloud model is optional, not required.

• Survives crashes. A supervisor process fronts the server: if it ever dies, the supervisor respawns it and replays the MCP handshake, so your session keeps its tools — no manual /mcp → Reconnect.

• Multi-monitor aware. Capture any display by id; coordinates map back to the right monitor's physical pixels.

• Shell without the screenshot tax. run_command returns real text output — no opening a terminal, typing into it, and screenshotting the result.

• Safe by construction. Input is released on startup/shutdown so a stuck modifier can never freeze your keyboard; failing and elevated commands are reported honestly, never as silent successes.

Related MCP server: computer-use-windows

Tools

Tool reference

Every parameter, default and return value. Coordinate-taking tools all accept an optional coord_space: "screenshot" (default — numbers read off the last screenshot's grid) or "screen" (raw absolute pixels).

👁️ Eyes

screenshot

Captures the screen and returns a JPEG with a coordinate grid drawn every 100 px, plus a text line with screen size, DPI scale and the active window title.

Coordinates you read off the grid map 1:1 to click / move_mouse arguments — no math needed, even on 125 % / 150 % / 200 % scaled displays or a secondary monitor.

describe_screen

Turns the screen into a short text list of interactive elements with (x,y) coordinates and a one-line state, so the main model reasons over a few hundred text tokens instead of 1000+ image tokens. Two engines, chosen by engine:

• local (default) — Windows UI Automation lists real interactive elements (buttons, fields, list/menu items) with names, and the built-in Windows OCR reads on-screen text. No API key, no network. Reports screen pixels, so click back with coord_space:"screen".

• api — a cheap Claude vision model (Haiku by default) describes the frame. Needs ANTHROPIC_API_KEY (or GRASP_API_KEY). Reports screenshot-grid coordinates (default coord_space).

• auto — api if a key is set, else local.

Coordinate space differs by engine. local gives screen pixels → click with coord_space:"screen" (the tool's header reminds you). api gives screenshot-grid coordinates → use the default coord_space. The header line always states which.

get_active_window

No parameters. Returns the focused window's title, process name and bounds (x, y, width, height in physical pixels).

list_windows

No parameters. Returns every open top-level window: title, process, size and position.

get_screen_info

No parameters. Returns each monitor's resolution, capture size, DPI scale factor, origin and whether it is the primary display.

✋ Hands

click

Glides the cursor to the point (so hover-triggered UI reacts naturally) and clicks.

double_click

Same as click (left button only) but a double click — open files, select words. Parameters: x, y, coord_space.

move_mouse

Moves the cursor without clicking — reveal hover-only menus and tooltips before deciding. Parameters: x, y, coord_space.

drag

Press at one point, glide with the button held, release at another. Selections, sliders, drag-and-drop, window moves.

type_text

Inserts literal text at the current focus. Click the target field first.

How it works (v1.1.1+): the text goes through the clipboard + Ctrl+V, not per-key injection. That makes it instant for any length, fully Unicode-safe (Cyrillic, emoji, mixed scripts) and independent of the active keyboard layout. Your previous clipboard text is restored right after the paste (non-text clipboard content — images, files — is not preserved). Direct key injection remains as an automatic fallback if the clipboard is unavailable.

Why not raw key events? Injected key floods lose events on real apps — in live testing a 119-character string arrived as 15 characters and "OK" arrived as "KKKKKKK" (a lost key-up plus autorepeat). The clipboard path has none of these failure modes.

press_key

Presses a single key or a combo. Use this — not type_text — for shortcuts and navigation.

Recognised names: letters/digits, enter tab esc backspace delete home end pageup pagedown up down left right space f1–f12, and modifiers ctrl alt shift win chained with +.

scroll

Scrolls the wheel at a point (the cursor moves there first).

⚙️ System

run_command

Runs a PowerShell command and returns its combined stdout+stderr as UTF-8 text, prefixed with a status header: exit=<code> [(via UAC)] [[TIMED OUT]] [[truncated]] in <ms>ms.

A non-zero exit code marks the result as an error so the model reacts instead of assuming success. Grasp restores PATH/PATHEXT for MCP hosts that launch servers with a stripped environment, so cmd, ping, net and friends resolve normally.

wait

Pauses ms milliseconds (1–30000). Use it to let an app finish loading before the next look.

Token economy: cheap eyes, your brain

Sending a full screenshot to a top-tier model on every step is expensive — a 1280-px frame costs ~1000+ image tokens each look. Grasp splits vision from reasoning:

1. describe_screen returns a terse text map — Play button [button] — (720,410), Library tab [tabitem] — (556,184), plus a one-line state. A few hundred text tokens.

   • Local engine (default): Windows UI Automation + OCR read the elements and text directly off the desktop. No key, no network, no per-look cost.

   • Cloud engine (optional): a cheap model (Claude Haiku) describes the frame instead.

2. Your model (whatever the host runs — Sonnet, Opus, Fable; your choice) reads that text, decides, and calls click. It never pays the image-token tax unless it explicitly asks for a screenshot to check fine detail.

The local engine needs no configuration. To use the optional cloud engine instead, set:

"env": {
  "ANTHROPIC_API_KEY": "sk-ant-...",        // key for the cheap recognition model
  "GRASP_VISION_MODEL": "claude-haiku-4-5"  // optional; this is the default
}

With a key, engine:"auto" prefers the cloud model; without one it falls back to local. You can always force either with engine:"local" / engine:"api".

Reliability: the server survives crashes

An MCP host launches a stdio server once. If that process dies mid-session — a native fault, an OOM, a stray error — the host does not relaunch it, and every Grasp tool silently disappears until you run /mcp → Reconnect by hand.

Grasp fixes this with a supervisor (src/supervisor.js) that the host launches instead of the server directly. It:

1. spawns the real server as a child and proxies JSON-RPC both ways;

2. records the initialize handshake the host sent;

3. if the child ever exits, respawns it and replays that handshake, so the new process is in the exact MCP state the host believes it is — the tools stay live with no reconnect;

4. answers any in-flight request with a clear error while the child is down, so a call fails fast instead of hanging, and has a restart back-off so a hard-broken build can't spin forever.

The server itself also guards uncaughtException/unhandledRejection and keeps serving. This is covered by an automated test (npm run test:supervisor) that kills the underlying server mid-session and asserts the same client keeps working.

Operating rules (baked in)

Grasp ships MCP instructions that the host injects into the model's context, so any Claude driving Grasp follows two hard rules:

• Token economy — see with describe_screen first; only reach for screenshot when the text map isn't enough.

• Finish the job — carry a task through to completion. If a GUI path fails (some apps like Steam ignore synthetic clicks), switch to run_command, a config/registry edit, or keyboard shortcuts — there is almost always a reliable non-GUI path. Stop early only for destructive actions or things that genuinely need the user.

How the coordinate model works

1. Claude calls screenshot. Grasp captures the screen, resizes the longest edge to 1280 px, and overlays a grid labelled every 100 px.

2. Claude reads a coordinate straight off the grid (e.g. "the button is near x≈540, y≈300").

3. Claude calls click with those numbers. Grasp maps them back through the resize + crop + DPI transform to the exact physical pixel and clicks there.

You never do the math — pass coordinates in the same space you see them. (Advanced: pass coord_space: "screen" to use raw physical pixels instead.)

Multi-monitor. Pass monitor: <id> (ids come from get_screen_info) to screenshot or describe_screen to work on a secondary display. Grasp folds that monitor's origin into the transform, so a coordinate read off a secondary-monitor capture still maps to the correct absolute pixel. The local describe_screen engine reports screen pixels directly — click them back with coord_space:"screen".

Requirements

• Windows 10/11, x64. The mouse/keyboard backend ships as a prebuilt native binary for win32-x64. Screen capture, window enumeration and image processing are cross-platform, but Grasp is Windows-first today.

• Node.js ≥ 18.

• An MCP host — Claude Code, Claude Desktop, or anything that speaks MCP over stdio.

Install

git clone https://github.com/Jamshed7470/grasp-mcp.git
cd grasp-mcp
npm install
npm run doctor   # verify Grasp can see and control this machine

npm run doctor should end with ✅ Grasp is ready — Claude has eyes and hands.

Add to Claude Code

From the repo directory:

claude mcp add grasp -- node "%CD%\src\supervisor.js"

…or with an absolute path from anywhere:

claude mcp add grasp -- node "C:\path\to\grasp-mcp\src\supervisor.js"

Point the host at supervisor.js, not index.js — that's what keeps the tools alive across a crash (see Reliability). Running index.js directly still works and is handy for debugging; you just lose auto-recovery.

Then just ask Claude, e.g. "take a screenshot and open Settings for me."

Add to Claude Desktop

Edit claude_desktop_config.json (%APPDATA%\Claude\claude_desktop_config.json) and add:

{
  "mcpServers": {
    "grasp": {
      "command": "node",
      "args": ["C:\\path\\to\\grasp-mcp\\src\\supervisor.js"]
    }
  }
}

Restart Claude Desktop. Grasp's tools appear under the 🔨 menu.

Usage examples

Once installed, drive it in plain language — Claude picks the tools:

• "Open Notepad, type today's date, and save it to the Desktop."

• "Find the Wi-Fi icon in the tray and tell me which network I'm on." (screenshot → read)

• "Update all my winget packages." (run_command with elevated: true)

• "Scroll the page down and click the first search result."

What a real session looks like

A verified live run — "open Notepad and write a note", GUI only, like a human at the desk:

1. screenshot                      → sees the desktop, finds the Start button on the taskbar
2. click(505, 713)                 → opens the Start menu (grid coordinate read off the shot)
3. type_text("notepad")            → search field auto-focuses, query typed
4. screenshot                      → sees "Notepad — App" as the best match
5. click(540, 349)                 → launches Notepad
6. get_active_window               → confirms: "Untitled - Notepad" (Notepad.exe) is focused
7. press_key("ctrl+n")             → fresh tab, never touching existing unsaved tabs
8. type_text("Meeting notes …")    → full text lands instantly via the clipboard path
9. screenshot(mode=active_window)  → verifies the text is really in the editor before saving

Each look → decide → act → look-again cycle is exactly how Grasp is meant to be driven: never click blind, always re-check coordinates against the most recent capture.

Configuration

Per-tool options (image size, grid on/off, JPEG quality, command timeout, working directory, elevation) are passed as tool arguments — see each tool's description in the MCP schema.

⚠️ Security

Grasp gives an AI model real control of your computer — the same reach you have. Treat it accordingly.

• You are always in the loop. MCP hosts ask before each tool call by default. Keep it that way for anything destructive.

• Close sensitive windows (password managers, private messages) before running an agent — whatever is on screen is what Claude sees, and screenshots may briefly hold that content.

• Elevated commands need your hand. elevated: true triggers a Windows UAC prompt you must approve; UAC runs on a secure desktop that cannot be automated. If you dismiss it, Grasp reports the command as not run — never a silent success.

• Never commit screenshots. This repo's .gitignore blocks *.png/*.jpg for exactly this reason.

• Run Grasp only on machines and tasks you're comfortable handing to an assistant.

Troubleshooting

Run npm run doctor any time to re-check every backend.

Known limitations

• Windows-first. The input backend ships as a prebuilt win32-x64 binary. macOS/Linux contributions welcome (see Development).

• Some apps ignore synthetic input — by design, and Grasp won't fight it. Steam's UI and games with anti-cheat deliberately drop injected clicks; that's a security control, and bypassing it isn't something Grasp attempts. Use the reliable non-GUI path instead — run_command, config-file or registry edits, keyboard shortcuts. The baked-in operating rules already tell the model to switch paths rather than give up.

• UAC cannot be automated — by design. Elevation prompts render on a secure desktop that no synthetic input can reach. A human must click Yes; Grasp reports honestly when they don't.

• describe_screen local engine covers standard apps well. UI Automation is rich for Win32/WinUI/WPF/Electron; it can be sparse for custom-drawn surfaces (some games, canvas apps), where OCR fills in text. For pixel-exact detail, screenshot is always there. The optional cloud engine needs an API key (subscription OAuth doesn't apply); the local engine needs neither key nor network.

• type_text replaces clipboard text only briefly. Your clipboard text is restored after the paste; images or copied files in the clipboard are not preserved.

• One virtual desktop at a time. Grasp sees and drives the active desktop session; it does not reach across RDP sessions, secure desktops or lock screens.

Changelog

1.2.0

• describe_screen works with no API key. New local engine (now the default): Windows UI Automation lists real interactive elements with names + coordinates, and the built-in Windows OCR reads on-screen text — no key, no network. The cloud (Haiku) engine is still available via engine:"api"; engine:"auto" picks based on whether a key is set.

• Crash-proof supervisor. The host now launches src/supervisor.js, which respawns the server and replays the MCP handshake if it ever dies — the session keeps its tools without a manual /mcp → Reconnect. In-flight calls fail fast instead of hanging; the server also guards uncaught errors and keeps serving. Covered by npm run test:supervisor.

• Multi-monitor. screenshot and describe_screen accept a monitor id; the capture transform folds in that monitor's origin so coordinates map to the correct absolute pixel on any display.

1.1.1

• Fixed: type_text lost or repeated characters on long/fast input. Raw injected key events flood real apps — live testing showed 119 chars arriving as 15, and "OK" arriving as "KKKKKKK" (lost key-up → autorepeat). Text now goes through clipboard + Ctrl+V: instant at any length, Unicode-safe, keyboard-layout independent, previous clipboard text restored. Direct injection kept as automatic fallback.

1.1.0

• Two-model token economy: new describe_screen tool — a cheap model (Claude Haiku) turns the frame into a text map of elements + coordinates so the main model doesn't pay the image-token tax on every look.

• Operating rules shipped as MCP instructions: token economy + finish-the-job (switch to run_command/config edits when a GUI path fails; never stop at the first obstacle).

1.0.0

• Initial release: 13 tools (eyes / hands / system), grid-overlay coordinate model with DPI-correct click mapping, PowerShell runner with honest UAC elevation, doctor + end-to-end MCP tests.

How it's built

src/
  supervisor.js       crash-proof stdio proxy — respawns index.js + replays the handshake (host entry)
  index.js            MCP server — registers the 14 tools + operating instructions over stdio
  native.js           screen capture + mouse/keyboard (node-enigo, node-screenshots, get-windows)
  image-processor.js  sharp pipeline: crop → resize → coordinate-grid overlay
  uia.js              local key-free vision — UI Automation elements + Windows OCR → text map
  vision.js           optional cloud recognition — screenshot → text map (Anthropic API)
  shell.js            PowerShell runner with UTF-8 output + UAC elevation
  doctor.js           standalone backend health check
scripts/
  uia-dump.ps1        UI Automation element dump (interactive elements + screen coordinates)
  ocr-screen.ps1      Windows.Media.Ocr text recognition of a captured frame
native/
  node-enigo-win32-x64.node   prebuilt Rust input backend
test/
  mcp-test.js         end-to-end test over real MCP stdio (read-only + safe tools)
  supervisor-test.js  kills the server mid-session, asserts the supervisor keeps tools alive
  live-notepad.js     opt-in live test that types into a throwaway file and reads it back

Backends: node-enigo (Rust, mouse/keyboard) · node-screenshots (Rust, capture) · get-windows · sharp · @modelcontextprotocol/sdk.

Development

npm run doctor            # health check, no server
npm test                  # 14 end-to-end assertions over MCP stdio
npm run test:supervisor   # crash-recovery test: kills the server, checks tools stay alive
node test/live-notepad.js # opt-in: drives the real desktop, cleans up after itself

Contributions welcome — especially a macOS/Linux input backend to make Grasp truly cross-platform.

License

MIT. Grasp bundles a prebuilt node-enigo binary and depends on node-screenshots, both built on the Rust enigo / xcap ecosystem.