Grasp MCP Server
by Jamshed7470
README.md
<div align="center">
# ποΈ Grasp
### Give Claude eyes and hands β full desktop control over MCP.
Grasp is a [Model Context Protocol](https://modelcontextprotocol.io) server that lets Claude **see your screen** and **drive your mouse, keyboard and shell**. Install it once and Claude Code β or any MCP host β can operate your Windows PC the way a person would: look, point, click, type, run commands.
[](https://modelcontextprotocol.io)
[](https://nodejs.org)
[](#requirements)
[](LICENSE)
</div>
---
> **What "eyes and hands" means.** *Eyes* β Grasp captures the screen, resizes it to a vision-friendly resolution and draws a coordinate grid on top, so Claude can read exact positions instead of guessing. It can also read the screen as **text with zero API key** using Windows UI Automation + built-in OCR. *Hands* β Grasp injects real, DPI-correct mouse and keyboard input at the OS level (a Rust backend), and runs shell commands directly. Together they close the loop: Claude looks, decides, acts, looks again β and a **crash-proof supervisor** keeps the tools alive across the whole session.
## Why Grasp
Most "computer use" setups are either a cloud VM you don't control, or a browser-only automation that can't touch the rest of your machine. Grasp runs **locally**, drives the **real desktop**, and plugs into the tools you already use through a single `mcp add` command.
- **Pixel-perfect on scaled displays.** The Rust input backend respects Windows display scaling (125% / 150% / 200%) β clicks land exactly where Claude looked, not two centimetres off.
- **Vision tuned for accuracy.** Screenshots are cropped/resized to ~1280 px with a 100-px coordinate grid overlay, the single biggest lever on click precision.
- **Key-free text vision.** `describe_screen` works with **no API key and no network** β Windows UI Automation reads real interactive elements and built-in OCR reads on-screen text, both with screen coordinates. A cheap cloud model is optional, not required.
- **Survives crashes.** A supervisor process fronts the server: if it ever dies, the supervisor respawns it and replays the MCP handshake, so your session keeps its tools β no manual `/mcp` β Reconnect.
- **Multi-monitor aware.** Capture any display by id; coordinates map back to the right monitor's physical pixels.
- **Shell without the screenshot tax.** `run_command` returns real text output β no opening a terminal, typing into it, and screenshotting the result.
- **Safe by construction.** Input is released on startup/shutdown so a stuck modifier can never freeze your keyboard; failing and elevated commands are reported honestly, never as silent successes.
## Tools
| | Tool | What it does |
|---|---|---|
| ποΈ | `describe_screen` | **Cheap eyes:** turn the screen into a short text list of elements + coordinates β reason over that instead of spending image tokens. Default engine is **local** (UI Automation + OCR, no API key); a cheap cloud model is optional. |
| ποΈ | `screenshot` | Capture the screen (full or active window) as an image with a coordinate grid. |
| ποΈ | `get_active_window` | Title, process and bounds of the focused window. |
| ποΈ | `list_windows` | All open top-level windows. |
| ποΈ | `get_screen_info` | Monitor geometry, resolution and DPI scale. |
| β | `click` | Glide the cursor to a point and click (left / right / middle). |
| β | `double_click` | Double-click at a point. |
| β | `move_mouse` | Move the cursor (to reveal hover-only menus). |
| β | `drag` | Press, glide with the button held, release β selections, sliders, drag-and-drop. |
| β | `type_text` | Insert literal text at the current focus β any length, any language (clipboard-backed, see reference below). |
| β | `press_key` | Keys and combos: `enter`, `ctrl+c`, `alt+f4`, `win+r`, `ctrl+shift+esc`β¦ |
| β | `scroll` | Scroll up/down at a point. |
| βοΈ | `run_command` | Run a shell command (PowerShell), optionally elevated via UAC; returns text output. |
| βοΈ | `wait` | Pause N ms to let an app settle before the next screenshot. |
## Tool reference
Every parameter, default and return value. Coordinate-taking tools all accept an optional
`coord_space`: `"screenshot"` (default β numbers read off the last screenshot's grid) or
`"screen"` (raw absolute pixels).
### ποΈ Eyes
#### `screenshot`
Captures the screen and returns a JPEG with a coordinate grid drawn every 100 px, plus a text
line with screen size, DPI scale and the active window title.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `mode` | `"fullscreen"` \| `"active_window"` | `fullscreen` | Crop to the focused window for a closer look at one app. |
| `grid` | boolean | `true` | Draw the coordinate grid overlay. |
| `max_dimension` | int 320β3840 | `1280` | Longest edge of the returned image in px. Raise it to read small text. |
| `quality` | int 1β100 | `80` | JPEG quality. |
| `monitor` | int | primary | Monitor id to capture (from `get_screen_info`) on a multi-display setup. |
> Coordinates you read off the grid map **1:1** to `click` / `move_mouse` arguments β no math needed,
> even on 125 % / 150 % / 200 % scaled displays or a secondary monitor.
#### `describe_screen`
Turns the screen into a short text list of interactive elements with `(x,y)` coordinates and a
one-line state, so the main model reasons over a few hundred text tokens instead of 1000+ image
tokens. Two engines, chosen by `engine`:
- **`local` (default)** β Windows **UI Automation** lists real interactive elements (buttons,
fields, list/menu items) with names, and the built-in **Windows OCR** reads on-screen text.
**No API key, no network.** Reports **screen** pixels, so click back with `coord_space:"screen"`.
- **`api`** β a cheap Claude vision model (Haiku by default) describes the frame. Needs
`ANTHROPIC_API_KEY` (or `GRASP_API_KEY`). Reports **screenshot-grid** coordinates (default
`coord_space`).
- **`auto`** β `api` if a key is set, else `local`.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `mode` | `"fullscreen"` \| `"active_window"` | `fullscreen` | What to capture. |
| `engine` | `"local"` \| `"api"` \| `"auto"` | `local` | Recognition engine (see above). |
| `include_text` | boolean | `true` | `local` engine: also OCR on-screen text, not just UIA elements. |
| `monitor` | int | primary | `local` engine: monitor id for OCR (from `get_screen_info`). |
| `question` | string | β | Optional focus, e.g. `"where is the Library tab?"`. |
> **Coordinate space differs by engine.** `local` gives screen pixels β click with
> `coord_space:"screen"` (the tool's header reminds you). `api` gives screenshot-grid coordinates β
> use the default `coord_space`. The header line always states which.
#### `get_active_window`
No parameters. Returns the focused window's title, process name and bounds
(`x`, `y`, `width`, `height` in physical pixels).
#### `list_windows`
No parameters. Returns every open top-level window: title, process, size and position.
#### `get_screen_info`
No parameters. Returns each monitor's resolution, capture size, DPI scale factor, origin and
whether it is the primary display.
### β Hands
#### `click`
Glides the cursor to the point (so hover-triggered UI reacts naturally) and clicks.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `x`, `y` | number | required | Target point. |
| `button` | `"left"` \| `"right"` \| `"middle"` | `left` | Mouse button. |
| `coord_space` | see above | `screenshot` | |
#### `double_click`
Same as `click` (left button only) but a double click β open files, select words.
Parameters: `x`, `y`, `coord_space`.
#### `move_mouse`
Moves the cursor without clicking β reveal hover-only menus and tooltips before deciding.
Parameters: `x`, `y`, `coord_space`.
#### `drag`
Press at one point, glide with the button held, release at another. Selections, sliders,
drag-and-drop, window moves.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `from_x`, `from_y` | number | required | Start point. |
| `to_x`, `to_y` | number | required | End point. |
| `button` | `"left"` \| `"right"` | `left` | Button to hold. |
| `coord_space` | see above | `screenshot` | |
#### `type_text`
Inserts literal text at the current focus. Click the target field first.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `text` | string | required | The text to insert. |
**How it works (v1.1.1+):** the text goes through the **clipboard + Ctrl+V**, not per-key
injection. That makes it instant for any length, fully Unicode-safe (Cyrillic, emoji, mixed
scripts) and independent of the active keyboard layout. Your previous clipboard **text** is
restored right after the paste (non-text clipboard content β images, files β is not preserved).
Direct key injection remains as an automatic fallback if the clipboard is unavailable.
> Why not raw key events? Injected key floods lose events on real apps β in live testing a
> 119-character string arrived as 15 characters and `"OK"` arrived as `"KKKKKKK"` (a lost key-up
> plus autorepeat). The clipboard path has none of these failure modes.
#### `press_key`
Presses a single key or a combo. Use this β **not** `type_text` β for shortcuts and navigation.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `key` | string | required | e.g. `enter`, `esc`, `tab`, `f5`, `ctrl+c`, `ctrl+shift+esc`, `alt+f4`, `win+r` |
Recognised names: letters/digits, `enter` `tab` `esc` `backspace` `delete` `home` `end`
`pageup` `pagedown` `up` `down` `left` `right` `space` `f1`β`f12`, and modifiers
`ctrl` `alt` `shift` `win` chained with `+`.
#### `scroll`
Scrolls the wheel at a point (the cursor moves there first).
| Parameter | Type | Default | Description |
|---|---|---|---|
| `x`, `y` | number | required | Where to scroll. |
| `direction` | `"up"` \| `"down"` | required | Scroll direction. |
| `amount` | int 1β20 | `3` | Scroll steps. |
| `coord_space` | see above | `screenshot` | |
### βοΈ System
#### `run_command`
Runs a PowerShell command and returns its combined stdout+stderr as UTF-8 text, prefixed with a
status header: `exit=<code> [(via UAC)] [[TIMED OUT]] [[truncated]] in <ms>ms`.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `command` | string | required | The command to run. |
| `elevated` | boolean | `false` | Run as administrator via a **UAC prompt the user must approve by hand**. If the user dismisses it, the result honestly says the command was *not run* β never a silent success. |
| `timeout_ms` | int 1000β600000 | `60000` | Kill the command after this long. |
| `cwd` | string | β | Working directory. |
A non-zero exit code marks the result as an error so the model reacts instead of assuming success.
Grasp restores `PATH`/`PATHEXT` for MCP hosts that launch servers with a stripped environment, so
`cmd`, `ping`, `net` and friends resolve normally.
#### `wait`
Pauses `ms` milliseconds (1β30000). Use it to let an app finish loading before the next look.
## Token economy: cheap eyes, your brain
Sending a full screenshot to a top-tier model on every step is expensive β a 1280-px frame costs
~1000+ image tokens each look. Grasp splits vision from reasoning:
1. **`describe_screen`** returns a terse text map β `Play button [button] β (720,410)`,
`Library tab [tabitem] β (556,184)`, plus a one-line state. A few hundred text tokens.
- **Local engine (default):** Windows UI Automation + OCR read the elements and text directly
off the desktop. **No key, no network, no per-look cost.**
- **Cloud engine (optional):** a cheap model (Claude Haiku) describes the frame instead.
2. **Your model** (whatever the host runs β Sonnet, Opus, Fable; your choice) reads that text,
decides, and calls `click`. It never pays the image-token tax unless it explicitly asks for a
`screenshot` to check fine detail.
The local engine needs **no configuration**. To use the optional cloud engine instead, set:
```jsonc
"env": {
"ANTHROPIC_API_KEY": "sk-ant-...", // key for the cheap recognition model
"GRASP_VISION_MODEL": "claude-haiku-4-5" // optional; this is the default
}
```
With a key, `engine:"auto"` prefers the cloud model; without one it falls back to local. You can
always force either with `engine:"local"` / `engine:"api"`.
## Reliability: the server survives crashes
An MCP host launches a stdio server **once**. If that process dies mid-session β a native fault, an
OOM, a stray error β the host does **not** relaunch it, and every Grasp tool silently disappears
until you run `/mcp` β Reconnect by hand.
Grasp fixes this with a **supervisor** (`src/supervisor.js`) that the host launches instead of the
server directly. It:
1. spawns the real server as a child and proxies JSON-RPC both ways;
2. records the `initialize` handshake the host sent;
3. if the child ever exits, **respawns it and replays that handshake**, so the new process is in the
exact MCP state the host believes it is β the tools stay live with no reconnect;
4. answers any in-flight request with a clear error while the child is down, so a call fails fast
instead of hanging, and has a restart back-off so a hard-broken build can't spin forever.
The server itself also guards `uncaughtException`/`unhandledRejection` and keeps serving. This is
covered by an automated test (`npm run test:supervisor`) that kills the underlying server mid-session
and asserts the same client keeps working.
## Operating rules (baked in)
Grasp ships MCP **`instructions`** that the host injects into the model's context, so any Claude
driving Grasp follows two hard rules:
- **Token economy** β see with `describe_screen` first; only reach for `screenshot` when the text
map isn't enough.
- **Finish the job** β carry a task through to completion. If a GUI path fails (some apps like Steam
ignore synthetic clicks), switch to `run_command`, a config/registry edit, or keyboard shortcuts β
there is almost always a reliable non-GUI path. Stop early only for destructive actions or things
that genuinely need the user.
## How the coordinate model works
1. Claude calls **`screenshot`**. Grasp captures the screen, resizes the longest edge to 1280 px, and overlays a grid labelled every 100 px.
2. Claude reads a coordinate straight off the grid (e.g. "the button is near `xβ540, yβ300`").
3. Claude calls **`click`** with those numbers. Grasp maps them back through the resize + crop + DPI transform to the exact physical pixel and clicks there.
You never do the math β pass coordinates in the same space you see them. (Advanced: pass `coord_space: "screen"` to use raw physical pixels instead.)
**Multi-monitor.** Pass `monitor: <id>` (ids come from `get_screen_info`) to `screenshot` or
`describe_screen` to work on a secondary display. Grasp folds that monitor's origin into the
transform, so a coordinate read off a secondary-monitor capture still maps to the correct absolute
pixel. The `local` `describe_screen` engine reports screen pixels directly β click them back with
`coord_space:"screen"`.
## Requirements
- **Windows 10/11, x64.** The mouse/keyboard backend ships as a prebuilt native binary for `win32-x64`. Screen capture, window enumeration and image processing are cross-platform, but Grasp is Windows-first today.
- **Node.js β₯ 18.**
- An MCP host β **Claude Code**, **Claude Desktop**, or anything that speaks MCP over stdio.
## Install
```bash
git clone https://github.com/Jamshed7470/grasp-mcp.git
cd grasp-mcp
npm install
npm run doctor # verify Grasp can see and control this machine
```
`npm run doctor` should end with **β
Grasp is ready β Claude has eyes and hands.**
### Add to Claude Code
From the repo directory:
```bash
claude mcp add grasp -- node "%CD%\src\supervisor.js"
```
β¦or with an absolute path from anywhere:
```bash
claude mcp add grasp -- node "C:\path\to\grasp-mcp\src\supervisor.js"
```
> Point the host at **`supervisor.js`**, not `index.js` β that's what keeps the tools alive across a
> crash (see [Reliability](#reliability-the-server-survives-crashes)). Running `index.js` directly
> still works and is handy for debugging; you just lose auto-recovery.
Then just ask Claude, e.g. *"take a screenshot and open Settings for me."*
### Add to Claude Desktop
Edit `claude_desktop_config.json`
(`%APPDATA%\Claude\claude_desktop_config.json`) and add:
```json
{
"mcpServers": {
"grasp": {
"command": "node",
"args": ["C:\\path\\to\\grasp-mcp\\src\\supervisor.js"]
}
}
}
```
Restart Claude Desktop. Grasp's tools appear under the π¨ menu.
## Usage examples
Once installed, drive it in plain language β Claude picks the tools:
- *"Open Notepad, type today's date, and save it to the Desktop."*
- *"Find the Wi-Fi icon in the tray and tell me which network I'm on."* (screenshot β read)
- *"Update all my winget packages."* (`run_command` with `elevated: true`)
- *"Scroll the page down and click the first search result."*
### What a real session looks like
A verified live run β "open Notepad and write a note", GUI only, like a human at the desk:
```text
1. screenshot β sees the desktop, finds the Start button on the taskbar
2. click(505, 713) β opens the Start menu (grid coordinate read off the shot)
3. type_text("notepad") β search field auto-focuses, query typed
4. screenshot β sees "Notepad β App" as the best match
5. click(540, 349) β launches Notepad
6. get_active_window β confirms: "Untitled - Notepad" (Notepad.exe) is focused
7. press_key("ctrl+n") β fresh tab, never touching existing unsaved tabs
8. type_text("Meeting notes β¦") β full text lands instantly via the clipboard path
9. screenshot(mode=active_window) β verifies the text is really in the editor before saving
```
Each look β decide β act β look-again cycle is exactly how Grasp is meant to be driven: never
click blind, always re-check coordinates against the most recent capture.
## Configuration
| Environment variable | Purpose |
|---|---|
| `ANTHROPIC_API_KEY` / `GRASP_API_KEY` | Optional. Key for the cloud (`api`) engine of `describe_screen`. Not needed for the default local engine. |
| `GRASP_VISION_MODEL` | Cloud model id for `describe_screen` `api` engine (default `claude-haiku-4-5`). |
| `GRASP_ENIGO_PATH` | Absolute path to the `node-enigo` `.node` binary, if it isn't in the bundled `native/` folder. |
| `GRASP_POWERSHELL` | Path to the PowerShell executable used by the local vision engine (default `powershell.exe`). |
Per-tool options (image size, grid on/off, JPEG quality, command timeout, working directory, elevation) are passed as tool arguments β see each tool's description in the MCP schema.
## β οΈ Security
Grasp gives an AI model **real control of your computer** β the same reach you have. Treat it accordingly.
- **You are always in the loop.** MCP hosts ask before each tool call by default. Keep it that way for anything destructive.
- **Close sensitive windows** (password managers, private messages) before running an agent β whatever is on screen is what Claude sees, and screenshots may briefly hold that content.
- **Elevated commands need your hand.** `elevated: true` triggers a Windows UAC prompt you must approve; UAC runs on a secure desktop that cannot be automated. If you dismiss it, Grasp reports the command as **not run** β never a silent success.
- **Never commit screenshots.** This repo's `.gitignore` blocks `*.png`/`*.jpg` for exactly this reason.
- Run Grasp only on machines and tasks you're comfortable handing to an assistant.
## Troubleshooting
| Symptom | Fix |
|---|---|
| `enigo NOT loaded β hands disabled` | The native binary wasn't found. Confirm `native/node-enigo-win32-x64.node` exists, or set `GRASP_ENIGO_PATH`. You're likely not on Windows x64. |
| `node-screenshots NOT loaded` | Run `npm install` (native module needs its prebuild). |
| Clicks land in the wrong place | Take a fresh `screenshot` right before clicking β coordinates are relative to the **most recent** capture. |
| `run_command` can't find `cmd`/`ping`/`net` | Grasp already restores `PATH`/`PATHEXT` for hosts that strip the environment; update to the latest version if you see this. |
Run `npm run doctor` any time to re-check every backend.
## Known limitations
- **Windows-first.** The input backend ships as a prebuilt `win32-x64` binary. macOS/Linux
contributions welcome (see [Development](#development)).
- **Some apps ignore synthetic input β by design, and Grasp won't fight it.** Steam's UI and games
with anti-cheat deliberately drop injected clicks; that's a security control, and bypassing it
isn't something Grasp attempts. Use the reliable non-GUI path instead β `run_command`, config-file
or registry edits, keyboard shortcuts. The baked-in operating rules already tell the model to
switch paths rather than give up.
- **UAC cannot be automated β by design.** Elevation prompts render on a secure desktop that no
synthetic input can reach. A human must click Yes; Grasp reports honestly when they don't.
- **`describe_screen` local engine covers standard apps well.** UI Automation is rich for
Win32/WinUI/WPF/Electron; it can be sparse for custom-drawn surfaces (some games, canvas apps),
where OCR fills in text. For pixel-exact detail, `screenshot` is always there. The optional cloud
engine needs an API key (subscription OAuth doesn't apply); the local engine needs neither key nor
network.
- **`type_text` replaces clipboard *text* only briefly.** Your clipboard text is restored after the
paste; images or copied files in the clipboard are not preserved.
- **One virtual desktop at a time.** Grasp sees and drives the active desktop session; it does not
reach across RDP sessions, secure desktops or lock screens.
## Changelog
### 1.2.0
- **`describe_screen` works with no API key.** New **local** engine (now the default): Windows UI
Automation lists real interactive elements with names + coordinates, and the built-in Windows OCR
reads on-screen text β no key, no network. The cloud (Haiku) engine is still available via
`engine:"api"`; `engine:"auto"` picks based on whether a key is set.
- **Crash-proof supervisor.** The host now launches `src/supervisor.js`, which respawns the server
and replays the MCP handshake if it ever dies β the session keeps its tools without a manual
`/mcp` β Reconnect. In-flight calls fail fast instead of hanging; the server also guards uncaught
errors and keeps serving. Covered by `npm run test:supervisor`.
- **Multi-monitor.** `screenshot` and `describe_screen` accept a `monitor` id; the capture transform
folds in that monitor's origin so coordinates map to the correct absolute pixel on any display.
### 1.1.1
- **Fixed: `type_text` lost or repeated characters on long/fast input.** Raw injected key events
flood real apps β live testing showed 119 chars arriving as 15, and `"OK"` arriving as
`"KKKKKKK"` (lost key-up β autorepeat). Text now goes through **clipboard + Ctrl+V**: instant at
any length, Unicode-safe, keyboard-layout independent, previous clipboard text restored. Direct
injection kept as automatic fallback.
### 1.1.0
- **Two-model token economy:** new `describe_screen` tool β a cheap model (Claude Haiku) turns the
frame into a text map of elements + coordinates so the main model doesn't pay the image-token tax
on every look.
- **Operating rules shipped as MCP instructions:** token economy + finish-the-job (switch to
`run_command`/config edits when a GUI path fails; never stop at the first obstacle).
### 1.0.0
- Initial release: 13 tools (eyes / hands / system), grid-overlay coordinate model with DPI-correct
click mapping, PowerShell runner with honest UAC elevation, doctor + end-to-end MCP tests.
## How it's built
```
src/
supervisor.js crash-proof stdio proxy β respawns index.js + replays the handshake (host entry)
index.js MCP server β registers the 14 tools + operating instructions over stdio
native.js screen capture + mouse/keyboard (node-enigo, node-screenshots, get-windows)
image-processor.js sharp pipeline: crop β resize β coordinate-grid overlay
uia.js local key-free vision β UI Automation elements + Windows OCR β text map
vision.js optional cloud recognition β screenshot β text map (Anthropic API)
shell.js PowerShell runner with UTF-8 output + UAC elevation
doctor.js standalone backend health check
scripts/
uia-dump.ps1 UI Automation element dump (interactive elements + screen coordinates)
ocr-screen.ps1 Windows.Media.Ocr text recognition of a captured frame
native/
node-enigo-win32-x64.node prebuilt Rust input backend
test/
mcp-test.js end-to-end test over real MCP stdio (read-only + safe tools)
supervisor-test.js kills the server mid-session, asserts the supervisor keeps tools alive
live-notepad.js opt-in live test that types into a throwaway file and reads it back
```
**Backends:** [`node-enigo`](https://github.com/) (Rust, mouse/keyboard) Β· [`node-screenshots`](https://www.npmjs.com/package/node-screenshots) (Rust, capture) Β· [`get-windows`](https://www.npmjs.com/package/get-windows) Β· [`sharp`](https://sharp.pixelplumbing.com) Β· [`@modelcontextprotocol/sdk`](https://github.com/modelcontextprotocol).
## Development
```bash
npm run doctor # health check, no server
npm test # 14 end-to-end assertions over MCP stdio
npm run test:supervisor # crash-recovery test: kills the server, checks tools stay alive
node test/live-notepad.js # opt-in: drives the real desktop, cleans up after itself
```
Contributions welcome β especially a macOS/Linux input backend to make Grasp truly cross-platform.
## License
[MIT](LICENSE). Grasp bundles a prebuilt `node-enigo` binary and depends on `node-screenshots`, both built on the Rust [`enigo`](https://github.com/enigo-rs/enigo) / `xcap` ecosystem.
<div align="center">
<sub>Grasp β <i>to grasp</i> is both to see and to hold. Built for Claude.</sub>
</div>
This server cannot be deployed
Maintenance
ActivitySlowing
ResponsivenessNo issues