ClawdCursor
by AmrDab
README.md
<h1 align="center">Clawd Cursor</h1>
<p align="center">
<strong>Safe desktop control for any AI agent.</strong> Compiles the screen into a UI map and acts on elements by stable id (screenshot/vision only as a last resort),<br>
<strong>verifies its own actions</strong>, and gates everything through one safety checkpoint. Local · cross-OS · any model.
</p>
<p align="center">
<a href="LICENSE"><img src="https://img.shields.io/github/license/AmrDab/clawdcursor?color=a855f7" alt="MIT license"></a>
<a href="https://github.com/AmrDab/clawdcursor/releases/latest"><img src="https://img.shields.io/github/v/release/AmrDab/clawdcursor?color=22c55e&label=release" alt="Latest release"></a>
<a href="https://www.npmjs.com/package/clawdcursor"><img src="https://img.shields.io/npm/v/clawdcursor?color=cb3837&logo=npm" alt="npm"></a>
<img src="https://img.shields.io/badge/node-%E2%89%A520-339933?logo=nodedotjs&logoColor=white" alt="Node 20+">
<img src="https://img.shields.io/badge/platform-win%20%7C%20mac%20%7C%20linux-0ea5e9" alt="Cross-platform">
<a href="https://glama.ai/mcp/servers/AmrDab/clawdcursor"><img src="https://glama.ai/mcp/servers/AmrDab/clawdcursor/badges/score.svg" alt="clawdcursor MCP server on Glama" height="20"></a>
<a href="https://github.com/AmrDab/clawdcursor/actions/workflows/cross-platform.yml"><img src="https://img.shields.io/github/actions/workflow/status/AmrDab/clawdcursor/cross-platform.yml?branch=main&label=tests" alt="Tests"></a>
<a href="https://discord.gg/hW29nrEZ8G"><img src="https://img.shields.io/badge/Discord-join-5865F2?logo=discord&logoColor=white" alt="Discord"></a>
</p>
<p align="center">
<a href="#install">Quickstart</a> ·
<a href="#whats-new">What's new</a> ·
<a href="#why-its-different">Why it's different</a> ·
<a href="#the-engine">The engine</a> ·
<a href="#how-it-works">How it works</a> ·
<a href="#the-toolbox">Tools</a> ·
<a href="#platform-support">Platforms</a> ·
<a href="CHANGELOG.md">Changelog</a>
</p>
<!-- ───────────────────────────────────────────────────────────────────────────
DEMO: the single highest-leverage thing on this page for a *visual* tool.
Record a 15–25s screen capture of clawdcursor driving a real app end-to-end
(open app → find a field by name → type → verify), export to docs/demo.gif,
and uncomment the block below (it will sit right under the hero).
<p align="center">
<img src="docs/demo.gif" width="760" alt="clawdcursor driving a desktop app">
<br><em>Opening an app, finding a field by name, typing, and verifying the result — accessibility-first, locally.</em>
</p>
─────────────────────────────────────────────────────────────────────────── -->
---
## What's new
**1.5.27** — **Results you can trust, and logs that say what happened.**
- Ticks, dropdowns and expanders read and change the same way on Windows, macOS and Linux; `toggle` reports the state it actually reached, and actions by name aim at the control, not its label.
- A stop always answers "stopped by the user" first, and also blocks hover, clipboard writes, page script and tab switches.
- Typing goes only to the window the agent was working in, dialogs like Save As included.
- Linux reads deep web pages, and a search that runs out of time or elements says so instead of "not found".
- Logs in `~/.clawdcursor/logs/` show every call's timing and outcome, slow internal steps, why the server exited, and who stopped it, with no typed text, error text or window titles.
**1.5.26** — **The agent asks you on screen, and the pill says what it is doing.**
- Security (Windows, high): text the agent typed or copied could run as PowerShell code ([GHSA-9ppf-38v8-h97v](https://github.com/AmrDab/clawdcursor/security/advisories/GHSA-9ppf-38v8-h97v)). Fixed; upgrade.
- The pause prompt is a card: the agent can offer 2-6 choices to click, or a text box, and gets your exact answer back. A choice is never a safety approval.
- A redesigned pill: what the agent is doing right now ("Typing in Edge"), how long it has been running, and a Stop button.
- A field below the fold is scrolled into view instead of refused (verified on Windows; the macOS and Linux calls are in but not yet run on a real desktop).
- macOS: when a password field holds secure input, `window focus` names the app holding it instead of failing vaguely.
**1.5.25** — **A stop really stops, and keys only go to the window the agent was working in.**
- Double-click the pill and every clawdcursor process stops: a running wait ends at once, the rest of a batch never runs, and it stays stopped (even across restarts) until you click Resume.
- Every key, paste and click first checks that the agent's window is still in front. If you switched apps, nothing is sent and the agent is told so.
- Safety gate: key combos are blocked however their modifiers are spelled, `shortcuts_execute` and compound actions are gated like the tools they run, and every element an invoke may act on is checked.
- `--no-vision` now really keeps screenshots from every model; the daemon is loopback-only (use an SSH tunnel for remote), and `server.port` in config works.
- macOS: the Screen Recording check works without the helper built, and `doctor` asks the same question screenshots do.
**1.5.24** — **The AI never touches your pill, and failures stop looking like results.**
- While a pause waits for you, nothing reads, clicks or types. The pill and pause prompt are never a target, even from another clawdcursor process.
- On macOS, a screenshot without Screen Recording permission is an error, not a wallpaper image, and open popovers are readable.
- The browser clawdcursor drives comes to the front when it navigates.
- A failed OCR read reports its error instead of "No text detected", and OCR `confidence` is only shown when the engine really has one.
**1.5.23** — **Windows: finding an element by name no longer matches stray letters.** A lone "C" on another page could match "User code 0", and "Don't Save" could match **Save**. A partial match now has to be a real part of the name you asked for.
**1.5.22** — **Your turn is as short as possible.** When an agent needs you (a sign-in, a 2FA code), it asks for that one step and does the rest itself. With `done_when` (what shows on screen once you're done), the prompt turns from blue back to amber the moment you finish, with no Resume click.
**1.5.21**
- **`set_value` never guesses a field.** With no field name or role it used to type into the window's largest text box; it now asks you to name the field.
- **macOS: no screen-recording permission no longer breaks everything.** Only screenshots and OCR fail, and they say how to fix it; windows, keys and accessibility keep working.
- **Cleaner page text on Windows** (no placeholder characters), and a security update to a dependency.
**1.5.20**
- **The on-screen pill no longer blocks the agent's own clicks.** It sits where browsers keep their tabs and address bar; it now steps aside for a moment before every click, drag or scroll, then comes back. Windows, macOS and Linux.
- **Reading a page no longer highlights it on Linux and macOS either.** `copy_all_text` reads the document through accessibility, as on Windows: exact text, nothing selected, your clipboard untouched.
- **Your clipboard is put back after typing** (`smart_type` too), and only once the paste has landed, so a slow app can't paste what you had copied.
- **macOS:** `window focus` focuses the process you asked for, a locked screen says so, sidebar clicks are verified, and a bare modifier key (`shift`) works.
**1.5.19** — **Typing never pastes your own clipboard.** clawdcursor types by pasting; on Windows a clipboard write could fail silently and the paste then sent whatever *you* had copied. Every paste now checks the clipboard first and types key by key when it doesn't hold the text.
**1.5.18**
- **"Something else" on the pause prompt.** When the agent pauses for you, the prompt has a third button between Resume and End task: type an answer instead of doing the step — "use the test account instead", a code, a change of plan — and press Enter. The agent continues from what you wrote. Windows, macOS and Linux; over MCP `pause_for_user` returns `{status:"replied", reply}`.
- **Reading a page no longer highlights it (Windows).** `copy_all_text` reads the document the way a screen reader does: exact text, no Ctrl+A, nothing selected, your clipboard untouched.
- **MCP tool annotations** on every tool, and `window` / `system` now describe every action, its side effects and what it returns.
**1.5.17**
- **macOS: text outside ASCII is typed and pasted as itself** — "café 日本語" no longer comes out as "cafa aaa" or "caf√©".
- **`copy_all_text` never selects an editor or a list** (Ctrl+A there selected a whole document or every email).
**1.5.16**
- **It pauses for you at a roadblock, then carries on.** A sign-in, a 2FA code, a CAPTCHA, an upgrade page: instead of giving up, the agent pauses. The glow around the screen turns **blue** and an on-screen prompt says what it needs ("Sign in to Outlook, then click Resume"). Click **Resume** and it continues where it left off, or **End task** to stop. Your own agent can do the same over MCP with `system({ action: "pause_for_user", message })`; from a terminal, `clawdcursor resume` (or `--end`).
- **The glow and banner on every OS.** macOS and Linux now get the same screen-edge glow and banner as Windows: amber while the agent works, blue while it waits for you. Double-click the banner to stop everything.
- **Blind mode on macOS caught up with Windows.** A native accessibility helper lists windows in 0.3 s (was 11 s) and reads a window's tree in 0.06–0.2 s. Typing is refused while a system dialog owns the keyboard, and the agent is told when another app's dialog took its keys.
- **Sharper blind mode on Windows and Linux.** Classic controls no longer read as blank panes, the open dialog is what gets read, no 10 s stall on buttons that open a modal, the right field is found by name, and focusing a window no longer un-maximizes it.
- **Upgrade walls are a roadblock.** Clicks on `Upgrade`, `Subscribe` or `Start trial` need confirmation.
- **Know what you're running.** `clawdcursor versions` lists every installed copy (global npm, Claude Desktop extension, Claude Code plugin), and the installers say what they updated.
Everything else is in the [changelog](CHANGELOG.md).
---
## What it is
Clawd Cursor is a **local MCP server** that gives any tool-calling agent — Claude Code, Cursor, Windsurf, OpenClaw, the Claude Agent SDK, or your own loop — safe control of the **real desktop**. It clicks, types, reads the screen, opens apps, and drives any GUI the way a human would: native apps, the browser, even a canvas.
Most "let an agent use the computer" tools take a screenshot and feed it to a vision model — slow, expensive, and brittle. Clawd Cursor **compiles the screen into one UI map**: it fuses the accessibility tree and OCR into a confidence-scored set of elements, each tagged with a stable `el_NN` id, and acts on elements by id — not pixel coordinates. Coordinates appear only in the last-resort screenshot/vision tier (live pixels off the current frame), for canvas-only apps or tasks that genuinely need spatial reasoning. The result is cheaper, faster, private, and — uniquely — it **checks that each action actually did what it claimed**.
> **If a human can do it on a screen, your agent can too.** No API, no integration, no problem — only the right sequence of reads, clicks, keys, and waits. Use it as the **last-mile fallback**: native API exists? Use it. CLI? Use it. Clawd Cursor is for the click, the legacy app, the GUI with no public surface.
---
## Why it's different
The desktop-agent space is crowded. The closest **install-and-go** peers are [Windows-MCP](https://github.com/CursorTouch/Windows-MCP) and [Terminator](https://t8r.tech/) (desktop MCP servers); browser-only tools (browser-use, Playwright MCP) are adjacent; and [OmniParser](https://github.com/microsoft/OmniParser) / [UI-TARS](https://github.com/bytedance/UI-TARS-desktop) are vision-centric *parsing approaches* you'd build an agent around, not products you install. Here's the honest comparison across those approaches — what Clawd Cursor does that the popular options don't:
| | **Clawd Cursor** | browser-use | Playwright MCP | OmniParser / UI-TARS | computer-use |
|------------------------------------------|:-----------------------:|:-----------:|:--------------:|:--------------------:|:------------:|
| Any desktop app, not just the web | ✅ | web only | web only | ✅ | ✅ |
| Cross-OS (Windows + macOS + Linux) | ✅ | — | — | varies | sandbox |
| Perception **without a vision model** | ✅ compiled a11y + OCR map | DOM | a11y tree | ❌ vision-centric | ❌ vision |
| **Verifies its own actions** (deviation) | ✅ | — | — | — | — |
| Single safety chokepoint (allow/confirm/block) | ✅ | — | — | — | — |
| Any model / vendor | ✅ | ✅ | not an agent | model-specific | Claude only |
| MCP-native (one config, any host) | ✅ | library | test framework | — | tool-use API |
| Local-only, no cloud required | ✅ | ✅ | ✅ | needs a model | screens → cloud |
Three things here are genuinely rare:
1. **Cheapest-tier-first perception, fully local.** Accessibility tree (free) → OCR (cheap) → screenshot (expensive — the *only* tier that puts pixels in the model's context; "screenshot" and "vision" are the same step). The agent climbs only when it must, so token cost tracks task difficulty — and with a local model, nothing leaves the machine. Vision-centric agents (OmniParser, UI-TARS) need a screenshot in the model for *every* observation.
2. **It verifies.** Pass `expect` on a consequential action and Clawd Cursor re-checks the live screen (with a short settle window for async UIs) and reports a **DEVIATION** instead of a hollow "success." A completed task can't be marked done on evidence that was already true before it acted.
3. **One safety gate.** Every call — from an editor over stdio, an external agent over HTTP, or the built-in loop — routes through a single `safety.evaluate()` chokepoint (allow / confirm / block) before it touches the desktop. The agent cannot bypass it.
Plus: whenever an agent is driving, an on-screen **banner that says what it is doing** ("Typing in Edge") and an amber **glow around the screen edges**, on Windows, macOS and Linux — **press Stop on the banner to stop.** A stop holds until you give control back: the agent's next request comes up as a prompt, and only your **Resume** lets it act again (or run `clawdcursor resume`). Keys and clicks also go only to the window the agent was working in: if you switch apps while it waits, its next input is refused instead of landing in your window. When the agent needs you (a sign-in, a 2FA code), the glow turns blue and the banner says what to do, with **Resume** and **End task**. A human at the machine always knows, and always has a kill switch.
---
## Install
clawdcursor is an MCP server published to npm — install it into **any** MCP-capable
agent (Claude Code, Claude Desktop, Cursor, Windsurf, Zed, OpenAI Codex, or your own
loop) the same way you install any other MCP server.
### Claude Desktop — one-click extension (no terminal)
Download the extension for your OS and open it — Claude Desktop installs it:
| Windows | macOS | Linux |
|---|---|---|
| [clawdcursor-win32.mcpb](https://github.com/AmrDab/clawdcursor/releases/latest/download/clawdcursor-win32.mcpb) | [clawdcursor-darwin.mcpb](https://github.com/AmrDab/clawdcursor/releases/latest/download/clawdcursor-darwin.mcpb) | [clawdcursor-linux.mcpb](https://github.com/AmrDab/clawdcursor/releases/latest/download/clawdcursor-linux.mcpb) |
1. Double-click the file (or **Settings → Extensions → Advanced settings → Install
Extension…** and pick it).
2. Turn on **Allow clawdcursor to control this computer** — that is the one-time
desktop-control consent; until it is on, every tool returns a consent prompt.
3. **macOS:** the first time macOS asks, allow Claude under **Privacy & Security →
Accessibility** and **Screen Recording**.
No Node.js, npm or JSON editing needed — Claude Desktop runs the bundled server
itself. Everything below is for other agents, or for Claude Desktop without the
extension.
### 1 — Install the engine + grant consent (once)
```bash
npm i -g clawdcursor
clawdcursor consent --accept # one-time desktop-control consent (required)
clawdcursor grant # macOS only — approve Accessibility + Screen Recording
```
> Hosts that support MCP elicitation (e.g. Claude Code, VS Code) also let you skip
> `consent --accept`: the first tool call asks **"Allow clawdcursor to control this
> computer"** right in the app.
> **Zero-install** also works — swap `clawdcursor` for `npx -y clawdcursor` in any
> snippet below and npx fetches it on demand. A **global install is recommended**
> anyway: it's pinnable and inspectable on disk (safer for a tool with full desktop
> control than auto-fetching `latest` every run), and it's the path on which the macOS
> native helper builds at install time. Requires **Node.js 20+**.
> **Per-OS prerequisites.** **Windows** installs clean — `sharp` and `@nut-tree-fork/nut-js`
> ship prebuilt binaries, so no C++/Python build tools are needed. **macOS** needs Xcode
> Command Line Tools (`xcode-select --install`) for screenshots / vision; core
> accessibility-driven control still works without them. **Linux** needs a few system
> packages npm can't install: `tesseract-ocr` (OCR), `python3-gi` + `gir1.2-atspi-2.0`
> (accessibility tree), and — on Wayland — `ydotool` (synthetic input).
### 2 — Add it to your agent (pick your host)
**One click** (Cursor, VS Code):
[](https://cursor.com/install-mcp?name=clawdcursor&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsImNsYXdkY3Vyc29yIiwibWNwIiwiLS1jb21wYWN0Il19)
[](https://vscode.dev/redirect/mcp/install?name=clawdcursor&config=%7B%22name%22%3A%22clawdcursor%22%2C%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22clawdcursor%22%2C%22mcp%22%2C%22--compact%22%5D%7D)
Every host runs the same command — `clawdcursor mcp --compact`, or
`npx -y clawdcursor mcp --compact` with no global install. Only the config shape
differs between hosts:
**Claude Code**
```bash
claude mcp add clawdcursor -s user -- clawdcursor mcp --compact
```
**Cursor · Windsurf · Claude Desktop · Cline · Gemini CLI** — the common `mcpServers` shape:
```json
{
"mcpServers": {
"clawdcursor": { "command": "clawdcursor", "args": ["mcp", "--compact"] }
}
}
```
Claude Desktop reads `%APPDATA%\Claude\claude_desktop_config.json` (Windows) or
`~/Library/Application Support/Claude/claude_desktop_config.json` (macOS).
Gemini CLI reads `~/.gemini/settings.json`.
**VS Code (Copilot)** — `.vscode/mcp.json` uses `servers`, **not** `mcpServers`:
```json
{
"servers": {
"clawdcursor": { "command": "clawdcursor", "args": ["mcp", "--compact"] }
}
}
```
**Zed** — `context_servers` in `settings.json`:
```json
{
"context_servers": {
"clawdcursor": { "command": "clawdcursor", "args": ["mcp", "--compact"] }
}
}
```
**OpenAI Codex** — `~/.codex/config.toml`. The first launch can take a few seconds,
so raise the 10 s default startup timeout:
```toml
[mcp_servers.clawdcursor]
command = "clawdcursor"
args = ["mcp", "--compact"]
startup_timeout_sec = 30
```
**opencode** — `opencode.json` uses `mcp`, and `command` is an array:
```json
{
"mcp": {
"clawdcursor": { "type": "local", "command": ["clawdcursor", "mcp", "--compact"] }
}
}
```
**Any other MCP host** (Goose, Continue, …) — register a **stdio** server with command
`clawdcursor` and args `mcp --compact`.
> **Windows:** some hosts start the server without a shell, and then can't find
> `clawdcursor` or `npx` (both are `.cmd` shims). If the server won't start, wrap it in
> `cmd /c` — e.g. `"command": "cmd", "args": ["/c", "clawdcursor", "mcp", "--compact"]`,
> or `claude mcp add clawdcursor -s user -- cmd /c clawdcursor mcp --compact`.
That's the whole setup. Ask your agent: *"open Outlook and reply to the latest email
from Sarah."*
### Or: one-command plugin (Claude Code)
Skip the manual config — this repo ships a plugin that registers the tools **and**
bundles the usage skill in one step. It resolves the package's `bin` (never a
hard-coded `dist/` path), so an upgrade can't break it:
```bash
claude plugin marketplace add AmrDab/clawdcursor
claude plugin install clawdcursor@clawdcursor
```
<details>
<summary>One-line installers (npm i -g, plus Node-version, PATH, and consent checks)</summary>
```powershell
# Windows (PowerShell)
powershell -c "irm https://clawdcursor.com/install.ps1 | iex"
```
```bash
# macOS / Linux
curl -fsSL https://clawdcursor.com/install.sh | bash
```
</details>
> **Notes.** You never run `clawdcursor mcp` yourself — the host spawns it over stdio
> on demand. `clawdcursor doctor` is **not** part of MCP setup; it only configures the
> built-in LLM for the autonomous `agent` daemon. On macOS, **Accessibility is
> required** (primary control path); **Screen Recording is optional** (vision fallback
> only). For editor permission allowlists, use the server-level wildcard
> `mcp__clawdcursor` rather than per-tool entries — it survives tool renames.
---
## The engine
The perception + verification core (the **UI State Compiler**, since v1.5.0):
- **`compile_ui`** fuses the accessibility tree and OCR into one confidence-scored map of the screen, every element tagged with a stable `el_NN` id. Act on an element by `{element_id, snapshot_id}` instead of pixels — near-free in tokens, and it survives DPI, resize, and layout shifts. `find_button` / `find_field` locate a target by meaning and hand you the id.
- **Reactive verification.** `expect` on an action → Clawd Cursor confirms the outcome on the live screen and returns a **DEVIATION** when the UI didn't obey.
- **Cross-platform parity.** The compiler, secure-field redaction, and coordinate handling run on Windows, macOS, and Linux; the external-agent (MCP) surface resolves `el_NN` refs through the safety gate and discloses when it attached to your existing browser.
> Set-of-Mark-style element IDs and a11y/OCR fusion aren't new ideas on their own — what's rare is doing them **locally, a11y-first (no vision model required), with a built-in verification gate and one safety chokepoint**, across three operating systems, behind a single MCP config.
See the [changelog](CHANGELOG.md) for the full release history, or the [latest release](https://github.com/AmrDab/clawdcursor/releases/latest).
---
## How it works
**Where the brain lives** decides how you run it. Both modes can run side-by-side.
> **Deprecated in 1.5.27:** the built-in LLM loop (`task`, `submit_task`, `system delegate` and the `clawdcursor agent` model set up by `doctor`) will be removed in 1.6, and clawdcursor will no longer call a model itself. Drive the MCP tools from your own agent instead (Claude Code, Codex, Cursor, …). The daemon's HTTP MCP transport stays for remote use.
| Brain lives… | Mode | Command | What you call |
|---|---|---|---|
| In your editor (Claude Code, Cursor, Windsurf, Zed) | Direct tools | `clawdcursor mcp` | Each tool, via stdio MCP |
| In a headless agent with its own LLM (OpenClaw, Agent SDK, your loop) | Direct tools | `clawdcursor agent --no-llm` | Same, over HTTP MCP |
| Inside Clawd Cursor itself (scheduled / "submit and walk away") — *deprecated, removed in 1.6* | Thin agent loop | `clawdcursor agent` + `doctor`-configured LLM | `task` / `submit_task` |
| External brain that delegates sub-tasks to the built-in loop — *deprecated, removed in 1.6* | Direct + delegation | `clawdcursor agent` + your client | `task({instruction:…})` to hand off |
### The loop
Read the a11y tree (cheap) → act on named targets → verify from fresh observations → escalate perception only when needed (OCR → screenshot, the one tier that sends pixels to the model). Sparse a11y tree? `system.detect_webview` switches Electron/WebView2 apps to `browser.*` over CDP. Canvas-only (Paint, Figma, games)? Screenshot + coordinate click.
```mermaid
flowchart TB
task["User task"] --> loop["Agent LLM loop<br/>plans · chooses tools · verifies"]
loop --> observe{"Cheapest observation<br/>that answers the question"}
observe -- "obs·a11y — free" --> a11y["A11y tree<br/>(structured text + el_NN handles)"]
observe -- "obs·ocr — cheap" --> ocr["OCR (OS-level, no vision LLM)"]
observe -- "obs·dom — medium" --> dom["Browser DOM (CDP)"]
observe -- "obs·vision — expensive" --> vision["Screenshot (image into context)"]
a11y --> act
ocr --> act
dom --> act
vision --> act
act["Act<br/>click/type/key/drag · invoke/set_value · open_app · batch"] --> safety
safety["Single safety gate<br/>safety.evaluate() → allow / confirm / block"] -- allowed --> tools["Tool registry<br/>100+ granular + 7 compound"]
safety -- needs user --> confirm["Human confirmation"] --> tools
safety -- denied --> blocked["blocked"]
tools --> desktop["Real desktop"]
desktop --> verify{"expect → does state match?"}
verify -- pass --> done["done"]
verify -- "DEVIATION" --> loop
classDef agentNode fill:#dbeafe,stroke:#2563eb,color:#0f172a;
classDef gate fill:#ede9fe,stroke:#7c3aed,color:#0f172a;
classDef obsNode fill:#fef9c3,stroke:#ca8a04,color:#0f172a;
classDef actNode fill:#ffedd5,stroke:#ea580c,color:#0f172a;
classDef stop fill:#fee2e2,stroke:#dc2626,color:#0f172a;
class loop,verify agentNode;
class safety,confirm,tools gate;
class observe,a11y,ocr,dom,vision obsNode;
class act actNode;
class blocked stop;
```
**`batch` for deterministic stretches.** When the next N steps are known, collapse them into one call — each step still routes through the safety gate; on any guard miss or error the batch halts with a per-step trace.
**Task delegation (deprecated, removed in 1.6).** With an LLM configured on the daemon, an external agent can hand off at any point: `task({"instruction":"…"})`. The built-in loop takes the wheel and reports back — offload grunt work to a cheaper model without burning your own context.
---
## The toolbox
Two catalogs ship side-by-side. The **toolbox** is 7 compound tools, each with an `action` enum covering ~10–20 verbs (about 7,000 tokens for the whole catalog, schemas included — about a third of the granular surface's ~20,000; the `computer_20250124` shape editor hosts already know). The **granular** surface is the 100+ underlying primitives, one schema per verb (for runtimes that need top-level tools, or for debugging). Both run through the same `safety.evaluate()` chokepoint; the full catalog is always visible via MCP `tools/list`.
| Toolbox | Actions |
|---|---|
| `computer` | `screenshot`, `click`, `double_click`, `right_click`, `triple_click`, `hover`, `move`, `scroll`, `scroll_horizontal`, `drag`, `drag_path`, `type`, `key`, `wait` |
| `accessibility` | `read_tree`, `find`, `get_element`, `focused`, `invoke`, `focus`, `set_value`, `get_value`, `expand`, `collapse`, `toggle`, `select`, `state`, `list_children`, `wait_for`, `compile_ui`, `find_button`, `find_field`, `smart_click`, `smart_type`, `smart_read` |
| `window` | `list`, `active`, `focus`, `maximize`, `minimize`, `restore`, `close`, `resize`, `list_displays`, `screen_size`, `open_app`, `open_file`, `open_url`, `switch_tab`, `navigate` |
| `system` | `copy_all_text`, `ocr`, `clipboard_read`, `clipboard_write`, `shortcuts_list`, `shortcuts_run`, `undo`, `pause_for_user`, `report`, `build_uri`, `open_uri`, `open_app`, `open_file`, `open_url`, `detect_webview`, `relaunch_with_cdp`, `delegate`, `system_time`, `system_prompt` |
| `browser` | `connect`, `page_context`, `read_text`, `click`, `type`, `select_option`, `evaluate`, `wait_for`, `list_tabs`, `switch_tab`, `scroll` |
| `task` | `run` (default; **bounded-sync** — waits up to `timeout`s, returns `{status:"running"}` + progress if longer, re-call to keep waiting), `status`, `abort`, `resume` (continue a task paused for the user). Delegates to the built-in loop. Requires `clawdcursor agent` with an LLM. Deprecated, removed in 1.6. |
| `batch` | `{steps:[…]}` — collapse N calls into one round-trip; each step `{name, arguments, expect?}`, re-perceived and safety-gated, halts with a trace on any miss. |
```js
computer({ action: "key", combo: "mod+s" }) // Cmd+S / Ctrl+S, resolved per-OS
accessibility({ action: "invoke", name: "Send" }) // click by name, not pixels
window({ action: "open_app", name: "Outlook" })
```
---
## Cheapest-tier-first perception
Every observation has a cost. Start at the cheapest rung that works; climb only when it fails. The live log (`CLAWD_LOG=pretty`, default on a TTY) shows the ladder in real time via per-call badges.
| Tier | Badge | Cost | Source | When |
|---|---|---|---|---|
| **T1** structured | `obs·a11y` | ~free | `accessibility.*`, `window.*`, `browser.read_text`, clipboard | Default. Text + bounds, no image, no vision LLM. |
| **T2** OCR | `obs·ocr` | cheap | `system.ocr`, `smart_read` / `smart_click` / `smart_type` | A11y tree empty/sparse. OS-level text out, no image bytes. |
| **T3** DOM | `obs·dom` | medium | `browser.read_text` / `page_context` (CDP) | WebView / Electron / Chrome content. |
| **T4** screenshot (vision) | `obs·vision` | expensive | `computer.screenshot` | The only tier that puts pixels in the model's context. Canvas-only apps or spatial reasoning. Last resort. |
Acting tools log `act`. Watching `obs·a11y → act → obs·a11y` on a normal turn — and the rare climb to `obs·vision` — is the whole efficiency model, visible.
---
## Transports
One protocol — **MCP** — two transports, same catalog and JSON-RPC envelope. Both stateless; no session handshake.
| Transport | When | Client config |
|---|---|---|
| **stdio MCP** | Editor hosts. Tools appear on demand — no daemon. | `{"command":"clawdcursor","args":["mcp","--compact"]}` |
| **HTTP MCP** | Headless agents, daemons, orchestration, Agent SDK. POST JSON-RPC to `http://127.0.0.1:3847/mcp`. | Run `clawdcursor agent`. Bearer token at `~/.clawdcursor/token`. |
```bash
# HTTP MCP — list tools
# (the Accept header is required — the MCP spec's Streamable HTTP transport
# rejects requests that don't accept both JSON and SSE with a 406)
curl -s -X POST http://127.0.0.1:3847/mcp \
-H "Authorization: Bearer $(cat ~/.clawdcursor/token)" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```
---
## Platform support
Platform code lives behind a single `PlatformAdapter` interface (`src/platform/{windows,macos,linux}.ts` + `wayland-backend.ts`). Business logic never reads `process.platform`.
| Platform | UI Automation | OCR | Browser (CDP) | Input |
|---|---|---|---|---|
| **Windows** 10/11 (x64 / ARM64) | UIA via PowerShell bridge | `Windows.Media.Ocr` | Chrome / Edge | nut-js |
| **macOS** 12+ (Intel / Apple Silicon) | Native AX helper (Swift), JXA fallback | Apple Vision | Chrome / Edge | nut-js + System Events |
| **Linux** X11 | AT-SPI via `python3-gi` | Tesseract | Chrome / Edge | nut-js |
| **Linux** Wayland | AT-SPI via `python3-gi` | Tesseract | Chrome / Edge | `ydotool` / `wtype` |
- **Windows** — no setup; the PowerShell bridge spawns on demand.
- **macOS** — first run needs Accessibility (required) + Screen Recording (optional); `clawdcursor grant` walks the dialogs. Retina/HiDPI handled in-adapter — **don't pre-scale coordinates.**
- **Linux X11** — `apt install tesseract-ocr python3-gi gir1.2-atspi-2.0`.
- **Linux Wayland** — same, plus `ydotool` + `ydotoold` (preferred) or `wtype` (keyboard only).
---
## Safety & privacy
| Tier | Actions | Behavior |
|---|---|---|
| **Allow** | Reading, opening apps, navigation, typing into non-sensitive fields, minimize | Executes immediately |
| **Confirm** | Sends, deletes, purchases, transfers, close-window/quit-app & show-desktop key combos, sensitive apps | Pauses for approval (`batch({allowConfirm:true})` to authorize) |
| **Block** | `Ctrl+Alt+Del`, lock / log-out / force-quit / shutdown key sequences | Refused outright (no path) |
- **Pauses instead of pushing through.** At a sign-in, 2FA, CAPTCHA or plan-limit page the agent hands the desktop to you (blue glow, Resume / End task) instead of guessing. A pause times out after 15 minutes (`CLAWD_PAUSE_TIMEOUT_MIN`); `CLAWD_NO_PAUSE=1` turns pausing off.
- **Network isolation.** Binds to `127.0.0.1` only; there is no setting that opens it to the network. Verify: `netstat -an | findstr 3847` (Windows) / `| grep 3847` (Unix). To drive it from another machine, tunnel the port over SSH: `ssh -L 3847:127.0.0.1:3847 user@host`, then use `http://127.0.0.1:3847/mcp` on your side.
- **Bearer-token auth** on every HTTP request (`~/.clawdcursor/token`).
- **Sensitive-app policy.** Email, banking, password managers, private messaging auto-elevate to Confirm.
- **No telemetry by default.** Nothing phones home. Screenshots stay in RAM; with a local model nothing leaves the machine; with a cloud provider, screenshots go only to the endpoint you configured. `clawdcursor report` is opt-in and previews exactly what it sends.
- **No screenshots to any model, if you want.** `clawdcursor agent --no-vision` (or `CLAWD_DISABLE_VISION=1`, which also covers stdio `clawdcursor mcp`) keeps every image from the built-in agent and from MCP clients; they work from the accessibility tree and OCR text.
- **Prompt-injection defense.** Screen text is returned inside `<untrusted-screen-content>` tags — data, never instructions.
- **Log privacy.** Logs redact password-field values (`AXSecureTextField`, UIA `IsPassword=true`).
See [SECURITY.md](SECURITY.md) for private vulnerability reporting.
---
## Architecture
| Directory | What lives here |
|---|---|
| `src/core/` | Thin agent loop (`runAgent`), sense layer (a11y / snapshot / fingerprint / UI compiler), reactive verification, focus guard, safety gate. |
| `src/tools/` | 100+ granular tools + 7 compound aggregators + `batch`, playbooks, registry, dispatch. |
| `src/platform/` | `PlatformAdapter` + Windows / macOS / Linux / Wayland, OCR engine, CDP driver, URI handler. |
| `src/llm/` | Provider clients (Claude, GPT, Gemini, Llama, Kimi, Ollama, …), credentials, model config. |
| `src/surface/` | CLI, MCP server (stdio + HTTP), dashboard, doctor, onboarding, control banner. |
The `PlatformAdapter` is the only thing platform code talks to; `safety.evaluate()` is the only way tools execute. Those two seams are the whole point.
---
## CLI
For humans diagnosing an install. Agents connect via MCP.
```
clawdcursor consent Manage desktop-control consent (--accept / --revoke / --status)
clawdcursor grant Grant macOS permissions (interactive, macOS only)
clawdcursor doctor Configure the AI provider for `agent` mode (+ diagnostics)
clawdcursor status Readiness check (consent, permissions, AI config)
clawdcursor versions Every installed copy and its version (global, Claude Desktop extension, Claude Code plugin)
clawdcursor resume Continue a task paused for you (same as the on-screen Resume); --end to stop it
clawdcursor mcp stdio MCP server — editor hosts spawn this; you don't
clawdcursor agent Daemon: HTTP MCP on :3847, optional built-in thin loop
clawdcursor agent --no-llm Daemon, tool surface only (no built-in brain)
clawdcursor stop Stop every running mode
clawdcursor uninstall Remove all config and data
Options: --port <n> (default 3847) · --compact · --no-banner · --provider <name> · --accept
```
---
## Development
```bash
git clone https://github.com/AmrDab/clawdcursor.git && cd clawdcursor
npm install
npm run build # tsc + postbuild → dist/surface/cli.js
npm test # vitest (1,000+ tests)
npm run lint # eslint
npm link # global `clawdcursor` shim (Admin shell on Windows)
```
Tests run on Node 20 & 22 against Ubuntu, macOS, and Windows in CI, plus a coverage ratchet, a perf tripwire, and an `npm audit` gate.
**Tech:** TypeScript · Node 20+ · nut-js · Playwright · sharp · Express · Model Context Protocol SDK · Zod · commander.
---
## Contributing
PRs welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for the dev loop, branch conventions, and the test matrix every change clears. Bugs and features in [issues](https://github.com/AmrDab/clawdcursor/issues); private security reports via [SECURITY.md](SECURITY.md).
## License
MIT — see [LICENSE](LICENSE).
## Acknowledgments
Built on the Model Context Protocol SDK, nut-js, Playwright, the Anthropic `computer_20250124` tool shape, and the AT-SPI / UIA / AX trees that make app-agnostic GUI automation possible at all.
---
<p align="center">
<a href="https://clawdcursor.com">clawdcursor.com</a> ·
<a href="https://discord.gg/hW29nrEZ8G">Discord</a> ·
<a href="CHANGELOG.md">Changelog</a> ·
<a href="https://www.npmjs.com/package/clawdcursor">npm</a>
</p>
TDQS
A4.4/5.0
Scored across 7 tools
Disambiguation4/5
Tools are cleanly divided by modality—accessibility, computer, browser, window, system—and the descriptions explicitly steer between them (e.g., prefer accessibility for named elements, use computer only for pixel-level control). Minor overlap remains: open_app/open_file/open_url and navigate appear across system, window, and browser, and the deprecated task tool overlaps with direct agent operation.
Naming Consistency5/5
All seven tools are lowercase single-word category nouns (accessibility, system, computer, window, browser, batch, task), a consistent and predictable pattern. Sub-actions are namespaced under each tool's action parameter, keeping top-level naming uniform.
Tool Count5/5
Seven top-level tools is well within the 3–15 sweet spot for a desktop-automation server, with each tool earning its place as a distinct modality. Even counting the deprecated task tool, the surface is not bloated.
Completeness5/5
Covers the full automation lifecycle: perception (accessibility tree, DOM, OCR, screenshot), action (invoke, click, type), window management, clipboard, shortcuts, user handoff, and batching. Deprecated entries (task, delegate) are the only signs of missing or outdated operations; there are no major workflow dead ends.
Maintenance
ActivityActive
ResponsivenessLimited data