# AgentCursor

**Local, free, human-like cursor for AI agents: any Mac app or browser tab, over MCP.**

AgentCursor gives you (and any coding agent or automation script) a **real browser** driven with **visible, convincingly human cursor movement and timing**.

Use it as:
- A powerful MCP tool for Claude, Cursor, Grok, custom agents, etc.
- A realistic E2E / acceptance testing tool that works on actual production sites (human paths + timing are more resilient than robotic Playwright clicks).
- A workflow automation engine for complex multi-step processes (logins, form flows, data entry, admin tasks) with natural hover, move, type, and scroll behavior.
- A debugging / demo automation tool (the cursor is visible so you can watch exactly what the automation did).

All local. All free. MIT licensed. No cloud, no paywalled stealth.

The major browser automation MCPs often make realistic movement a cloud-only feature. AgentCursor brings the realistic cursor to your local machine for agents **and** traditional testing/automation use cases.

> Status: phase 1 (Chrome extension) and phase 2 (macOS OS-cursor for genuinely
> trusted events) are both implemented. See [`docs/DESIGN.md`](docs/DESIGN.md).

## Quick start

```bash
git clone https://github.com/kumard3/agentcursor.git
cd agentcursor && pnpm install && pnpm build
node dist/index.js setup
```

`setup` starts the local service, lists the AI apps it finds on your machine (Claude Code, Cursor, VS Code, Codex, Windsurf, Claude Desktop, Gemini CLI) and opens a setup page at `http://127.0.0.1:8931`. From there you connect each app with one click, grant the two macOS permissions, load the optional Chrome extension, and press **Test the cursor**. Prefer the terminal? `node dist/index.js setup --all` connects every detected app (or `--client=cursor,codex`).

Then ask your AI app something like: *"Use agentcursor: open Notes, create a new note and write a 3 item shopping list."*

How it fits together: every AI app launches `agentcursor` as an ordinary stdio MCP server. The first launch starts one shared background service (MCP over HTTP at `127.0.0.1:8931/mcp`, the extension bridge on `8930`) and every later launch reuses it, so several apps can use AgentCursor at once without fighting over a port. After a rebuild, the next launch replaces an older running service; an idle service exits after 10 minutes. The service only accepts requests from this machine and rejects other websites (Host and Origin checks).

## Changelog (key updates)

- **Unreleased**: E2E testing. `AgentCursor.launch()` starts a private Chrome (throwaway profile, its own copy of the extension on a free port) with its own visible cursor, so tests never touch your mouse, your browser or the MCP server on 8930, and several can run side by side. Works headed or `headless: true`. Computer use reaches the SDK too: `Desktop.open("Notes")` drives any Mac app with the real cursor and text-first reads. New agent CLI: every tool is a shell command (`agentcursor read_page`, `agentcursor desktop_click --text Save`) sharing one background service, so no tool schemas ever enter the model's context. Refs now stay with an element across reads, so `read_page --changes` and `desktop_read --changes` return only what changed: a repeat read of an unchanged window costs ~8 estimated tokens instead of ~397. Page reads got ~43% smaller, and the stdio tool list dropped from about 3,178 to 2,735 estimated tokens. New `expect()` with auto-retrying matchers (`toBeVisible`, `toBeHidden`, `toHaveText`, `toContainText`, `toHaveCount`, `toHaveURL`, `.not`). `navigate` now waits for the page to load, commands wait briefly for the content script after a navigation, `isVisible()`/`count()` answer immediately instead of waiting 5s, and headless screenshots no longer return the previous frame.
- **0.4.0**: Desktop control for any Mac app: `desktop_read` returns the window as compact text with `[dN]` refs from the macOS accessibility tree (a small Swift helper), then `desktop_click` / `desktop_type` / `desktop_key` / `desktop_scroll` drive the real cursor and keyboard with the same persona-based human motion. `desktop_screenshot` is a cropped, downscaled fallback. Chromium and Electron apps get their accessibility tree switched on automatically. New onboarding: `agentcursor setup` plus a local setup page that connects seven AI apps, checks permissions and tests the cursor. All AI apps now share one background service (stdio launches proxy to it), which removes the `port 8930 already in use` failure. The extension WebSocket now rejects connections from web pages.

- **0.3.0**: Programmatic SDK — `import { AgentCursor } from "agentcursor"` with a Playwright-shaped locator API (`getByRole`/`getByText`/`getByLabel`/`getByPlaceholder`/`getByTestId`/css + chaining + `filter`/`nth` + `click`/`type`/`fill`/`hover`/`dragTo`/`press`/`scrollIntoView` + `boundingBox`/`isVisible`/`count`/`waitFor`), every action driven by the human cursor. `connect()` and `os()` lifecycles. Library entry split from the MCP bin so importing the package no longer boots a server; built with `tsup` (ships `.d.ts`). Locator resolution uses `@testing-library/dom` in the content script.
- **0.2.9**: Active-tab resolution no longer requires Chrome to be the OS-focused window — it falls back to the active tab in any window, then any open http(s) tab. Fixes `No active tab found` when an agent drives the browser while you're in your editor/terminal (the normal case).
- **0.2.8**: Interaction: `press_key` — press Enter / Escape / Tab / arrows / Home / End / etc. on the focused element, content or stealth (trusted CDP key event). Rounds out the Comet-style action set: Navigation, Identification, Interaction.
- **0.2.7**: Identification tools — `find` (locate elements by visible text / accessible name, shadow-DOM aware) and `click_text` (find the best text match, then human-move + click, content or stealth, with a re-read retry). Target by what the element *says*, not by ref or pixel coords.
- **0.2.6**: Stealth typing inserts the whole string in one `Input.insertText` call. Per-character insertion landed at a reset caret on controlled editors (X's Draft.js) and typed text backward.
- **0.2.5**: Content-script `drag` reports the held button (`buttons` mask) during the move, matching the stealth and OS drivers, so JS drag handlers see a real drag. Added a Known Limitations section.
- **0.2.4**: `read_page` resolves multi-ID `aria-labelledby` names (it was passing the whole space-separated list to `getElementById` as one ID, so those labels came back empty); now shadow-DOM-tree-scope aware.
- **0.2.3**: `drag` now performs a real drag (press at the start, move with the button held, release at the end) in the stealth (`chrome.debugger`) and OS-cursor drivers — previously a move-then-click. `pnpm smoke` asserts the `screenshot` image content.
- **0.2.2**: `drag` tool. `screenshot` now returns a viewport-scaled image (1 image pixel = 1 click coordinate) for a vision loop — see the page, then `click`/`move_to` by `x/y`. Stealth (`chrome.debugger`) moves also animate the visible overlay, so the cursor stays on screen. Ships as a Claude Code plugin. Internal snapshot refresh resolves refs past the 60th element. `pnpm build` no longer mutates version files (use `pnpm reload` for the extension dev loop).
- **0.2.0**: Added `screenshot`, `hover`, `status` MCP tools. Deep shadow DOM traversal in `read_page` / snapshot (critical for X.com, Reddit, modern SPAs). Library re-exports for programmatic use. Repositioned as general local automation/testing/workflow tool over MCP. Version bumps and packaging polish.
- 0.1.0: Initial MCP server, human path engine, extension bridge, OS cursor driver, basic tools (read_page, click, type, etc.).

## How it works

Three layers, with one shared wire contract (`src/protocol`):

```
coding agent ──MCP/stdio──▶ MCP server ──localhost WebSocket──▶ Chrome extension ──▶ your real tab
                            (src/server)                        (extension/)
                                 │
                                 ▼
                          human-path engine (src/path-engine)
            from + to → timed cursor samples with overshoot, log-normal
            velocity, jitter, off-center landing, dwell — fresh every call
```

The MCP server generates the cursor sample stream; the extension is a thin
replayer. The same stream works for the content-script driver, the
`chrome.debugger` stealth driver, and (phase 2) the OS cursor — they all
implement one `BrowserDriver` interface.

## Why human-like movement is hard

Modern detectors (DataDome, Castle, reCAPTCHA v3, PerimeterX) flag overly smooth
Bézier paths, constant velocity, dead-center clicks, zero dwell, teleporting
jumps, and replayed identical paths. The engine addresses each:

- **Fitts's law** sets per-move duration from distance and target size.
- **Asymmetric, eased velocity** — not a symmetric min-jerk bell.
- **Overshoot-and-correct** on long moves.
- **Sub-pixel Gaussian jitter**, zero at the endpoints.
- **Off-center landing** inside the target.
- **Right-skewed dwell** before the press.
- **Per-call entropy** — paths are never cached or replayed.

Realism is necessary but not sufficient: content-script events are
`isTrusted=false`, and `chrome.debugger` still leaks CDP tells. The real evasion
endgame is the phase-2 OS cursor (genuine, trusted OS events).

## Install

```bash
git clone https://github.com/kumard3/agentcursor.git
cd agentcursor
pnpm install
pnpm build      # builds dist/index.js + extension/dist/*
```

### Install as a Claude Code plugin (one step)

AgentCursor ships as a Claude Code plugin that registers the MCP server for you:

```bash
claude plugin marketplace add kumard3/agentcursor
claude plugin install agentcursor
```

That registers the `agentcursor` MCP server automatically (no manual `claude mcp add`). You still load the extension once (step 1 below) if you want browser tabs. If you previously registered it by hand, remove that to avoid duplicate tools: `claude mcp remove agentcursor`.

### 1. Load the extension

1. Open `chrome://extensions`, enable **Developer mode**.
2. **Load unpacked** → select the `extension/` folder.
3. Keep a normal `http(s)` tab open and focused (not `chrome://` or the Web
   Store — content scripts can't run there).

### 2. Connect via MCP (agents, Cursor, Claude, custom tools, etc.)

The easy way is `node dist/index.js setup` (see Quick start). To do it by hand:

**Claude Code:**

```bash
claude mcp add --scope user agentcursor -- node /absolute/path/to/agentcursor/dist/index.js
```

**Cursor, Windsurf, or any MCP-capable coding environment:**

Add to your MCP servers config (exact format depends on the host):

```json
{
  "mcpServers": {
    "agentcursor": {
      "command": "node",
      "args": ["/absolute/path/to/agentcursor/dist/index.js"]
    }
  }
}
```

**Any other MCP client** (including future Grok harnesses, custom agents, test runners that speak MCP) — just point it at the stdio server the same way.

**HTTP-capable clients** can skip the stdio launcher and point straight at `http://127.0.0.1:8931/mcp` while the service runs (`node dist/index.js serve` keeps it in the foreground).

The service exposes the WebSocket bridge on `ws://127.0.0.1:8930` (override with `AGENTCURSOR_WS_PORT`; the HTTP port defaults to that plus one, or set `AGENTCURSOR_HTTP_PORT`). The extension auto-reconnects. `AGENTCURSOR_TOOLS=desktop` or `=browser` registers only one tool family, which keeps unused tool definitions out of your context.

### 3. Programmatic SDK (`import { AgentCursor }`)

Drive the human cursor from your own Node/TS code with a Playwright-shaped locator API — no MCP client needed. Same engine, same stealth; every action moves a real cursor.

```ts
import { AgentCursor } from "agentcursor";

// Attach to a running Chrome that has the extension loaded.
// Pass { stealth: true } for trusted CDP events; use AgentCursor.os() for the nut-js OS cursor.
const ac = await AgentCursor.connect();

await ac.navigate("https://example.com");
await ac.getByRole("button", { name: "Buy now" }).click();
await ac.getByLabel("Email").fill("a@b.com");
await ac.getByText("Submit").click();
await ac.getByLabel("Email").press("Enter");

// chaining + filtering, just like Playwright
await ac.locator(".row").filter({ hasText: "Pro" }).nth(0).getByText("Edit").click();

if (await ac.getByTestId("checkout").isVisible()) {
  await ac.getByTestId("checkout").click();
}

await ac.screenshot({ path: "out.png" });
await ac.close();
```

**Lifecycles**
- `AgentCursor.launch({ headless?, userDataDir?, args?, executablePath?, seed?, stealth? })`: start a browser with its own cursor. `executablePath` picks the binary (Chrome for Testing, Brave, Edge, BrowserOS); `userDataDir` drives a real, logged-in profile with no copy and leaves it intact; `headless` runs it in the background. See [Attach any browser](#attach-any-browser-real-profile-in-the-background) and [E2E tests](#e2e-tests-a-playwright-alternative-with-a-visible-cursor).
- `AgentCursor.connect({ port?, stealth?, timeoutMs? })`: attach to a running Chrome with the extension loaded (works with your real, logged-in profile).
- `AgentCursor.os({ stealth?, ... })`: same locator API, but the real OS cursor is moved via nut-js (genuinely trusted events); page sensing still goes through the extension.

**Run JS in the page**: `ac.evaluate(fn, ...args)`, Playwright-shaped, runs in the page realm via CDP (page cookies/session, awaits promises, ignores the page CSP):

```ts
const title = await ac.evaluate(() => document.title);
const res = await ac.evaluate(async (id) => {
  const r = await fetch(`/api/item/${id}`, { method: "POST", credentials: "include" });
  return { status: r.status, body: await r.text() };
}, 42);
```

**Locators** (lazy, chainable, Playwright-shaped)
- Find: `locator(css)`, `getByRole(role, { name })`, `getByText`, `getByLabel`, `getByPlaceholder`, `getByTestId`.
- Refine: `.filter({ hasText })`, `.nth(i)`, `.first()`, `.last()`, and chaining (`a.locator(b)`).
- Act: `.click()`, `.dblclick()`, `.hover()`, `.type()`, `.fill()`, `.press(key)`, `.dragTo(other)`, `.scrollIntoView()`.
- Query: `.boundingBox()`, `.textContent()`, `.isVisible()`, `.count()`, `.waitFor({ state })`.

Each action resolves the locator in the page (role/label/etc. via `@testing-library/dom`, css/text via the DOM), then drives the human-path engine to the element. `ac.actions` exposes the lower-level `ActionService` (move by coords, `find`, `clickText`, scroll) as an escape hatch.

A full runnable example is in [`examples/sdk-quickstart.mjs`](examples/sdk-quickstart.mjs); `pnpm smoke:sdk` runs the end-to-end pipeline against a simulated browser.

> Note: in-page (`stealth: false`) events are `isTrusted=false`. Use `stealth: true` (CDP) or `AgentCursor.os()` when you need trusted events.

### 4. CLI for coding agents (`agentcursor <command>`)

Every MCP tool is also a shell command. One background service holds the page, the `[refs]` and the frontmost app, so commands chain like a session, and an agent that uses the CLI loads no tool schemas at all.

```bash
agentcursor help                        # one line per command (agentcursor tools for full descriptions)

# browser
agentcursor navigate https://example.com
agentcursor read_page --includeText false
agentcursor click_text "Buy now"
agentcursor read_page --changes         # only what changed since your last read
agentcursor find Submit                 # a handful of tokens instead of a whole page
agentcursor screenshot                  # writes a file, prints the path

# any Mac app (computer use)
agentcursor desktop_open Notes
agentcursor desktop_read --max 40       # the window as text, not pixels
agentcursor desktop_click --text "New Note"
agentcursor desktop_type "hello" --submit
```

Positionals fill a command's parameters in the order `agentcursor help` lists them, and anything can be given as `--flag value`. Dashes work too (`click-text`). `AGENTCURSOR_OUT` sets where screenshots are written.

**Reading only the changes.** A `[ref]` stays with the same element for the life of the page (or until you switch apps), so a ref you learned earlier keeps working and reads can be compared. Pass `--changes` to `read_page` or `desktop_read` and you get the added and removed lines plus a count of the ones that merely moved:

```
- [e5] text "Email" <input> @293,300
+ [e5] text "Email" value="a@b.com" <input> @293,300
(5 moved, 2 unchanged, 10 total)
```

Measured on a Finder window: a repeat read costs 29 chars (~8 estimated tokens) against 1390 (~397) for the full read. The first read, or a diff that would be bigger than the full read, returns the full read instead.

### 5. Computer use from the SDK (`Desktop`)

```ts
import { Desktop } from "agentcursor";

const d = await Desktop.open("TextEdit");     // seed: 7 for reproducible motion
console.log(await d.read());                  // window as compact text with [dN] refs
await d.type("Dear team,", { clear: true });
await d.click("Save");                        // a [dN] ref or any visible label
await d.key("cmd+s");
if (await d.waitForText("Saved")) console.log("done");
```

`find()` returns only matching elements (tens of tokens), `view()` gives the structured elements, and `screenshot({ path })` is the fallback when the text is not enough. macOS only, and it needs Accessibility permission for whichever app starts it.

**Background runs, and a cursor per session.** By default computer use moves the one real pointer, which takes your machine over. `background: true` posts input straight to the target process instead: your pointer never moves, the app is never raised, and every `Desktop` keeps a cursor of its own, so runs happen while you work and several can run at once.

```ts
const alice = await Desktop.open("Notes", { background: true, showCursor: { color: "#4ade80", label: "alice" } });
const bob = await Desktop.open("Reminders", { background: true, showCursor: { color: "#60a5fa", label: "bob" } });
await Promise.all([alice.type("from alice"), bob.type("from bob")]);
```

`showCursor` is off unless you ask for it: it draws a click-through cursor above every window, coloured and named, so a background run is watchable. For MCP and the CLI, set `AGENTCURSOR_BACKGROUND=1` and `AGENTCURSOR_SHOW_CURSOR=1`.

Measured on a background TextEdit while the terminal stayed frontmost: the text landed, the frontmost app did not change, and the pointer sat at 471,628 before and after.

**The limit worth knowing:** this works for native (AppKit) apps. Chromium and Electron apps ignore process-posted events, measured on Chrome, which ignored them even while frontmost, so it covers Notes, Mail, Finder, TextEdit and friends but not Chrome, Arc, VS Code, Slack or Discord. For browsers use `AgentCursor.launch({ headless: true })`, which is fully background anyway and draws its cursor in the page.

**Driving a browser with computer use instead of the DOM.** A page can also be read and clicked as an app, with no extension and no DOM: the clicks are real OS clicks, so the page sees `isTrusted=true`. Chrome only builds its accessibility tree for a screen reader, so ask for it at launch:

```ts
const ac = await AgentCursor.launch({ accessibility: true });   // --force-renderer-accessibility
await ac.goto("http://localhost:3000");

const d = await Desktop.open();
console.log(await d.find("Menu"));     // [d23] button "Menu" @87,481
await d.click("Menu");                  // real cursor, trusted click
```

Without that option a desktop read of Chrome sees the toolbar and no page content (16 elements against 61 on the same window, measured). For your own Chrome, start it with `--force-renderer-accessibility`.

Which path to pick:

| | Extension (DOM) | Computer use (accessibility tree) |
| --- | --- | --- |
| Events | synthetic, or trusted with `stealth: true` | always real OS input |
| Reads | ~112 est tokens for a page | ~609 for the same window, browser chrome included |
| Runs headless / in parallel | yes | no: needs a visible, frontmost window and the real mouse |
| Works outside the browser | no | any Mac app, including Electron |

`find` costs about 8 tokens on either path, and `--changes` works for desktop reads too.

## Attach any browser (real profile, in the background)

AgentCursor drives any Chromium browser (Chrome, Chrome for Testing, Brave, Edge, BrowserOS, and mostly Arc) using **your real, logged-in profile**, in the **background** (it dispatches synthetic or CDP events, so the window need not be focused and your physical mouse is never touched). This is "computer use, but token-cheap and headless": `read_page` / `find` / `evaluate` cost tens to a few hundred tokens each versus a screenshot vision loop. `os()` is the only mode that runs in the foreground (it moves the real OS cursor).

Three ways to attach:

| Mode | Browser | Profile | Background | Setup |
| --- | --- | --- | --- | --- |
| Extension (`connect()` / MCP, default) | any Chromium | your real one | yes | Load unpacked once |
| `agentcursor launch` / `AgentCursor.launch({ userDataDir })` | any Chromium binary | real (no copy) or throwaway | yes (`--headless`) | one command |
| `os()` | frontmost | real | no (moves your mouse) | Accessibility grant |

**A. Extension mode (recommended for agents).** Load `extension/` unpacked into the browser you want (its real profile), point your MCP client at the stdio server, and drive it. See [Load the extension](#1-load-the-extension) and [Connect via MCP](#2-connect-via-mcp-agents-cursor-claude-custom-tools-etc). Nothing is copied; the browser you already use is the one being driven.

**B. `agentcursor launch` (one command, real profile, no manual extension load).** Starts a browser wired to the running service and holds it open until Ctrl-C:

```bash
# real, logged-in profile, in the background:
agentcursor launch \
  --user-data-dir "$HOME/Library/Application Support/Google/Chrome" \
  --chrome "/Applications/Google Chrome for Testing.app/Contents/MacOS/Google Chrome for Testing" \
  --headless

# throwaway profile, visible window (the default):
agentcursor launch
```

Flags: `--user-data-dir DIR` (real profile, left intact; omit for a throwaway copy), `--chrome PATH` (any Chromium binary; also `AGENTCURSOR_CHROME`), `--headless`, `--port` (WS port, default 8930). Then drive it with the normal tools (`agentcursor read_page`, `click_text`, `evaluate`, ...).

**Caveats:**
- **Profile lock:** with `--user-data-dir`, the browser must be fully quit first. Chromium will not share a running profile's data dir.
- **Arc:** Chromium underneath, but it wraps its own launch, so `--user-data-dir` + the CDP pipe is flaky. For Arc, prefer extension mode (A).
- Same-origin `fetch` inside `evaluate` carries that profile's cookies, so authenticated requests work against the logged-in session.

## Tools (MCP)

| Tool | What it does |
| --- | --- |
| `read_page` | Interactive elements with stable `[ref]` handles, roles, rects, visible text. Call this first in almost every test or workflow. |
| `find` | Identification: locate elements by their visible text / accessible name (shadow-DOM aware). Returns ranked `[ref]` matches — target by what the element says, no ref needed. |
| `click_text` | Identification + interaction in one step: find the best text match and human-move + click it (re-reads if needed). Supports `nth`, `double`, `stealth`. |
| `move_to` | Human-like path to a `[ref]` or `x/y` (no click). |
| `click` | Full human move + click (supports button, double, stealth mode for trusted events). |
| `hover` | Human approach + hover events (mouseover/mouseenter). Critical for dropdowns, tooltips, nav, and realistic workflows. |
| `drag` | Human path drag from ref/coords to target while holding button (sliders, reorder, canvas). |
| `type` | Human-timed keystrokes (auto human-clicks ref to focus if provided). |
| `press_key` | Press a single key (Enter, Escape, Tab, arrows, Home/End, PageUp/Down, Space, or a character) on the focused element. content or stealth. |
| `scroll` | Eased, human-stepped scrolling. |
| `screenshot` | Capture the visible tab as an image, scaled so 1 image pixel = 1 click coordinate — see the page, then `click`/`move_to` by `x/y` (the vision loop). Also for visual assertions and agent grounding. |
| `navigate` | Load a URL in the active tab. |
| `get_url` | Current tab URL. |
| `evaluate` | Run a JS function in the page (CDP `Runtime.evaluate`): page realm, uses the page's own cookies/session, awaits promises, ignores the page CSP. For reads and requests with no button (`() => document.title`, `async () => (await fetch('/api/x', { method: 'POST', credentials: 'include' })).status`). Returns the JSON result. Shows the debugger banner while it runs. |
| `wait_for` | Wait for element ref or visible text (up to timeout). Use for resilient testing flows. |
| `status` | Health / connection status, driver, active URL, port. Great for CI, long-running workflows, and monitoring. |

Any driving action accepts `stealth: true` to deliver trusted events through the
`chrome.debugger` driver (this shows Chrome's "debugging this browser" banner).

## Desktop tools (any Mac app)

No extension needed. Grant Accessibility (and Screen Recording, only for screenshots) to the app your AI runs in; the setup page has buttons for both.

| Tool | What it does |
| --- | --- |
| `desktop_apps` | Running apps, frontmost marked. |
| `desktop_open` | Open or switch to an app by name and bring it to the front. |
| `desktop_read` | The window as compact text: buttons, fields, links, menus, list items and visible text, each with a `[dN]` ref and center point. `find` returns only the best matches for a label. |
| `desktop_click` | Human path + click on a `[dN]` ref, a visible label (`text`), or `x/y`. Double and right click supported. |
| `desktop_move` | Move there without clicking (hover menus, tooltips). |
| `desktop_type` | Persona-timed typing into the focused field, or click a field first (`ref`, `into` label, `x/y`). `clear` replaces, `submit` presses Enter. |
| `desktop_key` | Keys and shortcuts: `enter`, `esc`, `tab`, `cmd+s`, `cmd+shift+t`. |
| `desktop_scroll` | Scroll by pixels over a target or where the cursor is. |
| `desktop_screenshot` | One window (or the area around a ref), downscaled JPEG, with the formula to turn image pixels into screen `x/y`. For canvases and custom-drawn UI. |

**Why text first.** Measured on a MacBook (text tokens estimated at 3.5 characters per token; image tokens from Anthropic's `width x height / 750`):

| App | `desktop_read` (whole window) | `desktop_read find` / click by label | Screenshot at 1024px wide |
| --- | --- | --- | --- |
| Finder | ~454 | ~9 | 662 |
| System Settings | ~543 | ~8 | 1181 |
| Arc | ~705 | ~9 | 852 |

The bigger saving is the loop: clicking by ref or label needs no screenshot to find coordinates and none to check where the click landed.

## E2E tests: a Playwright alternative with a visible cursor

`AgentCursor.launch()` opens its own Chrome with its own cursor. Your mouse stays yours, your normal browser and the MCP agent's cursor keep working, and every launch is independent, so two launches give you two cursors (two users in a chat app, a buyer and a seller). Use any runner; this is plain `node:test`:

```ts
import { test } from "node:test";
import { AgentCursor, expect } from "agentcursor";

test("checkout", async () => {
  const ac = await AgentCursor.launch({ seed: 7 }); // headless: true for CI
  try {
    await ac.goto("http://localhost:3000");
    await ac.getByRole("button", { name: "Buy now" }).click();
    await ac.getByLabel("Email").fill("agent@cursor.dev");
    await ac.getByRole("button", { name: "Submit" }).click();
    await expect(ac.getByText("Order placed")).toBeVisible();
    await expect(ac).toHaveURL(/\/thanks$/);
  } finally {
    await ac.close();
  }
});

test("two cursors at once", async () => {
  const [alice, bob] = await Promise.all([
    AgentCursor.launch({ args: ["--window-position=0,0", "--window-size=760,900"] }),
    AgentCursor.launch({ args: ["--window-position=780,0", "--window-size=760,900"] }),
  ]);
  // alice and bob act in parallel, each with a visible cursor in its own window
  await Promise.all([alice.close(), bob.close()]);
});
```

- `expect(locator)`: `toBeVisible`, `toBeHidden`, `toHaveText` (string or RegExp), `toContainText`, `toHaveCount`, and `expect(ac).toHaveURL`. Each retries until it passes or `{ timeout }` (default 5s) runs out; `.not` inverts.
- Chrome is found automatically (or `executablePath` / `AGENTCURSOR_CHROME`). The extension is loaded over `--remote-debugging-pipe`, so stable Chrome works.
- Same `seed` means the same motion and typing, so runs (and recorded demos) are reproducible.
- Not there yet: multiple tabs per launch, network mocking, traces, and non-Chromium browsers. Default events are `isTrusted=false`; pass `stealth: true` for trusted CDP input.

`pnpm e2e` runs [`test/e2e/showcase.e2e.mjs`](test/e2e/showcase.e2e.mjs) (`HEADLESS=1 pnpm e2e` for CI).

## Using as a Testing & Workflow Automation Tool

AgentCursor is not only for agents — it's a practical local browser automation primitive you can use directly in tests and scripts via MCP or by importing the core.

**Why it shines for testing/automation on real sites**:
- Human cursor paths + dwell + jitter + off-center clicks make interactions look like a real person (useful when sites have light behavioral signals).
- The visible cursor + overlay makes it excellent for **demo videos**, **manual review of automation**, and debugging failing flows.
- `screenshot` + `read_page` + `wait_for` + `hover` give you the primitives for visual + functional checks.
- Works against your **real Chrome profile** (cookies, extensions, logins) — perfect for realistic E2E that headless tools struggle with.

Example flow an agent or a test script might do:

```
read_page
hover "nav-menu"
click "Products"
wait_for text:"Featured"
screenshot
type {ref: "search", text: "laptop"}
click "search button"
...
```

**Direct / programmatic use (API style)**: The core `ActionService`, path engine, and drivers are designed to be importable. See "Programmatic Use" below.

### Example: Using with Claude Code to post on X.com / Reddit

With the MCP integration, you can tell Claude Code (or Cursor) to use agentcursor for realistic posting/automation on real sites:

1. Have a logged-in tab open on x.com (or reddit.com).
2. Start the server (ideally with OS driver on mac for best results).
3. In Claude: "Add agentcursor MCP if not present, then use the tools to navigate to x.com if needed, read the page, hover and click the compose area, type a test post, screenshot for verification, and click the post button. Use human-like actions and wait_for as needed. Report status often."

The shadow DOM support (added in 0.2.0) helps surface elements inside X's web components. Combine with `screenshot` + `status` + loops of `read_page` / `wait_for` for resilience on SPAs.

See the testing section above for general flow patterns. Always start with `status` and `read_page`, use `screenshot` to ground the agent.

## Trusted OS cursor (phase 2, macOS)

Content-script events are `isTrusted=false`, and `chrome.debugger` still leaks
CDP tells. For genuinely trusted, indistinguishable input, switch to the
OS-cursor driver, which moves the real macOS system cursor along the same human
path:

```bash
pnpm add @nut-tree-fork/nut-js        # optional native dependency
AGENTCURSOR_DRIVER=os node dist/index.js
```

It still reads the page through the extension (keep a normal tab focused), but
every move/click/scroll becomes a real OS event. Requires the Chrome window
visible and foregrounded at 100% zoom, and Accessibility permission for your
terminal/Node in System Settings → Privacy & Security. Coordinate mapping for
multi-monitor / fractional-scaling setups is still rough.

## Known limitations

- **Desktop control is macOS only for now.** It reads the accessibility tree, so apps that draw their own UI without accessibility (games, some canvases) need `desktop_screenshot` plus `x/y` clicks.
- **macOS grants permissions to the app that launched AgentCursor** (Terminal, Cursor, Claude...), not to AgentCursor itself. If the shared service was first started from a different app, grant that one, or stop the service (`curl -X POST http://127.0.0.1:8931/shutdown`) and let your main app start it.
- **Desktop typing does not render typo corrections.** The persona's timing applies, but only the final characters are typed.

- **Content-script events are `isTrusted=false`.** For detection-sensitive sites pass `stealth: true` (the `chrome.debugger` driver, trusted events) or use the OS-cursor driver. The visible overlay cursor shows in every mode.
- **React-controlled inputs** (X's composer, some design systems) can ignore content-script typing, which sets `value` directly. Use `stealth: true` (CDP `Input.insertText`) or the OS driver there.
- **`wait_for` by text and the snapshot `text` field use `innerText`**, which does not pierce shadow DOM. `read_page`'s element list *does* traverse shadow roots, so prefer waiting on a `[ref]` over page text on web-component-heavy sites (X, Reddit).
- **The OS-cursor driver assumes 100% browser zoom and a single display**; multi-monitor and fractional scaling can be off.
- **`hover` dispatches its hover events through the content script** (the approach move is trusted under `stealth`, the explicit `mouseover`/`mouseenter` are not).
- **Content-mode `press_key` carries `key`/`code` but not legacy `keyCode`** — a constructed `KeyboardEvent` always reports `keyCode: 0`. Modern handlers read `key`; for sites that still check `keyCode`/`which`, use `stealth: true` (the CDP key event sets the real virtual key code).

## Measuring realism

Serve the detector over http (the extension's content script only runs on
`http(s)`, not `file://`):

```bash
python3 -m http.server 8080 --directory test-detector
```

Open `http://localhost:8080`, click the targets by hand, then drive them with
the agent. Each click is scored on straightness, velocity variance, dwell,
off-center landing, overshoot, and `isTrusted` — the same features detectors
use. Use it to tune the engine.

## Development

```bash
pnpm dev         # run the server with tsx (no build)
pnpm typecheck    # tsc --noEmit
pnpm test             # vitest (path-engine + coord-map unit tests)
pnpm build:ext    # rebuild just the extension (no version change)
pnpm reload       # rebuild the extension AND patch-bump the version, so a chrome://extensions reload is visibly new
pnpm smoke        # end-to-end run: real MCP client + server, simulated browser (now covers screenshot/hover/status too)
pnpm build:native # rebuild the macOS accessibility helper (dist/native/agentcursor-ax)
AGENTCURSOR_HTTP_PORT=8931 node scripts/desktop-live.mjs   # live desktop run: reads Finder, types in TextEdit, closes it without saving
```

The `smoke` script is also a good template for writing your own automation or test runners that drive AgentCursor over MCP.

## Credits

The path engine builds on the `ghost-cursor` lineage (Bézier + Fitts) and the
mouse-dynamics literature — WindMouse, SapiAgent, BeCAPTCHA-Mouse, and the
vendor write-ups from DataDome and Castle on what makes synthetic movement
detectable. See [`docs/DESIGN.md`](docs/DESIGN.md).

## License

[MIT](LICENSE)