# 🐵 MonkeySee

> Monkey see, monkey do. Your agent gets eyes and hands in a real Chrome.

MonkeySee lets an MCP client (Claude Code, Codex, or anything that speaks MCP) drive a
real, logged-in Chrome profile. It opens tabs, reads the page as a compact indexed list of
elements, clicks, types, scrolls, takes screenshots, and decides when the job is done.

The key word is **your** Chrome. Not a fresh headless sandbox that gets logged out of
everything. The actual browser where you are already signed into your email, your dashboard,
your everything.

## Demo

https://github.com/user-attachments/assets/bb544504-8406-49e4-97fc-94e51bb56d59

## This is not an agent

Let's be clear about who does the thinking. MonkeySee has **no agent loop, no LLM, no
"reasoning."** Your terminal agent already has all of that. MonkeySee is the dumb,
fast, reliable set of hands and eyeballs it has always wanted.

```
  the brain                      the hands & eyes
┌─────────────┐   MCP (stdio)   ┌──────────────────┐   ws://localhost:8787   ┌───────────┐
│ Claude Code │────────────────▶│ monkeysee-bridge │────────────────────────▶│ extension │
│   / Codex   │                 │  (dumb router)   │                         │ SW + DOM  │
└─────────────┘                 └──────────────────┘                         └───────────┘
```

The agent decides _what_ to do. MonkeySee just does it and reports back what it saw.

Run more than one terminal at once? It just works — no daemon, no setup. Each session gets
its own bridge; the first to start owns the Chrome link (the **leader**) and the rest relay
through it (**followers**) over a second local port (`8788`). Each session tracks its own
tab, so an action in terminal A hits A's tab and an action in terminal B hits B's. If the
leader's terminal closes, a surviving session takes over automatically and keeps its tab.

```
                   ┌───────────────────┐  ws://localhost:8787  ┌───────────┐
session A ─stdio─▶ │ bridge (LEADER)   │──────────────────────▶│ extension │──▶ Chrome
                   └───────────────────┘                       └───────────┘
                       ▲
                       │ ws://localhost:8788  (control relay)
                       │
                   ┌───────────────────┐
session B ─stdio─▶ │ bridge (FOLLOWER) │  proxies its RPCs to the leader
                   └───────────────────┘
```

## What your agent can do

Once it's wired up, your agent gets a toolbox:

- **Look:** `get_state` (the page as a numbered element list, optionally with a
  set-of-marks screenshot), `get_forms` (just the forms — every field with its type, label,
  current value, options and flags; far cheaper than `get_state` for filling tasks),
  `extract_text`, `screenshot`
- **Act on what it sees:** `click`, `type`, `select_option`, `fill_fields` (fill many fields
  in one call, either instantly or as a watchable typewriter fill), `hover`, `focus` (all by
  element index)
- **Act by hand:** `click_at`, `scroll`, `scroll_to`, `drag`, `press`, `type_text`
- **Get around:** `open_tab`, `navigate`, `go_back`, `go_forward`, `wait_for_load`
- **Juggle tabs:** `list_tabs`, `switch_tab`, `close_tab`
- **Call it:** `done` (grounds the answer with the final URL + a page snippet)

## What you can do with it

The point of driving _your_ logged-in Chrome is that the agent can act on the context it
already has in your terminal session and the sites you're already signed into. A few
workflows that fall out of that:

- **Fill forms with context the agent already has.** Point Claude Code or Codex at a signup,
  job application, expense report, or vendor onboarding form and let it populate the fields
  from a file, a prior conversation, or your repo. It reads the form with `get_forms` (one
  compact, grouped list of fields — labels, types, current values, options), fills them in a
  single `fill_fields` call, and tells you what it entered before submitting. By default
  `fill_fields` runs in **progressive** mode: it scrolls each field into view and types it in
  like a person, so you can watch the form fill itself (great for demos and recordings). Pass
  `mode: 'batch'` for an instant, unattended fill instead.
- **Pull data out of dashboards that have no API.** Analytics, billing, internal admin
  panels you're logged into. The agent navigates, `extract_text`s, and hands back structured
  notes, no scraping credentials or headless re-login required.
- **Reproduce and triage a bug from a report.** Hand it the repro steps; it clicks through
  your staging app, screenshots each state, and reports where the flow actually breaks.
- **File the boring tickets.** Open Jira/Linear/GitHub, create issues from a list in your
  conversation, and link them back, all in the tab where you're already authenticated.
- **Cross-check work against a live site.** Diff what your code _should_ render against what
  the deployed page actually shows, with set-of-marks screenshots to point at the mismatch.
- **Drive multi-step web flows you'd rather not script.** Cookie banners, paginated tables,
  multi-page wizards: the agent re-reads the page after each step instead of relying on a
  brittle recorded selector.

In all of these the agent is the brain and MonkeySee is the hands. You stay in the loop,
the allowlist gates anything mutating, and every action reports back what it saw.

## Install

One install ships both halves. There is no Chrome Web Store listing to hunt for.

```bash
npm install -g monkeysee-bridge
```

Then wire it up. The fastest path registers the MCP server and prints the extension path
in one step:

```bash
monkeysee-bridge init
```

That registers `monkeysee` with Claude Code for every project (user scope) and prints
where the bundled extension lives. Flags: `--scope project` writes a repo-local
`.mcp.json` instead of your user config, `--client codex` wires up Codex
(`~/.codex/config.toml`) instead of Claude Code, and `--print` shows the config without
writing anything. See `monkeysee-bridge init --help`.

Prefer to wire it yourself? Register the server for every project with one command:

```bash
claude mcp add monkeysee -s user -- npx -y monkeysee-bridge
```

...or drop this into a project's `.mcp.json` (it then applies only when you run Claude
Code from that folder):

```json
{ "mcpServers": { "monkeysee": { "command": "npx", "args": ["-y", "monkeysee-bridge"] } } }
```

Either way, one step stays manual because no installer can do it for you: teach Chrome
about the bundled extension.

1. Open `chrome://extensions`
2. Flip on **Developer mode** (top-right)
3. Click **Load unpacked** and pick the path `init` (or the install) printed

Missed the path? No problem. The bridge reprints it to stderr every time it starts, so
it's right there in your MCP logs.

Once the extension is loaded, confirm the link is live:

```bash
monkeysee-bridge doctor
```

It opens the WebSocket, waits for the extension to connect, and reports `OK` (with the
extension version), `DOWN` (extension not loaded or Chrome not running), or `INCOMPATIBLE`
(a protocol-version mismatch). If a bridge is already running on the port, it says so
instead of fighting for it.

## The 60-second demo

The bridge is fully testable from the CLI (`pnpm test`, no browser). The full loop needs
Chrome:

1. **Build:** `pnpm build`
2. **Load the extension:** `chrome://extensions` → Developer mode → Load unpacked →
   `packages/extension/dist`. (The published `monkeysee-bridge` ships the extension inside
   it, and both the installer and the bridge's startup line print the path to load.)
3. **Start Claude Code** here. Copy `.mcp.json.example` to `.mcp.json` first (it's
   git-ignored, so it stays local). That entry launches the bridge over stdio. Within a few
   seconds the extension's service worker connects to the bridge. Look for the green dot in
   the extension popup.
4. **Give it a job.** Be explicit that it should use the browser, otherwise a capable agent
   will just answer from memory or its own web search and never touch MonkeySee. Try:
   _"Using the monkeysee browser tools, find me a Wikipedia article about Wales."_ It should
   `open_tab`, `get_state`, `type` into search, `press('Enter')` or `click`, re-read,
   `extract_text`, and `done` with the URL and a snippet.

> **`ws://localhost:8787` connection refused?** Totally normal when nothing is running. The
> bridge only listens while an MCP client has launched it. Start Claude Code here (the
> `.mcp.json` spawns it), or run it standalone to test the link:
> `node packages/bridge/dist/index.js`. The service worker reconnects with backoff, so order
> never matters. You'll see `extension connected` / `hello` on the bridge's stderr.

## Develop

```bash
pnpm install
pnpm build        # build all packages
pnpm dev          # watch all packages
pnpm typecheck
pnpm lint
pnpm test         # bridge end-to-end + multi-session checks (no browser needed)
```

Three packages in a pnpm workspace:

| Package               | Role                                                                                                            |
| --------------------- | --------------------------------------------------------------------------------------------------------------- |
| `monkeysee-protocol` | Shared wire types + zod schemas. The compatibility spine. Published to npm.                                     |
| `monkeysee-bridge`   | MCP server (stdio) + WebSocket server. Translates tool calls to RPC. Published to npm with a `bin`.             |
| `extension`           | MV3 Chrome extension: service-worker router + content-script eyes/hands. Bundled into the bridge, loaded unpacked. |

The deep dive lives in [`docs/`](./docs): [`STRUCTURE.md`](./docs/STRUCTURE.md) (project map
+ design decisions).

## Safety (minimal, but real)

You are pointing a robot at a browser that's logged into your life. Treat it that way.

The extension popup has a **domain allowlist**. With **Enforce** on, mutating actions
(`click`, `type`, `select_option`, `click_at`, `drag`, `press`, `type_text`) on a domain
that isn't allowlisted return a `blocked` error. Looking and navigating are never gated.

## Trusted input (chrome.debugger)

By default, actions fire synthetic DOM events from the content script. Some sites are picky:
they check `event.isTrusted`, or shrug off a synthetic Enter in a search box.

Flip **"Trusted input (chrome.debugger)"** in the popup to route `click`, `type`,
`type_text`, `press`, `click_at`, and `drag` through real input dispatched via the Chrome
DevTools Protocol (`Input.dispatch*`). These events are trusted and behave like a real human
finger. Notes:

- `debugger` is a **required** permission (Chrome won't allow it as optional). It's only
  _used_ when you turn the toggle on.
- If attaching fails (say DevTools is already open on that tab), MonkeySee quietly falls back
  to synthetic events for that one action.
- The MCP tool surface is identical either way. The agent never knows or cares which backend
  ran.

### About that yellow "started debugging this browser" banner

Chrome itself shows the **"MonkeySee Browser Agent started debugging this browser"** infobar
the moment any extension calls `chrome.debugger.attach()`. In MonkeySee that happens **only**
when both are true:

1. the **Trusted input** backend is on in the popup, and
2. a debugger-backed action actually runs (`click`, `type`, `type_text`, `press`,
   `click_at`, or `drag`). Attach is lazy and per-tab, on the first such action, and the
   banner sticks around until MonkeySee detaches or the tab closes.

It does **not** show up for:

- opening a tab or navigating
- observation: `get_state`, `extract_text`
- `screenshot` / `get_state({ withScreenshot: true })` (these use
  `chrome.tabs.captureVisibleTab`, no debugger required)
- any action while the backend is left on the default `content` (synthetic events)

So: keep the backend on `content` and you'll never see the banner. Turn the trusted backend
on and you'll see it on each tab the first time MonkeySee dispatches input there.

## Frames + screenshots

- **Same-origin frames** are indexed automatically (`all_frames: true`). Elements inside a
  same-origin iframe show up in `get_state` with `frameId != 0`, are clickable by index, and
  report their boxes in top-viewport coordinates. Cross-origin iframes are out of scope for
  now.
- **`screenshot`** returns the controlled tab's visible viewport as a PNG.
- **`get_state({ withScreenshot: true })`** adds that image with numbered set-of-marks drawn
  over each in-viewport element. Great for the agent to "point" with confidence.

## Version compatibility

The bridge and extension swap a `protocolVersion` in the WebSocket `hello` handshake (the
contract lives in `monkeysee-protocol`). If their **major** versions disagree, the bridge
refuses the connection and never serves a tool call to a mismatched extension. The popup
shows an "incompatible bridge" status and the extension retries slowly until you update the
older side. The same handshake guards the follower→leader control channel, so a stale
session whose protocol differs is refused rather than served across a gap. Pre-1.0, both
sides are major `0` and move in lockstep through the workspace, so this only bites across a
future breaking bump.

## License

MIT. See [`LICENSE`](./LICENSE).