Browser Control

Open-source browser control for AI agents: an MCP server plus a Chrome extension, with vision-first control, multi-agent tabs, vault-filled logins and human handoff. Browser Control is a Chrome extension plus a small local host that lets AI agents drive a real, headful Chrome over MCP: they look at a screenshot and act at x/y, the way a person does. It is built for running several agents at once, each in its own browser profile, with credentials filled from your password manager rather than typed by the model.

Status: early (0.x). macOS is the supported host platform today.

Features

• Vision-first computer tool. Screenshot, click, double/triple/right click, drag, type, key chords, scroll, zoom into a region, wait — all by screenshot-pixel coordinates, with a visible cursor overlay. DOM helpers (read_page, find, refs) exist to locate things, not to act.

• Multi-agent tab groups. Every (tenant, agent) gets its own Chrome tab group; a call on a tab outside the caller's group fails with TAB_NOT_OWNED. Each tenant runs in its own Chrome for Testing profile (--user-data-dir), launched on demand and stopped when idle.

• Vault-filled logins via a pluggable password-manager adapter. vault_fill and fill_totp resolve a credential on the host and insert it straight into the focused field. The model never sees the value, and each tenant can only reach the vault items you allowlist. The default adapter uses the 1Password CLI (op); the resolver is one small interface (host/daemon/secretResolve.ts).

• Human handoff. handoff turns the agent's tab group red, shows a banner with the reason and a Done button, and handoff_resume waits for a person to finish a CAPTCHA, SMS code or similar.

• Secret-field masking. Screenshots paint black boxes over password, card and one-time-code fields (including inside cross-origin iframes) and fail closed if masking can't run; get_page_text refuses on secret pages; the action log never records typed text.

• Loud failures. Every call has a timeout; a dropped host or extension returns a structured error (NATIVE_HOST_DISCONNECTED, TIMEOUT) instead of hanging.

Related MCP server: Aether Browser MCP

How it works

agent --MCP/HTTP (Bearer)--> host daemon --unix socket--> shim --native messaging--> extension --CDP--> tab
                             (127.0.0.1)   auth, tenants,          (per Chrome)                (chrome.debugger)
                                           vault resolver

See ARCHITECTURE.md, host/README.md and docs/EXTENSION.md for the details.

Install

Requirements: macOS, Node.js ≥ 20, jq, and (optionally) the 1Password CLI for vault fills.

1. Extension key and ID

Chrome derives an extension's ID from its public key. This repo ships no key: generate your own so your build has a stable ID of its own.

npm ci
npm run gen-key          # writes .keys/extension.pem (private, gitignored) + .keys/extension.pub.b64
                         # and prints your extension ID

Keep .keys/extension.pem in your own secret store. At build time vite injects the public key from .keys/extension.pub.b64, or from the BROWSER_CONTROL_EXTENSION_KEY env var (use that in CI and for the host installer, which builds from a fresh clone). Without a key, Chrome assigns an ID from the unpacked directory path, and you must pass that ID to the host with BUE_EXTENSION_ID.

2. Extension

npm run build            # -> dist/

The host launches Chrome for Testing with --load-extension=dist/, so there is nothing to install by hand. To try the extension on its own, load dist/ in chrome://extensions → Developer mode → Load unpacked.

3. Host

cd host && npm ci && npm run build && cd ..
BROWSER_CONTROL_REPO=<your fork's git URL> \
BROWSER_CONTROL_EXTENSION_KEY="$(cat .keys/extension.pub.b64)" \
  host/scripts/install-mac.sh --tenant my-agent

The installer builds everything, installs a pinned Chrome for Testing, writes ~/Library/Application Support/BrowserControl/state/host.config.json (0600), generates a random bearer token for my-agent in state/tenant-keys.json (0600, never printed), installs the native-messaging manifests (sudo, the only privileged step), registers a launchd agent, and runs a smoke test.

Point your MCP client at http://127.0.0.1:8787/mcp with Authorization: Bearer <token from tenant-keys.json>.

Bearer tokens come from a local file by default. You can also supply them as JSON in the BUE_KEYS_JSON env var, or use the optional AWS Secrets Manager adapter (--auth-source aws-secrets-manager --keys-secret your-secret-name). See host/README.md.

Vault fills are off until you allowlist items: set vaultAllowlist.<tenant> in host.config.json (see host/allowlist.example.json), and give the daemon a 1Password service-account token via OP_TOKEN_FILE=/path/to/token when running the installer.

MCP tools

Tool names and argument shapes are compatible with Claude in Chrome's MCP surface where an equivalent exists. A few of those tools (gif_creator, upload_image, shortcuts_*, resize_window, multi-browser pairing) return NOT_IMPLEMENTED today.

Comparison

A factual summary as of this release; check each project's docs for current details.

Telemetry

Off by default, and nothing is sent unless you configure an endpoint. If you opt in, the host sends one event per MCP tool call containing exactly:

{ "tool": "computer", "durationMs": 42, "ok": true, "errorCode": null, "extensionVersion": "0.1.0.75" }

Never URLs, page text, screenshots, arguments, tenant/agent names or tokens. Enable it with BUE_TELEMETRY=1 and BUE_TELEMETRY_ENDPOINT=https://your-collector.example/events (or config telemetry: { enabled: true, endpoint: "..." }). Events are batched and POSTed as { "events": [...] } with no retries. The code is in host/daemon/telemetry.ts.

Development

npm ci && npx tsc -p . && npm test          # extension: typecheck + Playwright suite (headless Chromium)
cd host && npm ci && npx tsc -p . && npm test   # host: vitest suite

The acceptance scenarios in acceptance/ drive a full local stack against fixture pages; see acceptance/README.md.

Contributing and security

See CONTRIBUTING.md. To report a vulnerability, see SECURITY.md. Please do not open public issues for security problems.

License

MIT, see LICENSE.