🚀 BrowserPilot

Control your local desktop browser directly from remote VPS AI Agents via Model Context Protocol (MCP)

💡 Why BrowserPilot?

When building autonomous AI agents on a remote VPS, interacting with modern websites is challenging:

• Traditional headless browsers (like standard Puppeteer/Playwright) get blocked by Cloudflare, reCAPTCHA, and bot-detection systems.

• Authenticating into your personal accounts (Google, GitHub, banking, dashboards) on a headless VPS requires syncing cookies and session tokens.

• Remote debugging ports (--remote-debugging-port=9222) trigger Chrome's prominent yellow "Browser is being controlled by automated software" banner.

BrowserPilot solves this by establishing a secure, persistent outbound WebSocket bridge directly from your local Chrome/Brave/Edge browser to your VPS MCP server. Your VPS AI agent can interact with your real, authenticated browser tabs using realistic DOM events while keeping the connection ultra-lightweight and invisible.

Related MCP server: MCP Chrome Bridge

🧪 Quick Verification Test

Once your browser extension is connected, copy and paste this command directly to your AI agent to verify everything is working end-to-end:

Open instagram.com and follow abhinav.nexus

🏛️ Architecture

sequenceDiagram
    autonumber
    actor User as You (Local PC)
    participant Ext as Chrome Extension (MV3)
    participant Relay as Offscreen Document (Persistent WS)
    participant MCP as VPS MCP Server
    actor Agent as AI Agent (Claude/Cursor/AGY)

    User->>Ext: Loads Extension & Enters VPS Endpoint
    Ext->>Relay: Initializes Background Offscreen Relay
    Relay->>MCP: Outbound WebSocket Connect (ws://<your-vps-ip>:8765?token=...)
    MCP-->>Relay: Auth Verified (200 OK)
    
    rect rgb(30, 41, 59)
        note right of Agent: AI Agent executes browser action
        Agent->>MCP: Call Tool: browser_click({ selector: "#submit" })
        MCP->>Relay: Send JSON Command (ID: cmd_101)
        Relay->>Ext: Dispatch to Content Script
        Ext->>Ext: Highlight element & dispatch native mouse events
        Ext-->>Relay: Action Succeeded
        Relay-->>MCP: Return Result (ID: cmd_101)
        MCP-->>Agent: Tool Response: "Clicked #submit successfully"
    end

🛠️ MCP Tools Reference

BrowserPilot exposes 21 specialized tools directly to any MCP-compatible AI agent:

🛡️ Hybrid Privacy & Tab Permission Engine

BrowserPilot features an intelligent, multi-tier privacy boundary ensuring that AI agents cannot inspect or tamper with your personal tabs without consent:

• 🟢 Auto-Approved Agent Tabs (Zero Friction): Any tab spawned by the agent via browser_navigate({ url, newTab: true }) is tagged as agent-owned. The agent can browse, click, type, screenshot, and close within its own tabs with 100% autonomy.

• 🔒 Protected User Tabs: Pre-existing tabs (e.g. Gmail, Slack, banking, personal dashboards) are shielded by default. If an agent attempts to interact with an unapproved tab:

  • An elegant, floating banner slides down on that tab: "🤖 AI Agent is requesting permission to access this tab. [Allow Access] [Deny]".

  • The MCP server returns an immediate PERMISSION_REQUIRED response informing the agent that user authorization is required.

  • The agent can also explicitly call browser_request_tab_access({ tabId, reason }) to request permission with a custom explanation.

• ⚙️ Extension Popup Privacy Shield:

  • Privacy Mode Selector: Switch between Hybrid (Recommended), Full Access (Unrestricted), and Strict Sandbox (Agent Tabs Only) in one click.

  • Live Tabs Permission Manager: View all open browser tabs in the popup with real-time status chips (Agent Tab, Protected, Allowed) and quick Allow / Revoke action buttons.

📦 Project Structure

browserpilot/
├── mcp-server/              # Model Context Protocol Server (VPS side)
│   ├── src/
│   │   ├── index.ts         # Stdio MCP Server & lifecycle entry
│   │   ├── websocket-hub.ts # WebSocket server & command dispatcher
│   │   ├── tools.ts         # MCP tool definitions & schema validation
│   │   └── types.ts         # Protocol message interfaces
│   ├── test/                # Automated bridge integration tests
│   └── package.json
│
├── extension/               # Manifest V3 Chrome Extension (Local side)
│   ├── manifest.json        # Extension configuration & permissions
│   ├── background.js        # Service worker & tab router
│   ├── offscreen.html/js    # Offscreen document (unbreakable WebSocket keep-alive)
│   ├── content.js           # In-page DOM engine & element highlighter
│   ├── popup.html/css/js    # Settings popup UI
│   └── icons/               # Extension icons
│
└── package.json             # Root pnpm workspace

🚀 Quickstart Guide

Step 1: Start the MCP Server on your VPS

1. Clone or copy the repository to your VPS:

   cd /root/browserpilot
   pnpm install

2. Configure environment variables in mcp-server/.env:

   WS_PORT=8765
   SECRET_TOKEN=my-secure-browserpilot-token

3. Build and test the MCP server:

   pnpm build
   pnpm --filter browserpilot-mcp exec tsx test/test-bridge.ts

sudo ufw allow 8770/tcp comment "BrowserPilot WebSocket"
sudo ufw reload

sudo firewall-cmd --permanent --add-port=8770/tcp
sudo firewall-cmd --reload

sudo iptables -A INPUT -p tcp --dport 8770 -j ACCEPT

Step 2: Install the Chrome Extension on your Local Computer

1. Copy or download the extension/ folder from your VPS to your local PC.

2. In your local browser (Chrome, Brave, Edge):

   • Navigate to chrome://extensions

   • Enable Developer mode in the top-right toggle.

   • Click Load unpacked and select the extension folder.

3. Click the BrowserPilot icon in your browser toolbar:

   • VPS WebSocket Endpoint: ws://<your-vps-ip>:8765 (or wss://tunnel.yourdomain.com)

   • Secret Token: my-secure-browserpilot-token

   • Click Connect.

4. The badge will turn 🟢 ON and status will display "Connected to VPS".

Step 3: Connect your AI Agent to the MCP Server

Add BrowserPilot to your agent's MCP configuration:

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "browserpilot": {
      "command": "node",
      "args": ["/path/to/browserpilot/mcp-server/dist/index.js"],
      "env": {
        "WS_PORT": "8765",
        "SECRET_TOKEN": "my-secure-browserpilot-token"
      }
    }
  }
}

Antigravity CLI (agy) or Custom Agent

{
  "mcpServers": {
    "browserpilot": {
      "command": "node",
      "args": ["/root/browserpilot/mcp-server/dist/index.js"],
      "env": {
        "WS_PORT": "8765",
        "SECRET_TOKEN": "my-secure-browserpilot-token"
      }
    }
  }
}

🔒 Security & Privacy

🤖 AI Agent Skill

BrowserPilot includes a dedicated AI Agent skill definition located in skills/browserpilot/SKILL.md.

To equip your Antigravity or custom agent with native BrowserPilot capabilities:

1. Copy or symlink skills/browserpilot into your agent's skills directory (e.g. ~/.gemini/skills/browserpilot).

2. Your agent will automatically learn the tool semantics, resilience rules, non-intrusive multitasking behavior, and the standard verification prompt:

   Open instagram.com and follow abhinav.nexus

🔧 Troubleshooting & Known Issues

Facing port conflicts, Chrome MV3 restrictions, strict CSP errors, or disconnects?

👉 Read the comprehensive TROUBLESHOOTING.md guide for detailed solutions to every common edge case.