BrowserAgent

✨ Highlights

• One model, not two. observe returns the a11y snapshot and the pixel overlay in a single call, so the model never reconciles text and images itself.

• Diffs, not dumps. The diff engine returns only what changed since the last observation, with fingerprint-based uid rebinding across navigation.

• Events, not polling. Console, network, DOM, and navigation events are collected and pushed, so the model doesn't re-read the page every turn.

• Strict TDD at 100/100. Every module lands at 100% coverage and 100% mutation score, enforced by CI.

• TypeScript-only, zero tolerance. No .js/.mjs/.cjs anywhere; no as, any, !, or @ts-ignore in the codebase.

Related MCP server: hermes-computer-use

The thesis

Most browser integrations treat "read the page" and "act on the page" as separate worlds, one returning text and the other returning pixels. That split forces the model to re-read the page every turn and to guess at the relationship between what it sees and what it can click.

BrowserAgent's core idea: build one unified, event-driven, visually-rich model where the semantic layer and the visual layer are the same object, a model that can both watch and explain.

Why another browser MCP server?

It's not a fork of chrome-devtools-mcp. It borrows that project's proven patterns (stable element uids keyed by loaderId_backendNodeId, a ContextPage abstraction that hides Puppeteer behind a narrow contract, an "act then wait for stable DOM/navigation" wrapper, and token-optimized formatters) and rebuilds them from scratch around a different product shape.

• Unified observe, not split snapshot/screenshot. One call returns the a11y tree with the pixel overlay.

• Diff, don't re-read. The diff engine (with fingerprint-based uid rebinding) means the model doesn't pay to re-read the whole page every turn.

• Events, not polling. The server pushes changes instead of the model polling.

• Fewer round trips over micro-optimization. The browser and the LLM are the bottlenecks, so performance comes from the diff engine and event-driven observation rather than shaving milliseconds.

Architecture

flowchart TB
    Client[MCP Client / LLM]

    subgraph Protocol["MCP Protocol Layer"]
        Server[McpServer<br/>tools/list · tools/call · server/discover]
        Tasks[Tasks fallback<br/>get · list · cancel · wait]
        Apps[MCP Apps<br/>ui:// replay]
        Mrtr[MRTR confirm_action]
    end

    subgraph Framework["Tool Framework"]
        Handler[ToolHandler<br/>defineTool · gating · write mutex]
    end

    subgraph Core["Core Model"]
        direction LR
        Observe[observe<br/>snapshot + overlay]
        Diff[Diff engine<br/>changes + rebinding]
        Events[Event layer<br/>buffer + collector]
        Actions[Action layer<br/>log + act-then-wait]
    end

    Browser[ContextPage<br/>over Puppeteer / CDP]

    Client <-->|MCP| Server
    Server --> Tasks
    Server --> Apps
    Server --> Mrtr
    Server -->|registerTool| Handler
    Handler -->|read| Observe
    Handler -->|write| Actions
    Handler -->|intent| Intent[watch_until · run_flow · verify · explain]
    Observe --> Diff
    Observe --> Events
    Actions --> Events
    Actions --> Apps
    Observe --> Browser
    Actions --> Browser
    Intent --> Browser

The layers are thin and testable: the protocol layer bridges our tools onto MCP, the ToolHandler enforces gating, and the ContextPage hides Puppeteer behind a narrow contract so nothing touches the raw page.

The product surface, in dependency order:

Getting started

Requirements

Install

git clone https://github.com/PremierStudio/BrowserAgent.git
cd BrowserAgent
npm install

Run the server

BrowserAgent speaks MCP over stdio (and Streamable HTTP via createHttpHandler). npm start launches a headless Chrome via Puppeteer, attaches the event collector, and serves the fully-wired MCP server:

npm start

Streamable HTTP (same tool set) on port 3333, or PORT:

npm start -- --http

Point any MCP client (Claude Desktop, a custom host, etc.) at the stdio command node dist/cli.js. The server exposes tools/list, tools/call, and server/discover via the v2 @modelcontextprotocol/server SDK.

Page tools: observe, click, type, hover, scroll, select, press, navigate

Intent tools: watch_until, run_flow, verify, explain

Tasks fallback (decision #2, hosts without ext-tasks): get_task, list_tasks, cancel_task, wait_task

HITL: confirm_action returns an InputRequiredResult (MRTR / elicitation)

Resources: browser://events (JSON event stream) and ui://browser-agent/replay (MCP App HTML replay)

Run the full gate chain

This is exactly what CI enforces:

npm run ci

Individual gates

Software stack

Engineering: strict TDD with a 100/100 gate

Every module is written test-first (RED → GREEN → refactor) and must clear a hard, CI-enforced gate before it is committed:

typecheck → lint → format → knip → unit tests → coverage (100%) → mutation (100%) → survivor registry

• 100% coverage (lines/branches/functions/statements) via Vitest + @vitest/coverage-v8.

• 100% mutation score via Stryker. Coverage alone is a lie; mutation testing proves the tests actually catch real faults. The only escape from the mutation gate is a pre-approved, documented entry in mutation-survivors.json (currently empty).

• No dead code. knip runs with zero findings. Unused files, exports, and dependencies are removed, not ignored.

• TypeScript only, everywhere. No .js/.mjs/.cjs anywhere, including source, configs (eslint.config.ts, stryker.config.ts, vitest.config.ts), and scripts (emitted to dist-scripts/ via tsconfig.scripts.json).

• No banned constructs. as casts, any, ! non-null assertions, @ts-ignore/@ts-nocheck/@ts-expect-error, and .forEach are all lint errors.

• Deterministic tests. Clocks and timers are injected, so there are no sleeps and no flaky timing.

The full engineering spec lives in docs/mvp.md; every deviation and protocol decision is recorded in docs/decisions.md.

Repository layout

src/
  actions/       ActionLog (replay seed), ActionRunner, StabilityWaiter (act-then-wait)
  apps/          MCP App replay HTML renderer
  context/       ContextPage abstraction + CDP a11y-tree conversion (axTree)
  diff/          diff engine, fingerprint-based uid rebinding, DiffTracker
  events/        event types, bounded EventBuffer, normalizer, EventCollector
  intent/        watch_until, run_flow, verify, explain
  protocol/      MCP bridge: tools, Tasks fallback, MRTR, HTTP, resources
  session/       BrowserSession composition root (observe+diff+log)
  snapshot/      a11y snapshot builder + uid→box overlay
  tasks/         TaskStore + TaskRunner (owned Tasks state machine)
  tools/         tool framework: defineTool, ToolHandler, ToolMutex, Response, observe
  uid.ts         stable loaderId_backendNodeId uid generation
docs/
  mvp.md         the product plan and non-negotiable engineering requirements
  decisions.md   every amendment (supersedes mvp.md where they conflict)
scripts/         survivor-registry checker (TypeScript, emitted to dist-scripts/)
.github/workflows/ci.yml

Product surface

flowchart TB
    subgraph Observe["See"]
        O[observe]
        D[diff since last observe]
        E[browser://events]
    end

    subgraph Act["Act"]
        A[click type hover scroll select press navigate]
        F[run_flow]
        W[watch_until]
    end

    subgraph Reason["Reason"]
        V[verify]
        X[explain]
        C[confirm_action]
    end

    subgraph Replay["Replay"]
        U[ui://browser-agent/replay]
        T[get_task list_tasks cancel_task wait_task]
    end

    O --> D
    O --> E
    A --> U
    F --> A
    W --> O
    V --> O
    X --> O
    C --> T

observe is the unified primitive: a11y snapshot, screenshot, overlay, diff, and recent events in one object. Actions go through act-then-wait and seed the replay. Intent tools (watch_until, run_flow, verify, explain) sit on top of that model. Long-running work uses the owned Tasks state machine, with a blocking wait_task fallback for hosts that do not speak ext-tasks. Destructive steps can pause on confirm_action via ratified MRTR elicitation.

Contributing

This repo follows the rules in AGENTS.md (binding for every agent) and the spec in docs/mvp.md. The short version:

• Strict TDD. Write the failing test first (RED), confirm it fails for the right reason, then implement (GREEN), then refactor.

• The 100/100 gate. No module merges unless it's at 100% coverage and 100% mutation score, with typecheck/lint/format/knip clean.

• No survivor silencing. A surviving mutant is fixed by strengthening the test, never by weakening the code or adding ignore comments.

• TypeScript only. No .js/.mjs/.cjs, including configs and scripts.

License

Apache License 2.0 © Premier Studio. See LICENSE.