Cognitive Browser MCP

Cognitive Browser is an autonomous web browser agent and Model Context Protocol (MCP) server. It executes multi-step web navigation workflows by combining a local discriminative cross-encoder (Reflex Core) with selective generative LLM escalation (Cortex Engine).

By resolving routine element interactions through Natural Language Inference (NLI) rather than streaming raw HTML or full DOM trees to generative models on every step, it minimizes prompt token overhead and reduces execution latency during repetitive interactions.

Benchmark: 15-Case Web Automation Performance

Methodology Notice: The 15 scripted tasks in this suite serve as an automated regression benchmark to verify routing and execution accuracy across diverse layouts, not as proof of general browsing competence. Upstream planning tokens (formulating the initial sequence of steps from high-level natural language goals) are not included. Measured on NVIDIA GeForce RTX 4070 Ti (CUDA, 12 GB VRAM). Note that for a sample size of $n = 32$ interaction steps, the 95th percentile ($p95$) is approximate. For full hardware specifications, per-task breakdowns, and token accounting definitions, read docs/benchmark-methodology.md.

Results & Calibration Data

• 15-Case Benchmark Details: Empirical run logs and hardware setup in docs/benchmark-methodology.md (raw results: benchmarks/benchmark_15_cases_results.json | summary script: benchmarks/summarize_results.py).

• Reflex Threshold Sweep: Calibration metrics (precision, recall, escalation rates across thresholds 0.30–0.90) in docs/threshold-analysis.md (raw sweep data: benchmarks/threshold_sweep_results.json).

• Demo Recording Guide: Instructions for recording terminal demo GIFs in docs/demo.md.

Related MCP server: openmcp

Architecture

Standard browser agents feed full accessibility snapshots or raw HTML into a cloud LLM reasoning loop on every single turn. Cognitive Browser divides automation into a hybrid edge/cloud architecture:

flowchart TD
    User["Orchestrating Agent (Claude / Gemini / GPT)"] -->|"cognitive_browse(url, steps)"| Nav["Browser Navigation & Overlay Auto-Dismiss"]
    Nav --> Snapshot["Accessibility Tree (ARIA Snapshot)"]
    
    Snapshot --> Reflex["Reflex Core: DeBERTa-v3 NLI Cross-Encoder\n(Local GPU / MPS / CPU)"]
    
    Reflex -- "Confidence >= Threshold (default: 0.50)" --> Exec["Direct Playwright Action\n(fill / click) (~25 ms, 0 LLM tokens)"]
    
    Reflex -- "Confidence < Threshold (Ambiguity)" --> Esc["Pause & Return 'clarification_needed'\nwith Top Candidates via MCP"]
    Esc -->|"Clarifies target via smart_action"| User
    
    Exec -->|"Next step"| Next["Next Step / Completion"]

1. Reflex Core (Local System 1): Evaluates interactive candidates using Zero-Shot Natural Language Inference (cross-encoder/nli-deberta-v3-small, 140M parameters on GPU/CPU) in ~25 ms with zero cloud tokens.

2. Orchestrating Agent as Cortex (System 2): When confidence falls below threshold (REFLEX_CONFIDENCE_THRESHOLD, default 0.50) or an element is ambiguous, execution pauses and returns clarification_needed with top candidate elements directly to the orchestrating LLM via the MCP protocol. No local Ollama required.

3. DOM Locator Strategy:

   • Exact-match prioritized locator patterns (^\s*...\s*$) to eliminate substring collisions (e.g. 'new' matching 'Hacker News').

   • Heuristic cookie and consent overlay auto-dismissal.

   • Unicode icon character normalization (\ue000-\uf8ff stripping).

   • Input role filtering (require_input_only for text fill operations).

Quickstart

1. Installation

git clone https://github.com/mikita12/cognitive-browser.git
cd cognitive-browser

pip install -e .
playwright install chromium

2. Run as MCP Server

Add to your MCP client configuration (e.g., Claude Desktop claude_desktop_config.json or Antigravity mcp_config.json):

{
  "mcpServers": {
    "cognitive-browser": {
      "command": "python",
      "args": ["-m", "cognitive_browser.mcp_server"]
    }
  }
}

Or run via the installed console script:

cognitive-browser

3. Run as Docker Microservice

docker compose up -d --build

The FastAPI REST endpoint and SSE MCP server will be accessible on http://localhost:8000.

Router Modes: Local vs Cloud

The router selection is configured via environment variables or explicit parameters:

MCP Tools

1. cognitive_browse(url, steps):

   • Executes multi-step workflows autonomously with Reflex decisions and Cortex escalation.

   • Returns execution logs and structured telemetry (total_execution_seconds, reflex_gpu_inference_ms, cortex_llm_tokens).

2. get_aria_snapshot(url, max_candidates=40):

   • Extracts a structured semantic Accessibility tree (filtering decorative HTML).

3. smart_action(url, instruction, fill_value=None):

   • Evaluates and executes an atomic single action (click or fill).

Limitations

• Single Page Applications (SPAs): Pages with slow client-side hydration, dynamic virtual scrolling, or canvas-rendered interfaces may present incomplete accessibility trees.

• iframes and Shadow DOM: Cross-origin iframes cannot be traversed by top-level accessibility trees without frame switching. Closed shadow roots are hidden from standard DOM traversals.

• Authentication & CAPTCHAs: Cognitive Browser does not solve Cloudflare challenges, Google reCAPTCHA, or multi-factor authentication puzzles.

• Reflex Threshold Tuning: Setting REFLEX_CONFIDENCE_THRESHOLD higher increases the rate of Cortex LLM escalations (spending more tokens and time in exchange for generative verification on borderline matches), while lower thresholds prioritize local zero-token execution. See docs/threshold-analysis.md for empirical sweep data across thresholds.

• Pre-scripted vs Open-Ended Navigation: Cognitive Browser executes step sequences; formulating the high-level plan from an open-ended goal requires an upstream LLM agent.

Comparison with Related Projects

Security

Cognitive Browser handles untrusted web content and executes browser interactions:

• SSRF Protection: Multi-hop redirects (public -> public -> private) are NOT blocked by the application (a single intermediate public hop bypasses route interception), so network-level egress filtering is strictly required in multi-tenant environments until proxy-based connection enforcement is added. Direct requests, subresources, XHR/fetch, WebSockets, and single-hop HTTP redirects to loopback (127.0.0.1, [::1]), private IP ranges (RFC 1918, fc00::/7), link-local (fe80::/10), IPv4-mapped IPv6, and cloud metadata (169.254.169.254) are intercepted and aborted before connection (net::ERR_BLOCKED_BY_CLIENT) via Playwright (context.route("**/*") and route_web_socket), and service workers are blocked (service_workers="block"). Multi-hop bypass occurs due to a limitation of Playwright's context.route approach, where the driver automatically continues redirected requests without re-dispatching secondary hops to user handlers; closing this limitation requires a local egress proxy validating destination IPs upon connection (--proxy-server), which is not implemented in this version (see docs/threat-model.md). Domain allowlists can be enforced via ALLOWED_DOMAINS.

• Sensitive Data Masking: fill_value operations are masked (***) by default in logs and telemetry to prevent accidental credential leakage. Users can opt out via DISABLE_FILL_MASKING=true.

• Prompt Injection from Page Content: Untrusted webpage content and accessibility tree labels are passed to the Cortex LLM escalation tier. While inputs are length-capped and control characters are stripped (sanitize_untrusted_text), semantic prompt injection cannot be guaranteed to be filtered by text heuristics alone. Do not automate untrusted malicious websites without external container sandboxing.

• API Token Authentication: The REST API and SSE service on port 8000 support token authentication via the API_TOKEN environment variable. docker-compose.yml binds to 127.0.0.1:8000 by default. Do not expose port 8000 to public networks without setting API_TOKEN.

• For vulnerability reporting and full threat model analysis, see SECURITY.md and docs/threat-model.md.

Development & Testing

Run unit tests:

pytest tests/ -v

Run offline fixture benchmark:

python benchmarks/run_15_cases_benchmark.py --mode fixture --runs 2