HyperCrawl

Self-hosted, cookie-aware web-automation engine. Turns any website into callable MCP tools (DOM-selector based), runs them headlessly with stealth on our own servers, and pulls live auth cookies on-demand from a real logged-in browser. Internal NEB tool — works alongside Apify.

Status: production on the hypercrawl-vm GCE VM (Caddy auto-TLS + Firebase-Auth gate, hypercrawl.service). Capabilities: the acquire() capability ladder (hypercrawl_acquire — API replay/hydration → learned tool → DOM agent → set-of-marks → vision → specialist), multi-page discovery (hypercrawl_map, hypercrawl_crawl_site), cross-crawl change tracking (stateless previousHashes or DB-backed track streams, both intents), web search (hypercrawl_search), plus HTTP MCP transport, ProxyPool, and the Apify fallback adapter. See docs/USAGE.md (tools) and deploy/gcp/README.md (deploy/update — includes the required npm install + npx playwright install steps).

Browser sessions

Beyond one-shot acquisition, HyperCrawl exposes a stateful browser surface: 15 hypercrawl_browser_* MCP tools that drive a single live page across many calls, so an agent can read a page, click into it, fill a form, and submit — each step landing on the same tab. hypercrawl_browser_open returns a sessionId every other tool takes.

Sessions are a bounded resource: each pins a real Chromium context, so the server caps concurrency (HYPERCRAWL_MAX_SESSIONS, default 4) and sweeps sessions idle past five minutes. Close yours when done. An unknown sessionId is a loud error, never a silently fresh page.

Alongside them: hypercrawl_analyze / hypercrawl_quality (page structure, and whether a stored template's selectors still resolve against the live page — the signal a redesign broke it), hypercrawl_auth_save / _auth_list, and hypercrawl_metrics / _events / _speed / _setup for in-process observability. Tool reference and every env var: docs/USAGE.md.

Two capabilities are wired only when the server is started via hypercrawl serve — the auth pair and hypercrawl_setup — because their implementations live in the CLI package. Unwired, they say so rather than returning an empty result that would read as "nothing configured".

Related MCP server: web-scraper-server

Deploy to GCP

To run the full engine (including authed cookie-pull scraping) on a dedicated GCE VM with Caddy auto-TLS and a Firebase Auth token gate, use the idempotent provisioning scripts and operator runbook in deploy/gcp/README.md. Design: docs/superpowers/specs/2026-05-29-hypercrawl-gcp-deploy-design.md.