# HyperCrawl

Self-hosted, cookie-aware web-automation engine. Turns any website into callable MCP tools
(DOM-selector based), runs them headlessly with stealth on our own servers, and pulls live
auth cookies on-demand from a real logged-in browser. Internal NEB tool — works alongside Apify.

Status: production on the `hypercrawl-vm` GCE VM (Caddy auto-TLS + Firebase-Auth gate, `hypercrawl.service`). Capabilities: the `acquire()` capability ladder (`hypercrawl_acquire` — API replay/hydration → learned tool → DOM agent → set-of-marks → vision → specialist), multi-page discovery (`hypercrawl_map`, `hypercrawl_crawl_site`), cross-crawl change tracking (stateless `previousHashes` or DB-backed `track` streams, both intents), web search (`hypercrawl_search`), plus HTTP MCP transport, ProxyPool, and the Apify fallback adapter. See `docs/USAGE.md` (tools) and `deploy/gcp/README.md` (deploy/update — includes the required `npm install` + `npx playwright install` steps).

## Browser sessions

Beyond one-shot acquisition, HyperCrawl exposes a **stateful browser surface**: 15
`hypercrawl_browser_*` MCP tools that drive a single live page across many calls, so an
agent can read a page, click into it, fill a form, and submit — each step landing on the
same tab. `hypercrawl_browser_open` returns a `sessionId` every other tool takes.

Sessions are a bounded resource: each pins a real Chromium context, so the server caps
concurrency (`HYPERCRAWL_MAX_SESSIONS`, default 4) and sweeps sessions idle past five
minutes. Close yours when done. An unknown `sessionId` is a loud error, never a silently
fresh page.

Alongside them: `hypercrawl_analyze` / `hypercrawl_quality` (page structure, and whether a
stored template's selectors still resolve against the live page — the signal a redesign
broke it), `hypercrawl_auth_save` / `_auth_list`, and `hypercrawl_metrics` / `_events` /
`_speed` / `_setup` for in-process observability. Tool reference and every env var:
`docs/USAGE.md`.

Two capabilities are wired only when the server is started via `hypercrawl serve` — the
auth pair and `hypercrawl_setup` — because their implementations live in the CLI package.
Unwired, they say so rather than returning an empty result that would read as "nothing
configured".

## Deploy to GCP

To run the full engine (including authed cookie-pull scraping) on a dedicated GCE VM with Caddy
auto-TLS and a Firebase Auth token gate, use the idempotent provisioning scripts and operator
runbook in [`deploy/gcp/README.md`](deploy/gcp/README.md). Design:
[`docs/superpowers/specs/2026-05-29-hypercrawl-gcp-deploy-design.md`](docs/superpowers/specs/2026-05-29-hypercrawl-gcp-deploy-design.md).