alterlab_crawl
Crawl an entire website and return each page’s content for RAG, indexing, or site-wide export, including JavaScript-heavy and protected sites. Returns a crawl_id for retrieving results.
Instructions
Crawl a whole website and return the content of every page, for RAG, indexing or site-wide export. Use when you need many pages from one domain; for one URL use alterlab_scrape, for a URL list only use alterlab_map. Works on JavaScript-heavy and protected sites, with anti-bot handling on every page. Returns a crawl_id immediately; results are available from GET /api/v1/crawl/{crawl_id}.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start URL for the crawl | |
| cache | No | Serve pages from the response cache when available | |
| delay | No | Politeness delay between page requests, in seconds (0-30) | |
| budget | No | Per-path page budget: glob pattern -> max pages, most specific pattern wins, '*' is the catch-all (e.g., {"/products/*": 100, "/blog/*": 50, "*": 5}). max_pages stays the hard cap. | |
| formats | No | Output formats for each scraped page. 'markdown' is best for LLM consumption. 'json_v2' returns a structured section tree (headings + content blocks). 'rag' returns chunked text optimized for retrieval-augmented generation. 'raw' returns the raw response body without extraction. 'content' returns body_markdown + content_hash + images + links for AI/KB pipelines. | |
| headers | No | Custom HTTP headers injected into every page request during the crawl (e.g., {"Authorization": "Bearer token"}). Maximum 50 headers. | |
| sitemap | No | Sitemap mode: include (default), skip (link extraction only), only (sitemap URLs only) | include |
| storage | No | Where crawl results are stored (API default 'redis') | |
| timeout | No | Per-page request timeout in seconds (1-300) | |
| webhook | No | Detailed webhook config (events, signing secret, headers); use instead of webhook_url | |
| priority | No | Crawl prioritisation hints (keywords, boost_patterns, demote_patterns) | |
| wait_for | No | CSS selector to wait for before extracting each page (e.g., '#main-content'). Applied to all pages in the crawl. | |
| cache_ttl | No | Cache TTL in seconds (60-86400). Requires cache=true | |
| max_depth | No | Maximum link-following depth from start URL (0 = start page only) | |
| max_pages | No | Maximum number of pages to scrape | |
| render_js | No | Render JavaScript on crawled pages. true=always (Tier 4, $0.004/page), false=never, auto=smart detection per page | |
| use_proxy | No | Route all crawl requests through premium proxy (+$0.0002/page) | |
| extraction | No | Set to 'auto' for automatic page-type detection and extraction without a user-defined schema. Mutually exclusive with extraction_schema. | |
| screenshot | No | Capture a screenshot of each crawled page (requires render_js) | |
| webhook_url | No | Webhook URL to notify on crawl completion | |
| sitemap_path | No | Explicit path to the sitemap file (e.g., '/sitemap_index.xml'). Use when the sitemap is not at the standard /sitemap.xml location. | |
| cost_controls | No | Cost controls for the entire crawl — cap total spend or pin the scraping tier | |
| force_refresh | No | Bypass the cache and fetch every page fresh | |
| depth_strategy | No | Link-discovery strategy (API default 'sequential') | |
| respect_robots | No | Respect robots.txt rules for the target domain | |
| change_tracking | No | Track page changes between crawls | |
| max_concurrency | No | Maximum concurrent pages to scrape simultaneously | |
| exclude_patterns | No | Glob patterns — skip URLs whose path matches any (e.g., ['/tag/*', '/author/*']) | |
| extraction_model | No | Per-request LLM model override in provider-specific format (e.g. 'gpt-4o', 'claude-opus-4-5-20251101', 'llama3-70b-8192'). Overrides the model saved in your BYOK key settings for this request only. | |
| include_patterns | No | Glob patterns — only scrape URLs whose path matches at least one (e.g., ['/blog/*', '/docs/*']) | |
| extraction_schema | No | JSON schema for structured extraction on each page | |
| extraction_profile | No | Pre-defined extraction profile applied to every crawled page. 'auto' detects the page type automatically. | |
| include_subdomains | No | Include links to subdomains during discovery | |
| extraction_provider | No | BYOK LLM provider used for extraction on each page | |
| extraction_template | No | Alias for extraction_profile (mutually exclusive with it) |