alterlab_crawl
Crawl entire websites for RAG or indexing, automatically bypassing anti-bot protection. Returns markdown, text, or structured data with a crawl ID for polling.
Instructions
Crawl an entire website for RAG, index all pages on a site, or build a knowledge base from a domain. Automatically bypasses anti-bot protection on every page — works on Cloudflare-protected sites. Discovers URLs via sitemap parsing and link extraction, then scrapes each page asynchronously. Returns content as clean markdown, text, or structured data. Returns a crawl_id immediately — use alterlab_crawl_status to poll results. Use include_patterns/exclude_patterns to scope the crawl to specific sections. Use render_js='auto' for mixed sites to save 30-60% vs always rendering. Supports extraction_schema or extraction_profile to extract structured data from every page. Use cost_controls to cap total credits or pin the scraping tier for all pages. Use sitemap_path to specify a non-standard sitemap location. Use headers to inject custom HTTP headers (e.g., Authorization) into every page request.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start URL for the crawl | |
| formats | No | Output formats for each scraped page | |
| headers | No | Custom HTTP headers injected into every page request during the crawl (e.g., {"Authorization": "Bearer token"}). Maximum 50 headers. | |
| sitemap | No | Sitemap mode: include (default), skip (link extraction only), only (sitemap URLs only) | include |
| timeout | No | Per-page request timeout in seconds (1-300) | |
| wait_for | No | CSS selector to wait for before extracting each page (e.g., '#main-content'). Applied to all pages in the crawl. | |
| max_depth | No | Maximum link-following depth from start URL (0 = start page only) | |
| max_pages | No | Maximum number of pages to scrape | |
| render_js | No | Render JavaScript on crawled pages. true=always (Tier 4), false=never, auto=smart detection per page | |
| use_proxy | No | Route all crawl requests through premium proxy | |
| webhook_url | No | Webhook URL to notify on crawl completion | |
| block_images | No | Block image downloads during browser rendering on each crawled page. Reduces proxy bandwidth and speeds up crawls. Only effective with render_js=true. | |
| sitemap_path | No | Explicit path to the sitemap file (e.g., '/sitemap_index.xml'). Use when the sitemap is not at the standard /sitemap.xml location. | |
| cost_controls | No | Cost controls for the entire crawl — cap total spend or pin the scraping tier | |
| respect_robots | No | Respect robots.txt rules for the target domain | |
| max_concurrency | No | Maximum concurrent pages to scrape simultaneously | |
| exclude_patterns | No | Glob patterns — skip URLs whose path matches any (e.g., ['/tag/*', '/author/*']) | |
| extraction_model | No | Per-request LLM model override in provider-specific format (e.g. 'gpt-4o', 'claude-opus-4-5-20251101', 'llama3-70b-8192'). Overrides the model saved in your BYOK key settings for this request only. | |
| include_patterns | No | Glob patterns — only scrape URLs whose path matches at least one (e.g., ['/blog/*', '/docs/*']) | |
| extraction_schema | No | JSON schema for structured extraction on each page | |
| extraction_profile | No | Pre-defined extraction profile applied to every crawled page. 'auto' detects the page type automatically. | |
| include_subdomains | No | Include links to subdomains during discovery |