spider_crawl
Crawl a website to extract content from multiple pages, returning data in markdown, HTML, text, or other specified formats.
Instructions
Crawl a website and extract content from multiple pages. Returns page content in the specified format (markdown, HTML, text, etc.). Powered by Spider - the fastest web crawler at 100K+ pages/sec.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| tld | No | Allow TLDs | |
| url | Yes | The URL to crawl. Can be comma-separated for multiple URLs. | |
| cron | No | Schedule crawl | |
| cache | No | HTTP caching. Object: {maxAge, allowStale, period} | |
| delay | No | Crawl delay in ms (max 60000). Disables concurrency | |
| depth | No | Maximum crawl depth. Default: 25. 0 for no limit. | |
| limit | No | Maximum pages to crawl per website. 0 for all pages. Default: 0 | |
| proxy | No | Proxy pool type | |
| budget | No | Crawl budget by path (e.g. {'*':100}) | |
| locale | No | Locale for content (e.g. 'en-US') | |
| cookies | No | HTTP cookies for SSR authentication | |
| request | No | Request type. Default: smart | |
| session | No | Persist HTTP headers and cookies. Default: true | |
| sitemap | No | Include links from sitemaps | |
| timeout | No | Overall request timeout | |
| metadata | No | Collect page metadata (title, description, keywords) | |
| sitemaps | No | Specific sitemap URLs to use | |
| timezone | No | Timezone for content | |
| viewport | No | Device viewport settings | |
| wait_for | No | Chrome-only readiness gate (request: 'chrome' or 'smart'). Object with any of: selector ({selector, timeout}), idle_network ({timeout}), idle_network0, almost_idle_network0, dom ({selector, timeout}), delay ({timeout}), page_navigations (bool). Each timeout is a Rust Duration: { secs, nanos }. | |
| webhooks | No | Webhook config for events (on_find, on_credits_depleted, etc.) | |
| blacklist | No | Paths to exclude (supports regex) | |
| block_ads | No | Block advertisements. Default: true | |
| whitelist | No | Paths to include (supports regex) | |
| automation | No | Web automation actions (Click, Fill, Wait, Scroll, etc.) | |
| clean_html | No | Clean HTML of unwanted attributes | |
| filter_svg | No | Filter SVG elements from markup | |
| subdomains | No | Allow subdomains | |
| user_agent | No | Custom HTTP user agent | |
| fingerprint | No | Advanced fingerprint detection for Chrome. Default: true | |
| readability | No | Use readability algorithm for content preprocessing | |
| storageless | No | Prevent data storage. Default: true | |
| chunking_alg | No | Segment content: bysentence, bylines, bycharacterlength, bywords | |
| country_code | No | ISO country code for proxy (e.g. 'gb') | |
| remote_proxy | No | External proxy connection URL | |
| event_tracker | No | Track requests, responses, automation | |
| filter_images | No | Filter image elements from markup | |
| preserve_host | No | Preserve HOST header | |
| proxy_enabled | No | Enable premium proxies. Multiplies cost by 1.5x | |
| return_format | No | Output format. Default: raw | |
| root_selector | No | Root CSS query selector for content extraction | |
| full_resources | No | Download all website resources including assets | |
| respect_robots | No | Respect robots.txt. Default: true | |
| return_cookies | No | Return HTTP response cookies | |
| return_headers | No | Return HTTP response headers | |
| block_analytics | No | Block analytics. Default: true | |
| redirect_policy | No | Redirect policy. Default: Loose | |
| request_timeout | No | HTTP request timeout in ms | |
| exclude_selector | No | CSS selector for content to ignore | |
| external_domains | No | External domains to include. Use ['*'] for all | |
| filter_main_only | No | Filter to main content only. Default: enabled | |
| return_json_data | No | Return JSON data from SSR scripts | |
| block_stylesheets | No | Block stylesheets. Default: true | |
| concurrency_limit | No | Concurrency limit for slower websites | |
| disable_intercept | No | Disable request interception | |
| filter_output_svg | No | Filter SVG tags from output | |
| return_page_links | No | Return links found on each page | |
| run_in_background | No | Run in background. Requires storageless=false or webhooks | |
| css_extraction_map | No | CSS/XPath selectors for structured extraction. Shape: { '<url-path>': [{ name, selectors: [...] }] }. '/' matches all paths. Results returned under css_extracted. | |
| request_max_retries | No | Maximum request retries | |
| filter_output_images | No | Filter images from output | |
| request_redirect_limit | No | Maximum redirects to follow | |
| filter_output_main_only | No | Filter nav, aside, footer from output |