spider_crawl
Crawl websites to extract content from multiple pages with smart JS rendering. Returns data in markdown, HTML, or text formats up to specified depth and limit.
Instructions
Crawl a website and extract content from multiple pages. Follows links up to the specified depth/limit. Returns content in markdown, HTML, text, or other formats. Powered by Spider — crawls 100K+ pages/sec with smart JS rendering.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| tld | No | Follow top-level domain variations | |
| url | Yes | The URL to process. Comma-separate for multiple URLs. | |
| cron | No | Schedule recurring crawls | |
| cache | No | HTTP caching. true/false or {maxAge, allowStale, period} | |
| delay | No | Delay between requests in ms (max 60000). Disables concurrency | |
| depth | No | Maximum crawl depth from start URL. Default: 25 | |
| limit | No | Maximum pages to crawl. 0 for unlimited. Default: 0 | |
| proxy | No | Proxy pool type. residential (x1.2), mobile (x2), isp/datacenter (x1.2) | |
| budget | No | Page budget per URL path (e.g. {'*': 100, '/blog': 20}) | |
| locale | No | Browser locale (e.g. 'en-US') | |
| cookies | No | HTTP cookies for authenticated scraping | |
| request | No | Request type: http (fast), chrome (JS rendering), smart (auto-detect). Default: smart | |
| session | No | Persist cookies across requests. Default: true | |
| sitemap | No | Discover pages via sitemap.xml | |
| timeout | No | Overall request timeout in milliseconds | |
| metadata | No | Collect page metadata (title, description, keywords) | |
| sitemaps | No | Specific sitemap URLs to use | |
| timezone | No | Browser timezone | |
| viewport | No | Device viewport settings | |
| wait_for | No | Chrome-only readiness gate (request: 'chrome' or 'smart'). Object with any of: selector ({selector, timeout}), idle_network ({timeout}), idle_network0, almost_idle_network0, dom ({selector, timeout}), delay ({timeout}), page_navigations (bool). Each timeout is a Rust Duration: { secs, nanos }. | |
| webhooks | No | Webhook URLs for async events (on_find, on_credits_depleted) | |
| blacklist | No | URL path patterns to exclude (supports regex) | |
| block_ads | No | Block advertisements. Default: true | |
| whitelist | No | URL path patterns to include (supports regex) | |
| automation | No | Browser automation actions to run before extraction (Click, Fill, Wait, Scroll) | |
| clean_html | No | Strip unwanted HTML attributes (class, style, etc.) | |
| filter_svg | No | Remove SVG elements from markup before processing | |
| subdomains | No | Follow subdomains | |
| user_agent | No | Custom HTTP user agent string | |
| fingerprint | No | Advanced browser fingerprint detection. Default: true | |
| readability | No | Use readability algorithm for cleaner content extraction | |
| storageless | No | Prevent data storage. Default: true | |
| chunking_alg | No | Segment content: bysentence, bylines, bycharacterlength, bywords | |
| country_code | No | ISO country code for geo-located proxy (e.g. 'gb', 'us') | |
| remote_proxy | No | External proxy URL. Saves 50% on data transfer credits | |
| event_tracker | No | Track detailed request/response events | |
| filter_images | No | Remove image elements from markup before processing | |
| preserve_host | No | Preserve the HOST header on redirects | |
| proxy_enabled | No | Enable premium proxies. Multiplies credit cost by 1.5x | |
| return_format | No | Output format. Default: raw | |
| root_selector | No | Root CSS selector to scope extraction (e.g. "#main-content") | |
| full_resources | No | Download all resources including images, CSS, JS | |
| respect_robots | No | Obey robots.txt rules. Default: true | |
| return_cookies | No | Include HTTP response cookies | |
| return_headers | No | Include HTTP response headers | |
| block_analytics | No | Block analytics scripts. Default: true | |
| redirect_policy | No | How to handle redirects. Default: Loose | |
| request_timeout | No | Per-request timeout in milliseconds | |
| exclude_selector | No | CSS selector for elements to exclude from output | |
| external_domains | No | External domains to follow. Use ['*'] to allow all | |
| filter_main_only | No | Keep only main content. Default: enabled | |
| return_json_data | No | Extract JSON-LD and structured data from pages | |
| block_stylesheets | No | Block CSS stylesheets. Default: true | |
| concurrency_limit | No | Max concurrent requests to the target site | |
| disable_intercept | No | Disable request interception | |
| filter_output_svg | No | Remove SVGs from output | |
| return_page_links | No | Include links found on each page in the response | |
| run_in_background | No | Run asynchronously. Requires webhooks or storageless=false | |
| css_extraction_map | No | CSS/XPath selectors for structured extraction. Shape: { '<url-path>': [{ name, selectors: [...] }] }. '/' matches all paths. Results returned under css_extracted. | |
| filter_output_images | No | Remove images from output | |
| filter_output_main_only | No | Remove nav, aside, footer from output |