crawl
Fetch HTML content from a URL with configurable options. Optionally store the result in cloud storage for later access.
Instructions
Crawl a URL and return HTML content. Pass store=true to push the result to Crawlbase Cloud Storage instead of returning the body — the response will then contain only the RID and metadata, which can later be retrieved via storage_get.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to crawl | |
| store | No | Push the result to Crawlbase Cloud Storage. When true, the response returns only RID + metadata (no body) so the body can be fetched later via storage_get. | |
| device | No | Device type for crawling | |
| country | No | Country code for geo-targeting | |
| ajax_wait | No | Wait time for AJAX requests in milliseconds | |
| page_wait | No | Wait time for page load in milliseconds | |
| screenshot | No | Take a screenshot of the page | |
| user_agent | No | Custom user agent string |