Website Crawler
website_crawlerFetches raw HTML from any webpage in request mode without rendering, enabling scraping, audits, monitoring, and link or metadata extraction.
Instructions
Website Crawler. Website Crawler API fetches raw HTML from any webpage in request mode (no rendering) for fast scraping, audits, monitoring, and link/metadata extraction. Credit cost: 1 credit per call.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | A website URL with either the http or https protocol. | |
| fields | No | Comma-separated list of fields to keep in the response, with dot notation for nested fields (e.g. name,location.city). Does not change the credit cost. | |
| allow_redirects | No | If set to false, do not follow redirects. Default: true. | |
| return_page_source | No | If set to true, the response will be the HTML source of the page. Default: false. |