crawl_site
Start a background Screaming Frog crawl to analyze website structure and SEO data, saving results to an internal database for later analysis.
Instructions
Start a background Screaming Frog crawl that saves to SF's internal database.
Args: url: The URL to crawl (e.g. https://example.com) config_file: Optional path to a .seospiderconfig file for crawl settings (including crawl limits) label: Optional label for identifying this crawl (e.g. 'freshgovjobs')
Returns: A crawl_id to use with crawl_status to check progress. The crawl runs in the background - use crawl_status to poll.
Note: To limit the number of URLs crawled, export a .seospiderconfig from the SF GUI with the desired crawl limit, then pass it via config_file.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| config_file | No | ||
| label | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |