crawl_site
Start a background website crawl with Screaming Frog, returning a crawl ID to monitor progress via crawl_status. Accepts optional config file and label for crawl limits and identification.
Instructions
Start a background Screaming Frog crawl that saves to SF's internal database.
Args: url: The URL to crawl (e.g. https://example.com) config_file: Optional path to a .seospiderconfig file for crawl settings. To limit the number of URLs crawled, set the limit in a config file (Configuration > Spider > Limits in the SF GUI) and pass it here. label: Optional label for identifying this crawl (e.g. 'freshgovjobs')
Returns: A crawl_id to use with crawl_status to check progress. The crawl runs in the background - use crawl_status to poll.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "crawl_site"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| label | No | ||
| config_file | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |