crawler_job_start
Start an asynchronous crawl from a URL, following links matching a regex to scrape multiple pages. Returns a job ID for tracking progress via polling or webhook.
Instructions
Start a ScraperAPI crawl job from a starting URL.
Submits an asynchronous crawl that follows links matching a regex outward
from start_url and scrapes each page. Returns immediately with a job id and
status (e.g. {"status": "initiated", "jobId": "..."}); the crawl runs in the
background. Poll 'crawler_job_status' with the returned job id to track
progress, and/or provide a callback_url webhook to receive results.
When to use:
- Crawling multiple linked pages of a site (not a single known URL)
- Building a dataset by following links to a depth or credit budget
When NOT to use:
- Fetching one known URL (use the 'scrape' tool)
- A structured marketplace/SERP lookup (use the relevant SDE tool)
Args:
params (CrawlerJobStartParams): start_url and url_regexp_include are
required; provide either max_depth or crawl_budget to bound the
crawl. Optional: url_regexp_exclude, api_params, callback_url,
additional_data, schedule, enabled.
Returns:
str: JSON with the job id and initial status.
Raises:
ToolError: If the API key is missing, the rate limit is exceeded, the
inputs are invalid, or the request fails.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| params | Yes |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |