crawl_website
Crawl a website to discover and scrape multiple pages, following links up to a set depth. Filter URLs with regex and use saved or custom scraping configs.
Instructions
Crawl a website to discover and scrape multiple pages. Follows links up to specified depth. The scraper_config can include a config_id to use saved settings, or full scraping options.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| depth | No | ||
| domain | Yes | Domain to crawl | |
| max_urls | No | ||
| async_mode | No | ||
| url_pattern | No | Regex pattern to filter URLs | |
| scraper_config | No | Scraping config for each page. Can include 'config_id' to use a saved config, or full options like mode, output, extract_scheme, etc. |