multi_url_crawl
Crawl multiple websites simultaneously using pattern-based configurations to extract and analyze web content, PDFs, documents, and videos with AI-powered summarization.
Instructions
Multi-URL crawl with pattern-based config. Max 5 URL patterns per call.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url_configurations | Yes | URL-config mapping (max 5 URLs). Example: {'https://site1.com': {'wait_for_js': true}} | |
| pattern_matching | No | Pattern: 'wildcard' or 'regex' (default: wildcard) | wildcard |
| default_config | No | Default config | |
| base_timeout | No | Timeout per URL (default: 30) | |
| max_concurrent | No | Max concurrent (default: 3) |