webcrawl_crawl
Crawl web pages starting from a URL, discovering linked pages up to a specified depth. Control the crawl with limits on pages and URL patterns.
Instructions
Crawl multiple pages starting from a URL.
Uses BFS to discover and fetch pages up to max_depth links away. Respects rate limiting between requests.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL | |
| max_pages | No | Maximum number of pages to fetch (default: 10) | |
| max_depth | No | Maximum link depth from start (default: 2) | |
| include_patterns | No | Glob patterns for URLs to include (e.g., ["*/docs/*"]) |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |