discover_subpages
Crawl and identify all subpages from a specified URL, with customizable depth and page limits. Ideal for mapping website structures or extracting URLs for analysis.
Instructions
Discover all subpages from a given URL up to a specified depth.
Args: url: The base URL to start crawling from max_depth: Maximum depth to crawl (default: 2) max_pages: Maximum number of pages to discover (default: 100)
Returns: List of discovered URLs
Input Schema
Name | Required | Description | Default |
---|---|---|---|
max_depth | No | ||
max_pages | No | ||
url | Yes |