crawl_website
Discover a website's structure and pages by crawling from a starting URL. Returns a site map with statistics, customizable depth and page count.
Instructions
Crawl a website to discover its structure and pages.
Args:
start_url: Starting URL
max_pages: Maximum pages to crawl (default 50)
max_depth: Maximum link depth (default 3)
same_domain_only: Stay on same domain (default True)
Returns:
Site map with discovered pages and statistics
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| start_url | Yes | ||
| max_pages | No | ||
| max_depth | No | ||
| same_domain_only | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |