sitemap_crawler
Recursively crawl a website to generate a comprehensive sitemap with all endpoints, links, and visible text. Detect broken links and navigation issues.
Instructions
Crawl a website recursively to generate a comprehensive sitemap with all endpoints, links, and visible text content. Perfect for detecting inconsistent content, broken links, and navigation issues.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL to crawl (e.g., http://localhost:3000, https://example.com) | |
| maxDepth | No | Maximum depth to crawl (default: 3) | |
| includeText | No | Include visible text content from each page (default: true) | |
| sameDomainOnly | No | Only crawl links from the same domain (default: true) | |
| outputFormat | No | Output format for the sitemap (default: markdown) |