crawl
Crawl a website from a starting URL, follow links to a set depth, and return discovered pages with titles and optional content summaries. Use link filters and depth limits to control scope.
Instructions
Crawl a website starting from a URL, following links up to a specified depth. Returns a list of discovered pages with titles and optional content summaries. Respects same-domain and link filter constraints.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL for the crawl | |
| timeout | No | Timeout per page in ms (default: 30000) | |
| max_depth | No | Maximum link depth to follow (default: 2) | |
| max_pages | No | Maximum total pages to crawl (default: 10) | |
| link_filter | No | Regex pattern to filter which links to follow | |
| include_content | No | Include a markdown content summary per page (default: false) | |
| max_length_per_page | No | Max content length per page when include_content is true (default: 5000) |