Crawl a website starting from a seed URL
web_crawlCrawl a website starting from a URL to a specified depth or page limit, returning page titles, links, and excerpts. Full page markdown is cached for later retrieval.
Instructions
Perform a BFS crawl starting at startUrl up to maxDepth/maxPages. Returns page titles, link structures, and excerpts, while caching full markdown for web_crawl_get_page.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| maxDepth | No | max link depth to traverse (0 = startUrl only) | |
| maxPages | No | max total pages to visit | |
| startUrl | Yes | ||
| contentDepth | No | summary | |
| ignoreRobots | No | ||
| sameHostOnly | No | restrict crawl to pages on the same hostname | |
| excludePatterns | No | pathname glob patterns URLs must NOT match | |
| includePatterns | No | pathname glob patterns URLs must match, e.g. '/docs/*' |