Crawl a website
web_crawl_startCrawl a website from a start URL, following links and reading pages in a real browser as markdown, text, or HTML. Use it to extract site content while respecting robots.txt and host limits.
Instructions
Crawl a website from a start URL (no session needed): follow its links, read each page in a real browser and return the pages as markdown, text or HTML. Respects robots.txt and stays on the start page's host unless sameHost is false. Waits up to waitSeconds (default 45) and returns the pages read so far; while it is still running, web_crawl_get gives the rest. Public pages only.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| format | No | ||
| exclude | No | Never follow URLs that match one of these regular expressions | |
| include | No | Only follow URLs that match one of these regular expressions | |
| maxDepth | No | Link depth from the start page, 0 to 10 (default 3) | |
| maxPages | No | Pages to read, 1 to 1000 (default 20; the plan's crawl limit applies) | |
| sameHost | No | Stay on the start page's host (default true) | |
| waitSeconds | No | How long to wait for the crawl before answering (default 45) |