Crawl a website to Markdown
crawlCrawl a website using breadth-first link traversal and return each page as clean Markdown. Use to ingest documentation sites or sections of a site.
Instructions
Breadth-first crawl a site starting from a URL and return each page as clean Markdown. Follows in-domain links up to a depth/page limit. Same proxy and block-handling as scrape. Use to ingest a docs site or section of a site.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute http(s) URL to start crawling from. | |
| render | No | Render mode per page (see scrape). Default 'auto'. 'browser' is slower but handles JS sites. | |
| maxDepth | No | Link depth from the seed. Default 2. | |
| maxPages | No | Max pages to fetch. Default 20. | |
| sameDomain | No | Only follow links on the seed's host. Default true. |