crawl
Traverse a website from a seed URL, follow discovered links, and return an array of page URLs, titles, and content. Use link_pattern to restrict crawling to specific URL patterns.
Instructions
Crawl a website starting from a seed URL, following discovered links across multiple pages. Returns an array of {url, title, content} objects. Use link_pattern to restrict crawling to specific URL patterns (e.g. "https://example.com/blog/**").
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL to crawl from | |
| label | No | Label to attach to the initial request for categorising discovered links | |
| max_pages | No | Maximum number of pages to crawl. Default: 10, max: 50 | |
| render_js | No | Whether to use Playwright (JS rendering) or Cheerio (fast HTML-only). Default: true | |
| link_pattern | No | Glob pattern to filter discovered URLs (e.g. "https://example.com/blog/**"). Only URLs matching this pattern will be enqueued. | |
| link_selector | No | CSS selector for links to follow. Default: "a" | a |
| extract_content | No | Strip navigation, footer, and ads — return main content only. Default: true |