crawl
Visit one or more URLs, extract the page content, and return it as markdown or JSON for further processing.
Instructions
Crawl one or more URLs and extract content as markdown or JSON.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | List of URLs to crawl (required). Accepts a single URL or multiple URLs. | |
| timeout | No | Per-URL timeout in seconds (default: 15, must be >= 1) | |
| dedup_mode | No | Markdown dedup mode - "exact" (default) or "off" | exact |
| concurrency | No | Maximum concurrent crawls (default: 3) | |
| remove_links | No | Remove all links from the markdown output (default: false) | |
| output_format | No | 'markdown' (default) or 'json' - markdown: Clean concatenated markdown with URL headers and timestamps - json: Full JSON with metadata, references, and statistics | markdown |
| storage_state | No | Path to Playwright storage_state JSON for authenticated crawling |