Crawl
crawlFetch one or more URLs and extract page content as markdown or JSON for research, monitoring, or data analysis.
Instructions
Crawl one or more URLs and extract content as markdown or JSON.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | List of URLs to crawl (required). Accepts a single URL or multiple URLs. | |
| timeout | No | Per-URL timeout in seconds (default: 15, must be >= 1) | |
| dedup_mode | No | Markdown dedup mode - "exact" (default) or "off" | exact |
| concurrency | No | Maximum concurrent crawls (default: 3) | |
| remove_links | No | Remove all links from the markdown output (default: false) | |
| output_format | No | 'markdown' (default) or 'json' - markdown: Clean concatenated markdown with URL headers and timestamps - json: Full JSON with metadata, references, and statistics | markdown |
| storage_state | No | Path to Playwright storage_state JSON for authenticated crawling |