crawl_url
Crawl multiple web pages from a starting URL and retrieve all content as clean Markdown. Ideal for capturing entire documentation sections or blog archives in one request.
Instructions
Crawl multiple pages starting from a URL and return all content as Markdown.
Best for: Fetching entire documentation sections, blog archives, or multiple related pages at once.
Not recommended for: Single pages (use scrape_url — it's faster). Large sites without filters (responses can be very large and exceed token limits).
Tip: Use include_patterns to scope the crawl (e.g. "https://example.com/docs/**").
Args: url: The starting URL to crawl. limit: Maximum number of pages to crawl (default: 10, max: 100000). depth: Maximum link depth from the starting URL (default: 1). include_subdomains: If true, follows links to subdomains. include_external_links: If true, follows links to external domains. include_patterns: Only visit URLs matching these wildcard patterns. exclude_patterns: Skip URLs matching these wildcard patterns. render: If true (default), renders JavaScript. Set false for faster static fetch.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| depth | No | ||
| limit | No | ||
| render | No | ||
| exclude_patterns | No | ||
| include_patterns | No | ||
| include_subdomains | No | ||
| include_external_links | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |