flycrawl_crawl
Recursively crawl a website's internal links up to a set depth and page limit, converting full page bodies to Markdown while staying within the target domain.
Instructions
Recursively crawl multiple pages of an entire website following internal links up to a configured depth and page limit.
BEHAVIOR & SAFETY: Read-only web traversal. Strict domain boundary enforcement; will not traverse outside the target domain. AUTHENTICATION: Requires valid FLYCRAWL_API_KEY. RATE LIMITS & BILLING: Deducts 1 credit per successfully crawled page. 0 credits billed for failed pages. DYNAMIC JS RENDERING: Renders dynamic JavaScript on each crawled page with automated anti-bot bypass. DIFFERENCE FROM flycrawl_map: flycrawl_map only discovers URLs without downloading page contents; flycrawl_crawl fetches and converts full page bodies to Markdown. ERROR HANDLING: Returns aggregated Markdown or structured error JSON on network or authentication failure.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting root URL of the website to crawl. | |
| depth | No | Maximum traversal depth from root (0-3). Default is 1. | |
| limit | No | Maximum pages to crawl (1-50). Default is 10. |