tavily_crawl
Crawl a website starting from a root URL to extract content. Configure crawl depth, breadth, and target specific paths or domains.
Instructions
Crawl a website starting from a URL. Extracts content from pages with configurable depth and breadth.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The root URL to begin the crawl | |
| limit | No | Total number of links the crawler will process before stopping | |
| format | No | The format of the extracted web page content. markdown returns content in markdown format. text returns plain text and may increase latency. | markdown |
| max_depth | No | Max depth of the crawl. Defines how far from the base URL the crawler can explore. | |
| max_breadth | No | Max number of links to follow per level of the tree (i.e., per page) | |
| instructions | No | Natural language instructions for the crawler. Instructions specify which types of pages the crawler should return. | |
| select_paths | No | Regex patterns to select only URLs with specific path patterns (e.g., /docs/.*, /api/v1.*) | |
| extract_depth | No | Advanced extraction retrieves more data, including tables and embedded content, with higher success but may increase latency | basic |
| allow_external | No | Whether to return external links in the final response | |
| select_domains | No | Regex patterns to restrict crawling to specific domains or subdomains (e.g., ^docs\.example\.com$) | |
| include_favicon | No | Whether to include the favicon URL for each result |