tavily-crawl
Starts a structured web crawl from a given URL, following internal links to discover and extract content. Control crawl depth, breadth, and focus on specific site sections.
Instructions
A powerful web crawler that initiates a structured web crawl starting from a specified base URL. The crawler expands from that point like a graph, following internal links across pages. You can control how deep and wide it goes, and guide it to focus on specific sections of the site.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "tavily-crawl"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The root URL to begin the crawl | |
| limit | No | Total number of links the crawler will process before stopping | |
| format | No | The format of the extracted web page content. markdown returns content in markdown format. text returns plain text and may increase latency. | markdown |
| max_depth | No | Max depth of the crawl. Defines how far from the base URL the crawler can explore. | |
| max_breadth | No | Max number of links to follow per level of the tree (i.e., per page) | |
| instructions | No | Natural language instructions for the crawler. Instructions specify which types of pages the crawler should return. | |
| select_paths | No | Regex patterns to select only URLs with specific path patterns (e.g., /docs/.*, /api/v1.*) | |
| extract_depth | No | Advanced extraction retrieves more data, including tables and embedded content, with higher success but may increase latency | basic |
| allow_external | No | Whether to return external links in the final response | |
| select_domains | No | Regex patterns to restrict crawling to specific domains or subdomains (e.g., ^docs\.example\.com$) | |
| include_favicon | No | Whether to include the favicon URL for each result |