Initiate a web crawl from a base URL
tavily_post_crawlTavily Crawl is a graph-based website traversal tool that can explore hundreds of paths in parallel with built-in extraction and intelligent discovery. Uses the caller's OpenAny gateway credentials. Charges and authorization are determined by the gateway.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The root URL to begin the crawl. | |
| limit | No | Total number of links the crawler will process before stopping. | |
| format | No | The format of the extracted web page content. `markdown` returns content in markdown format. `text` returns plain text and may increase latency. | |
| max_depth | No | Max depth of the crawl. Defines how far from the base URL the crawler can explore. | |
| max_breadth | No | Max number of links to follow per level of the tree (i.e., per page). | |
| instructions | No | Natural language instructions for the crawler. When specified, the mapping cost increases to 2 API credits per 10 successful pages instead of 1 API credit per 10 pages. | |
| select_paths | No | Regex patterns to select only URLs with specific path patterns (e.g., `/docs/.*`, `/api/v1.*`). | |
| exclude_paths | No | Regex patterns to exclude URLs with specific path patterns (e.g., `/private/.*`, `/admin/.*`). | |
| extract_depth | No | Advanced extraction retrieves more data, including tables and embedded content, with higher success but may increase latency. `basic` extraction costs 1 credit per 5 successful extractions, while `advanced` extraction costs 2 credits per 5 successful extractions. | |
| allow_external | No | Whether to include external domain links in the final results list. | |
| include_images | No | Whether to include images in the crawl results. | |
| select_domains | No | Regex patterns to select crawling to specific domains or subdomains (e.g., `^docs\.example\.com$`). | |
| exclude_domains | No | Regex patterns to exclude specific domains or subdomains from crawling (e.g., `^private\.example\.com$`). | |
| include_favicon | No | Whether to include the favicon URL for each result. | |
| chunks_per_source | No | Chunks are short content snippets (about 800 characters each) pulled directly from the source. Use `chunks_per_source` to define the maximum number of relevant chunks returned per source and to control the `raw_content` length. Chunks will appear in the `raw_content` field as: `<chunk 1> [...] <chunk 2> [...] <chunk 3>`. If a page splits into no more chunks than `chunks_per_source`, its full content is returned. Available only when `instructions` are provided. Must be between 1 and 5. |