Skip to main content
Glama

AIsa Web Search & Research

Start Crawl Process

post_tavily_crawl
Destructive

Walk a site from a root url and return the content of the pages it finds. Steer it with natural-language instructions plus regex path and domain filters, and bound it with max_depth, max_breadth and limit. Returns base_url and results[] with url and raw_content. Measured at about 4.5 seconds for a 3-page limit; cost and time grow with the bounds you set, so set them. Use it for broad coverage of one site — documentation, a catalogue, a competitor's blog. It answers synchronously, which post_firecrawl_crawl does not: that one runs as a background job and suits crawls too large to wait on. For a handful of known pages post_tavily_extract is far cheaper; to size a site before paying to crawl it, run post_tavily_map first.

Usage-based billing; Default request estimate: $0.024 USD (not a fixed price). Declared unit rate: $0.008 USD per tavily_credit. Account-specific pricing and final settlement may differ.

Input Schema

NameRequiredDescriptionDefault
bodyYes

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed20 schema fields changed
    • addedInput schema / additionalProperties
      Added value: +false
    • removedInput schema / properties / allow_external
      Removed value: -{
      -  "default": true,
      -  "description": "Include external domain links in the final results list.",
      -  "type": "boolean"
      -}
    • addedInput schema / properties / body
      Added value: +{
      +  "properties": {
      +    "allow_external": {
      +      "default": true,
      +      "description": "Include external domain links in the final results list.",
      +      "type": "boolean"
      +    },
      +    "chunks_per_source": {
      +      "default": 3,
      +      "description": "Maximum number of relevant chunks returned per source.",
      +      "maximum": 5,
      +      "minimum": 1,
      +      "type": "integer"
      +    },
      +    "exclude_domains": {
      +      "description": "Regex patterns to exclude specific domains or subdomains from crawling.",
      +      "items": {
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    "exclude_paths": {
      +      "description": "Regex patterns to exclude URLs with specific path patterns.",
      +      "items": {
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    "extract_depth": {
      +      "default": "basic",
      +      "description": "Depth of the extraction process.",
      +      "enum": [
      +        "basic",
      +        "advanced"
      +      ],
      +      "type": "string"
      +    },
      +    "format": {
      +      "default": "markdown",
      +      "description": "Format of the extracted web page content.",
      +      "enum": [
      +        "markdown",
      +        "text"
      +      ],
      +      "type": "string"
      +    },
      +    "include_favicon": {
      +      "default": false,
      +      "description": "Include the favicon URL for each result.",
      +      "type": "boolean"
      +    },
      +    "include_images": {
      +      "default": false,
      +      "description": "Include images in the crawl results.",
      +      "type": "boolean"
      +    },
      +    "include_usage": {
      +      "default": false,
      +      "description": "Include credit usage information in the response.",
      +      "type": "boolean"
      +    },
      +    "instructions": {
      +      "description": "Natural language instructions for the crawler.",
      +      "type": "string"
      +    },
      +    "limit": {
      +      "default": 50,
      +      "description": "Total number of links the crawler will process before stopping.",
      +      "minimum": 1,
      +      "type": "integer"
      +    },
      +    "max_breadth": {
      +      "default": 20,
      +      "description": "Max number of links to follow per level of the tree.",
      +      "maximum": 500,
      +      "minimum": 1,
      +      "type": "integer"
      +    },
      +    "max_depth": {
      +      "default": 1,
      +      "description": "Max depth of the crawl.",
      +      "maximum": 5,
      +      "minimum": 1,
      +      "type": "integer"
      +    },
      +    "select_domains": {
      +      "description": "Regex patterns to select crawling to specific domains or subdomains.",
      +      "items": {
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    "select_paths": {
      +      "description": "Regex patterns to select only URLs with specific path patterns.",
      +      "items": {
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    "timeout": {
      +      "default": 150,
      +      "description": "Maximum time in seconds to wait for the crawl operation.",
      +      "format": "float",
      +      "maximum": 150,
      +      "minimum": 10,
      +      "type": "number"
      +    },
      +    "url": {
      +      "description": "The root URL to begin the crawl.",
      +      "example": "docs.tavily.com",
      +      "type": "string"
      +    }
      +  },
      +  "required": [
      +    "url"
      +  ],
      +  "type": "object"
      +}
    • removedInput schema / properties / chunks_per_source
      Removed value: -{
      -  "default": 3,
      -  "description": "Maximum number of relevant chunks returned per source.",
      -  "maximum": 5,
      -  "minimum": 1,
      -  "type": "integer"
      -}
    • removedInput schema / properties / exclude_domains
      Removed value: -{
      -  "description": "Regex patterns to exclude specific domains or subdomains from crawling.",
      -  "items": {
      -    "type": "string"
      -  },
      -  "type": "array"
      -}
    • removedInput schema / properties / exclude_paths
      Removed value: -{
      -  "description": "Regex patterns to exclude URLs with specific path patterns.",
      -  "items": {
      -    "type": "string"
      -  },
      -  "type": "array"
      -}
    • removedInput schema / properties / extract_depth
      Removed value: -{
      -  "default": "basic",
      -  "description": "Depth of the extraction process.",
      -  "enum": [
      -    "basic",
      -    "advanced"
      -  ],
      -  "type": "string"
      -}
    • removedInput schema / properties / format
      Removed value: -{
      -  "default": "markdown",
      -  "description": "Format of the extracted web page content.",
      -  "enum": [
      -    "markdown",
      -    "text"
      -  ],
      -  "type": "string"
      -}
    • removedInput schema / properties / include_favicon
      Removed value: -{
      -  "default": false,
      -  "description": "Include the favicon URL for each result.",
      -  "type": "boolean"
      -}
    • removedInput schema / properties / include_images
      Removed value: -{
      -  "default": false,
      -  "description": "Include images in the crawl results.",
      -  "type": "boolean"
      -}
    • removedInput schema / properties / include_usage
      Removed value: -{
      -  "default": false,
      -  "description": "Include credit usage information in the response.",
      -  "type": "boolean"
      -}
    • removedInput schema / properties / instructions
      Removed value: -{
      -  "description": "Natural language instructions for the crawler.",
      -  "type": "string"
      -}
    • removedInput schema / properties / limit
      Removed value: -{
      -  "default": 50,
      -  "description": "Total number of links the crawler will process before stopping.",
      -  "minimum": 1,
      -  "type": "integer"
      -}
    • removedInput schema / properties / max_breadth
      Removed value: -{
      -  "default": 20,
      -  "description": "Max number of links to follow per level of the tree.",
      -  "maximum": 500,
      -  "minimum": 1,
      -  "type": "integer"
      -}
    • removedInput schema / properties / max_depth
      Removed value: -{
      -  "default": 1,
      -  "description": "Max depth of the crawl.",
      -  "maximum": 5,
      -  "minimum": 1,
      -  "type": "integer"
      -}
    • removedInput schema / properties / select_domains
      Removed value: -{
      -  "description": "Regex patterns to select crawling to specific domains or subdomains.",
      -  "items": {
      -    "type": "string"
      -  },
      -  "type": "array"
      -}
    • removedInput schema / properties / select_paths
      Removed value: -{
      -  "description": "Regex patterns to select only URLs with specific path patterns.",
      -  "items": {
      -    "type": "string"
      -  },
      -  "type": "array"
      -}
    • removedInput schema / properties / timeout
      Removed value: -{
      -  "default": 150,
      -  "description": "Maximum time in seconds to wait for the crawl operation.",
      -  "format": "float",
      -  "maximum": 150,
      -  "minimum": 10,
      -  "type": "number"
      -}
    • removedInput schema / properties / url
      Removed value: -{
      -  "description": "The root URL to begin the crawl.",
      -  "example": "docs.tavily.com",
      -  "type": "string"
      -}
    • changedInput schema / required
      Previous value: -[
      -  "url"
      -]New value: +[
      +  "body"
      +]
  2. First observed

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare the safety profile (readOnly=false, destructive=true, openWorld=true, idempotent=false). The description goes well beyond that: it discloses synchronous vs background execution, an observed latency (~4.5s at a 3-page limit), that cost and time scale with the bounds, and even billing rates. No contradiction with the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose, then steering, then bounds, then alternatives — a good information hierarchy. It is dense with several parallel clauses and a boilerplate pricing block, but essentially every sentence carries routing or cost information an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a high-parameter, open-world, cost-incurring crawl tool this covers the decision-relevant surface: what it does, how to steer and bound it, how long it takes, what it costs, and which sibling to use instead. An output schema exists, so return values need not be re-explained, and the description still sketches them usefully.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Reported schema coverage is 0% at the top level, so the description carries real burden and does so for the steering and bounding parameters: url, instructions, regex path/domain filters, max_depth, max_breadth and limit, with their roles explained. It does not touch format, timeout, extract_depth, chunks_per_source or the include_* toggles, so it does not fully compensate for the coverage gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Walk a site from a root url and return the content of the pages it finds') and immediately distinguishes itself from siblings by naming post_firecrawl_crawl, post_tavily_extract and post_tavily_map with the conditions that select each. An agent can route correctly without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit when-to-use ('broad coverage of one site — documentation, a catalogue, a competitor's blog') and explicit alternatives: extract for a handful of known pages, map to size a site first, firecrawl_crawl when the crawl is too large to wait on synchronously. When-not guidance is effectively present via the sibling comparisons.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources