Skip to main content
Glama

Start a crawl

crawl

Start an asynchronous site harvest from a seed URL to collect pages at scale; returns an id to poll, with depth, page-count, and pattern filters to scope the run.

Instructions

Start an asynchronous crawl from a seed URL. Returns a crawl_id to poll. Each page is billed; use the estimate tool first to price it.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "seed_url": "<seed_url>"
    },
    "name": "crawl"
  }
}

Input Schema

NameRequiredDescriptionDefault
dry_runNoNot supported here: leave it out and use the estimate tool to price a call.
optionsNoOther API fields for this operation, merged into the request body as they are (see the Clawifi API docs).
seed_urlYes
max_depthNo
max_pagesNo
exclude_patternsNo
include_patternsNo
same_domain_onlyNo

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed9 schema fields changedv0.4.1
    • addedInput schema / properties / dry_run
      Added value: +{
      +  "default": null,
      +  "description": "Not supported here: leave it out and use the estimate tool to price a call.",
      +  "title": "Dry Run"
      +}
    • addedInput schema / properties / exclude_patterns
      Added value: +{
      +  "anyOf": [
      +    {
      +      "items": {
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Exclude Patterns"
      +}
    • addedInput schema / properties / include_patterns
      Added value: +{
      +  "anyOf": [
      +    {
      +      "items": {
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Include Patterns"
      +}
    • removedInput schema / properties / kwargs
      Removed value: -{
      -  "title": "Kwargs"
      -}
    • addedInput schema / properties / max_depth
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "integer"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Max Depth"
      +}
    • addedInput schema / properties / max_pages
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "integer"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Max Pages"
      +}
    • addedInput schema / properties / options
      Added value: +{
      +  "anyOf": [
      +    {
      +      "additionalProperties": true,
      +      "type": "object"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "description": "Other API fields for this operation, merged into the request body as they are (see the Clawifi API docs).",
      +  "title": "Options"
      +}
    • addedInput schema / properties / same_domain_only
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "boolean"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Same Domain Only"
      +}
    • changedInput schema / required
      Previous value: -[
      -  "seed_url",
      -  "kwargs"
      -]New value: +[
      +  "seed_url"
      +]
  2. First observedv1.0.0

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations, it discloses that the crawl is asynchronous, returns a crawl_id to poll, bills per page, and requires pricing via estimate first. This adds useful operational context, though it does not cover auth or rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences are front-loaded with the action, then return value, then billing and estimate requirement. No sentence is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the core async flow, return identifier, and billing prerequisite, and an output schema exists for return details. However, for an 8-parameter crawling tool with low schema description coverage, the missing parameter guidance leaves it only minimally complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is only 25%, and the description explains only the seed URL conceptually. It does not add meaning for max_depth, max_pages, include/exclude patterns, same_domain_only, options, or dry_run, leaving most parameters undocumented in prose.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: start an asynchronous crawl from a seed URL. It immediately distinguishes the tool as crawl-starting rather than crawl-retrieval or pricing, and clarifies the async nature.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear guidance to use the estimate tool first because each page is billed. It does not explain when to choose crawl over other crawling-adjacent siblings, but the pricing prerequisite is explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.