Skip to main content
Glama

Start a crawl

crawl

Start crawling a section of one site: from url, follow links in the start path's subtree on its host and www twin, up to maxPages pages and maxDepth links deep. Returns { taskId } at once (with maxPages when the default of 100 applied); wait_crawl waits for it, then get_crawl_pages reads the pages. Each page takes scrape's page options (formats, onlyMainContent, includeTags, waitFor, timeout, verify and the rest; their descriptions are on scrape). Similar URLs fold into one page, and every collapsed or refused link is counted in get_crawl's discovery. On hosted Octocrawl a job runs in slices: a crawl reads at most 500 pages and a batch 1,000 URLs, each page costs credits as a scrape does (maxCredits caps the job), and results are kept 7 days after it ends.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "crawl"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYes
modeNoThe declared identity the page is read under: standard (default) or research. A hosted server holds no saved logins, so it has no authed mode.
mobileNo
verifyNoSame as scrape.
formatsNoWhat to return (default markdown): markdown, links, html, rawHtml, images, tables, json, attributes ({ selectors }), list ({ itemSelector?, fields? }: a record per repeated item, detected without itemSelector), screenshot or screenshot@fullPage (browser only). On an account's credits a browser page costs 5; with fewer left, a screenshot is refused.
headersNo
sitemapNoinclude (default): also queue the site's sitemap URLs. skip: no sitemap. only: the start URL and the sitemap entries.
timeoutNo
waitForNo
blockAdsNo
fastModeNo
maxDepthNoLinks followed from the start URL at most. Omitted or null: no limit.
maxPagesNoPages fetched at most. Omitted: 100. null: the server's default (none on a local server; hosted servers cap it).
maxCreditsNoThe most credits this job may spend; it stops when the next page would pass it.
excludeTagsNo
includeTagsNo
integrationNoYour workflow label; recorded, not sent to sites.
excludePathsNoPathname regexes that skip a discovered link; they win over includePaths.
includeLinksNo
includePathsNoPathname regexes a discovered link must match; the start URL is always fetched.
maxFileBytesNo
idempotencyKeyNoMakes a retry safe: the same key and arguments return the first taskId. Kept 24 hours.
maxConcurrencyNoPages fetched at once, at most (only lowers the worker count).
regexOnFullURLNoMatch includePaths and excludePaths against each link's canonical URL instead of its pathname. Default false.
allowSubdomainsNoFollow links to every host under the start URL's apex. Default false.
onlyMainContentNo
crawlEntireDomainNoFollow links anywhere on the start host, not only in the start path's subtree. Default false.
allowExternalLinksNoFollow links to any host, each with its own robots.txt; maxDepth and maxPages bound the walk. Default false. Not with allowlistedDomains.
allowlistedDomainsNo
removeBase64ImagesNo
ignoreQueryParametersNoTreat URLs that differ only in their query as one page. Default false.
deduplicateSimilarURLsNoTreat /a and /a/, www and apex, http and https as one page. Default true.

Output Schema

NameRequiredDescriptionDefault
taskIdYes
replayedNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • changedInput schema / properties / maxConcurrency / maximum
      Previous value: -2New value: +8
  2. Changed1 schema field changed
    • changedInput schema / properties / integration / description
      Previous value: -"Your workflow label (printable, no spaces); recorded, not sent to sites."New value: +"Your workflow label; recorded, not sent to sites."
  3. Changed1 schema field changed
    • addedInput schema / properties / maxConcurrency / maximum
      Added value: +2
  4. Changed1 schema field changed
    • addedInput schema / properties / maxCredits
      Added value: +{
      +  "description": "The most credits this job may spend; it stops when the next page would pass it.",
      +  "minimum": 1,
      +  "type": "integer"
      +}
  5. Changed4 schema fields changed
    • changedInput schema / properties / formats / description
      Previous value: -"Same as scrape."New value: +"What to return (default markdown): markdown, links, html, rawHtml, images, tables, json, attributes ({ selectors }), list ({ itemSelector?, fields? }: a record per repeated item, detected without itemSelector), screenshot or screenshot@fullPage (browser only). On an account's credits a browser page costs 5; with fewer left, a screenshot is refused."
    • addedInput schema / properties / formats / items
      Added value: +{
      +  "anyOf": [
      +    {
      +      "enum": [
      +        "markdown",
      +        "links",
      +        "json",
      +        "html",
      +        "rawHtml",
      +        "images",
      +        "tables",
      +        "screenshot",
      +        "screenshot@fullPage"
      +      ],
      +      "type": "string"
      +    },
      +    {
      +      "additionalProperties": false,
      +      "properties": {
      +        "modelFallback": {
      +          "type": "boolean"
      +        },
      +        "prompt": {
      +          "maxLength": 4000,
      +          "type": "string"
      +        },
      +        "schema": {
      +          "type": "object"
      +        },
      +        "type": {
      +          "const": "json"
      +        }
      +      },
      +      "required": [
      +        "type",
      +        "schema"
      +      ],
      +      "type": "object"
      +    },
      +    {
      +      "additionalProperties": false,
      +      "properties": {
      +        "selectors": {
      +          "items": {
      +            "additionalProperties": false,
      +            "properties": {
      +              "attribute": {
      +                "maxLength": 100,
      +                "minLength": 1,
      +                "pattern": "^[A-Za-z_][A-Za-z0-9_:.-]*$",
      +                "type": "string"
      +              },
      +              "selector": {
      +                "maxLength": 200,
      +                "minLength": 1,
      +                "type": "string"
      +              }
      +            },
      +            "required": [
      +              "selector",
      +              "attribute"
      +            ],
      +            "type": "object"
      +          },
      +          "maxItems": 50,
      +          "minItems": 1,
      +          "type": "array"
      +        },
      +        "type": {
      +          "const": "attributes"
      +        }
      +      },
      +      "required": [
      +        "type",
      +        "selectors"
      +      ],
      +      "type": "object"
      +    },
      +    {
      +      "additionalProperties": false,
      +      "properties": {
      +        "fields": {
      +          "items": {
      +            "additionalProperties": false,
      +            "properties": {
      +              "attribute": {
      +                "pattern": "^[A-Za-z_][A-Za-z0-9_:.-]*$",
      +                "type": "string"
      +              },
      +              "name": {
      +                "maxLength": 64,
      +                "minLength": 1,
      +                "type": "string"
      +              },
      +              "selector": {
      +                "maxLength": 200,
      +                "minLength": 1,
      +                "type": "string"
      +              }
      +            },
      +            "required": [
      +              "name"
      +            ],
      +            "type": "object"
      +          },
      +          "maxItems": 50,
      +          "minItems": 1,
      +          "type": "array"
      +        },
      +        "itemSelector": {
      +          "maxLength": 200,
      +          "minLength": 1,
      +          "type": "string"
      +        },
      +        "type": {
      +          "const": "list"
      +        }
      +      },
      +      "required": [
      +        "type"
      +      ],
      +      "type": "object"
      +    },
      +    {
      +      "additionalProperties": false,
      +      "properties": {
      +        "fullPage": {
      +          "type": "boolean"
      +        },
      +        "quality": {
      +          "maximum": 100,
      +          "minimum": 1,
      +          "type": "integer"
      +        },
      +        "type": {
      +          "const": "screenshot"
      +        },
      +        "viewport": {
      +          "additionalProperties": false,
      +          "properties": {
      +            "height": {
      +              "maximum": 1080,
      +              "minimum": 240,
      +              "type": "integer"
      +            },
      +            "width": {
      +              "maximum": 1920,
      +              "minimum": 320,
      +              "type": "integer"
      +            }
      +          },
      +          "required": [
      +            "width",
      +            "height"
      +          ],
      +          "type": "object"
      +        }
      +      },
      +      "required": [
      +        "type"
      +      ],
      +      "type": "object"
      +    }
      +  ]
      +}
    • changedInput schema / properties / maxFileBytes / maximum
      Previous value: -524288000New value: +5242880
    • changedInput schema / properties / mode / description
      Previous value: -"standard (default) or research; for signed-in pages send a batch in mode authed."New value: +"The declared identity the page is read under: standard (default) or research. A hosted server holds no saved logins, so it has no authed mode."
  6. Added

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations: discloses hosted Octocrawl slicing limits (500 pages per crawl, 1,000 URLs per batch), per-page credit cost, that maxCredits caps the job, and that results are kept 7 days after completion. It also states the return shape and the default maxPages=100 behavior, which the annotations do not cover.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense but front-loaded: the traversal scope comes first, then the async return contract, then parameter notes, then hosted limits. Every sentence carries information, though the final hosted-limits sentence is lengthy and the middle section is a compressed list.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 32-parameter, nested-schema, output-schema-backed async tool, the description covers the async lifecycle, traversal bounds, deduplication behavior, hosted caps and credit costs. Remaining gaps are per-page scrape options, which the description explicitly redirects to the scrape tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 56% schema coverage across 32 params, the description adds real meaning: default 100 pages, null maxPages meaning, subtree/host/www-twin traversal model, similar-URL folding plus counting of collapsed/refused links, and it points to scrape for the shared page options. Some params (mobile, headers, blockAds, fastMode, integration) still get no explanation anywhere in the description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and scoped resource: 'Start crawling a section of one site: from url, follow links in the start path's subtree on its host and www twin.' The scope constraint (start path subtree, host + www twin, bounded by maxPages/maxDepth) clearly separates it from scrape, map and batch_scrape.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit workflow: it returns { taskId } at once, 'wait_crawl waits for it, then get_crawl_pages reads the pages,' so the agent knows which siblings to chain. It also notes that page-level options are delegated to scrape. It stops short of stating when-not to use this versus map or batch_scrape.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.