Skip to main content
Glama

Crawl site

crw_crawl

Start an asynchronous website crawl and receive a job ID to poll for status. Configure depth, page limits, JS rendering, and structured data extraction.

Instructions

Start an async site crawl; returns a job id to poll with crw_check_crawl_status.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesStarting URL
waitForNoMs to wait after JS render per page
maxDepthNoMax crawl depth (default 2)
maxPagesNoMax pages to crawl (default 10)
renderJsNoForce JS render (true), HTTP-only (false), omit = auto
rendererNoPin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs.
jsonSchemaNoOptional. A JSON Schema (draft 2020-12) describing fields to extract from each page via an LLM, e.g. {"type":"object","properties":{"title":{"type":"string"}}}. Free-form object. Omit to crawl without structured extraction.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changedv0.35.1
    • changedInput schema / properties / renderer / description
      Previous value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."New value: +"Pin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs."
    • changedInput schema / properties / renderer / enum
      Previous value: -[
      -  "auto",
      -  "lightpanda",
      -  "chrome",
      -  "playwright",
      -  "camoufox"
      -]New value: +[
      +  "auto",
      +  "lightpanda",
      +  "chrome",
      +  "playwright",
      +  "camoufox",
      +  "impersonated-http"
      +]
  2. Changed4 schema fields changedv0.18.0
    • addedInput schema / properties / jsonSchema / additionalProperties
      Added value: +true
    • changedInput schema / properties / jsonSchema / description
      Previous value: -"JSON schema for LLM extraction per page"New value: +"Optional. A JSON Schema (draft 2020-12) describing fields to extract from each page via an LLM, e.g. {\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}. Free-form object. Omit to crawl without structured extraction."
    • changedInput schema / properties / renderer / description
      Previous value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."
    • changedInput schema / properties / renderer / enum
      Previous value: -[
      -  "auto",
      -  "lightpanda",
      -  "chrome",
      -  "playwright"
      -]New value: +[
      +  "auto",
      +  "lightpanda",
      +  "chrome",
      +  "playwright",
      +  "camoufox"
      +]
  3. Changed7 schema fields changedv0.16.0
    • changedInput schema / properties / jsonSchema / description
      Previous value: -"JSON schema for LLM-based structured data extraction on each crawled page"New value: +"JSON schema for LLM extraction per page"
    • changedInput schema / properties / maxDepth / description
      Previous value: -"Maximum crawl depth (default: 2)"New value: +"Max crawl depth (default 2)"
    • changedInput schema / properties / maxPages / description
      Previous value: -"Maximum number of pages to crawl (default: 10)"New value: +"Max pages to crawl (default 10)"
    • changedInput schema / properties / renderJs / description
      Previous value: -"Render JavaScript on every crawled page (true = force JS, false = HTTP only, omit = auto-detect or use the server's render_js_default)"New value: +"Force JS render (true), HTTP-only (false), omit = auto"
    • changedInput schema / properties / renderer / description
      Previous value: -"Pin every crawled page to a specific renderer. \"auto\" (default if omitted) uses the configured fallback chain. Other values hard-pin with no fallback. Pinning a non-auto value implies renderJs:true unless renderJs:false is set explicitly."New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"
    • changedInput schema / properties / url / description
      Previous value: -"The starting URL to crawl"New value: +"Starting URL"
    • changedInput schema / properties / waitFor / description
      Previous value: -"Milliseconds to wait after JS rendering on each page"New value: +"Ms to wait after JS render per page"
  4. Addedv0.9.1
  5. Removedv0.8.3
  6. Changed3 schema fields changedv0.6.2
    • addedInput schema / properties / renderJs
      Added value: +{
      +  "description": "Render JavaScript on every crawled page (true = force JS, false = HTTP only, omit = auto-detect or use the server's render_js_default)",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / renderer
      Added value: +{
      +  "description": "Pin every crawled page to a specific renderer. \"auto\" (default if omitted) uses the configured fallback chain. Other values hard-pin with no fallback. Pinning a non-auto value implies renderJs:true unless renderJs:false is set explicitly.",
      +  "enum": [
      +    "auto",
      +    "lightpanda",
      +    "chrome",
      +    "playwright"
      +  ],
      +  "type": "string"
      +}
    • addedInput schema / properties / waitFor
      Added value: +{
      +  "description": "Milliseconds to wait after JS rendering on each page",
      +  "type": "integer"
      +}
  7. First observedv1.0.0

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate openWorldHint=true (external side effects) and idempotentHint=false. The description adds the important async behavior and the job-id return contract, which are not in the annotations. It does not contradict any annotation and provides a clear behavioral expectation, though it omits details like potential load on target site (covered by openWorldHint).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that states the core action and the immediate return value. There is zero fluff, and the most important behavioral fact (async + polling) is delivered first.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 7 parameters and no output schema, the description covers the essential contract: it starts an async crawl and returns a job id for polling. It does not explain the meaning of parameters (schema does) nor the external side effects (annotations do). The only minor gap is a lack of explicit warning about resource usage or cost, but openWorldHint and the async nature cover that adequately.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%: every parameter (url, waitFor, maxDepth, maxPages, renderJs, renderer, jsonSchema) has a descriptive definition. The tool description itself mentions none of these parameters, so it adds no additional semantics beyond the schema. Baseline 3 is appropriate because the schema already carries the full burden.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb ('start'), a clear resource ('site crawl'), and immediately explains the async nature and the follow-up mechanic (poll with crw_check_crawl_status). This differentiates it from siblings like crw_scrape or crw_check_crawl_status without ambiguity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It clearly establishes the typical workflow: start crawl, then poll status via the named sibling. It does not explicitly list alternatives or when not to use (e.g., for single-page extraction, use crw_scrape), but the async + polling context effectively guides usage and distinguishes it from crw_check_crawl_status.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.