Skip to main content
Glama

Crawl a site

pagesift_crawl

Starts a crawl job that follows links from a seed URL and scrapes discovered pages, returning a job ID for async polling. Use include/exclude paths and limits to keep multi-page collection focused.

Instructions

Start a crawl job that follows links from a URL and scrapes every page found. Use pagesift_scrape when you need one page and pagesift_map when you only need URLs. By default the crawl includes sitemap URLs, follows links within the seed's path scope, has no depth limit and uses per-host concurrency; maxDiscoveryDepth and maxConcurrency lower these. Pages blocked by robots.txt or failing are reported in the job's errors and the crawl continues. Returns a job id right away; poll it with pagesift_poll_job. Results can be large, so set a conservative limit and use includePaths/excludePaths to stay on topic.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "pagesift_crawl"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesThe URL to start crawling from.
delayNoSeconds to wait between requests to the same host.
limitNoMaximum number of pages to crawl. Defaults to 10000.
queryNoCrawl the pages most relevant to this query first.
scopeNoHow far the crawl may leave the seed's directory; auto widens one level when the seed looks like a leaf page.
sitemapNoinclude (default): crawl sitemap URLs and follow page links; skip: ignore sitemaps; only: crawl sitemap URLs without following page links.
webhookNoURL, or {url, headers, metadata, events}.
adaptiveNoWith query: stop once the collected pages cover it; an object sets coverageThreshold, evidencePerTerm, patience, minGain.
maxDepthNoFirecrawl v1 name; only checked against the limit, maxDiscoveryDepth sets the depth.
attachmentsNoWith query: also fetch PDF/Office documents linked from relevant pages.
excludePathsNoRegular expressions for paths to skip.
includePathsNoRegular expressions for paths to keep.
scrapeOptionsNoScrape options applied to every crawled page.
maxConcurrencyNoLower the job's share of fetch concurrency; it cannot exceed the server's per-host or global limit.
allowSubdomainsNoFollow links to other subdomains of the same site. Defaults to true only when query is set.
attachmentLimitNoMaximum attachments to fetch; defaults to limit.
crawlEntireDomainNoCrawl the whole host instead of the seed's path scope. Defaults to false.
maxDiscoveryDepthNoHow many links away from the seed to follow. Defaults to no limit.
allowExternalLinksNoFollow links to other hosts. Defaults to false.
ignoreQueryParametersNoTreat URLs that differ only by query string as one. Defaults to false.

Output Schema

NameRequiredDescriptionDefault
idYesJob id; pass it to pagesift_poll_job or pagesift_cancel_job.
nextNoSuggested next call.
statusYesThe job has started.
invalidURLsNoURLs that were rejected before the job started.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed36 schema fields changedv1.2.2
    • addedInput schema / properties / allowExternalLinks / description
      Added value: +"Follow links to other hosts. Defaults to false."
    • addedInput schema / properties / allowSubdomains / description
      Added value: +"Follow links to other subdomains of the same site. Defaults to true only when query is set."
    • addedInput schema / properties / attachmentLimit / description
      Added value: +"Maximum attachments to fetch; defaults to limit."
    • addedInput schema / properties / crawlEntireDomain / description
      Added value: +"Crawl the whole host instead of the seed's path scope. Defaults to false."
    • addedInput schema / properties / ignoreQueryParameters / description
      Added value: +"Treat URLs that differ only by query string as one. Defaults to false."
    • changedInput schema / properties / limit / description
      Previous value: -"Maximum number of pages to crawl."New value: +"Maximum number of pages to crawl. Defaults to 10000."
    • addedInput schema / properties / maxConcurrency / description
      Added value: +"Lower the job's share of fetch concurrency; it cannot exceed the server's per-host or global limit."
    • addedInput schema / properties / maxDiscoveryDepth / description
      Added value: +"How many links away from the seed to follow. Defaults to no limit."
    • changedInput schema / properties / scope / description
      Previous value: -"How far the crawl may leave the seed's directory."New value: +"How far the crawl may leave the seed's directory; auto widens one level when the seed looks like a leaf page."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / all / description
      Added value: +"Act on every element matching the selector instead of the first one."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / direction / description
      Added value: +"Scroll direction."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / format / description
      Added value: +"Output format for screenshot or PDF actions."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / fullPage / description
      Added value: +"Capture the full page instead of the viewport."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / key / description
      Added value: +"Key to press, for example Enter or ArrowDown."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / landscape / description
      Added value: +"Use landscape orientation for PDF output."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / milliseconds / description
      Added value: +"Wait time for wait actions, in milliseconds."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / quality / description
      Added value: +"JPEG quality from 1 to 100."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / scale / description
      Added value: +"Scale factor for PDF output."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / script / description
      Added value: +"JavaScript source for executeJavascript."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / selector / description
      Added value: +"CSS selector of the element to act on."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / text / description
      Added value: +"Text to type or to use as the JavaScript expression, depending on the action."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / type / description
      Added value: +"Action to run."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / viewport / description
      Added value: +"Viewport size for screenshot or PDF output."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / viewport / properties / height / description
      Added value: +"Viewport height in CSS pixels."
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / viewport / properties / width / description
      Added value: +"Viewport width in CSS pixels."
    • addedInput schema / properties / scrapeOptions / properties / chunking / properties / type / description
      Added value: +"Chunking strategy, for example recursive."
    • addedInput schema / properties / scrapeOptions / properties / excludeTags / description
      Added value: +"Remove content inside these CSS selectors."
    • changedInput schema / properties / scrapeOptions / properties / formats / items / anyOf
      Previous value: -[
      -  {
      -    "enum": [
      -      "markdown",
      -      "html",
      -      "rawHtml",
      -      "links",
      -      "images",
      -      "extract"
      -    ],
      -    "type": "string"
      -  },
      -  {
      -    "additionalProperties": false,
      -    "properties": {
      -      "intent": {
      -        "description": "extract without a schema: what to extract; the LLM writes the schema.",
      -        "type": "string"
      -      },
      -      "modes": {
      -        "description": "changeTracking: diff modes.",
      -        "items": {
      -          "type": "string"
      -        },
      -        "type": "array"
      -      },
      -      "prompt": {
      -        "description": "json: what to extract.",
      -        "type": "string"
      -      },
      -      "schema": {
      -        "description": "extract: CSS schema {baseSelector, fields}; json: JSON Schema for the LLM.",
      -        "type": "object"
      -      },
      -      "type": {
      -        "enum": [
      -          "markdown",
      -          "html",
      -          "rawHtml",
      -          "links",
      -          "images",
      -          "extract"
      -        ],
      -        "type": "string"
      -      }
      -    },
      -    "required": [
      -      "type"
      -    ],
      -    "type": "object"
      -  }
      -]New value: +[
      +  {
      +    "enum": [
      +      "markdown",
      +      "html",
      +      "rawHtml",
      +      "links",
      +      "images",
      +      "extract"
      +    ],
      +    "type": "string"
      +  },
      +  {
      +    "additionalProperties": false,
      +    "properties": {
      +      "intent": {
      +        "description": "extract without a schema: what to extract; the LLM writes the schema.",
      +        "type": "string"
      +      },
      +      "modes": {
      +        "description": "changeTracking: diff modes.",
      +        "items": {
      +          "type": "string"
      +        },
      +        "type": "array"
      +      },
      +      "prompt": {
      +        "description": "json: what to extract.",
      +        "type": "string"
      +      },
      +      "schema": {
      +        "description": "extract: CSS schema {baseSelector, fields}; json: JSON Schema for the LLM.",
      +        "type": "object"
      +      },
      +      "type": {
      +        "description": "Format name.",
      +        "enum": [
      +          "markdown",
      +          "html",
      +          "rawHtml",
      +          "links",
      +          "images",
      +          "extract"
      +        ],
      +        "type": "string"
      +      }
      +    },
      +    "required": [
      +      "type"
      +    ],
      +    "type": "object"
      +  }
      +]
    • addedInput schema / properties / scrapeOptions / properties / includeTags / description
      Added value: +"Only keep content inside these CSS selectors."
    • addedInput schema / properties / scrapeOptions / properties / location / description
      Added value: +"Geographic hint for the request, for example {country: \"US\", languages: [\"en\"]}."
    • addedInput schema / properties / scrapeOptions / properties / location / properties / country / description
      Added value: +"Two-letter country code."
    • addedInput schema / properties / scrapeOptions / properties / location / properties / languages / description
      Added value: +"Preferred language codes, most preferred first."
    • addedInput schema / properties / scrapeOptions / properties / removeBase64Images / description
      Added value: +"Remove inline base64 image data from the output. Defaults to true."
    • addedInput schema / properties / scrapeOptions / properties / storeInCache / description
      Added value: +"Store a successful result in the result cache. Defaults to true."
    • addedInput schema / properties / sitemap / description
      Added value: +"include (default): crawl sitemap URLs and follow page links; skip: ignore sitemaps; only: crawl sitemap URLs without following page links."
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "properties": {
      +    "id": {
      +      "description": "Job id; pass it to pagesift_poll_job or pagesift_cancel_job.",
      +      "type": "string"
      +    },
      +    "invalidURLs": {
      +      "description": "URLs that were rejected before the job started.",
      +      "items": {
      +        "description": "Invalid URL",
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    "next": {
      +      "description": "Suggested next call.",
      +      "type": "string"
      +    },
      +    "status": {
      +      "description": "The job has started.",
      +      "enum": [
      +        "scraping"
      +      ],
      +      "type": "string"
      +    }
      +  },
      +  "required": [
      +    "id",
      +    "status"
      +  ],
      +  "type": "object"
      +}
  2. Changed68 schema fields changedv0.81.6
    • addedInput schema / additionalProperties
      Added value: +false
    • changedInput schema / properties / adaptive / description
      Previous value: -"With query: stop once the collected pages cover it."New value: +"With query: stop once the collected pages cover it; an object sets coverageThreshold, evidencePerTerm, patience, minGain."
    • changedInput schema / properties / adaptive / type
      Previous value: -"boolean"New value: +[
      +  "boolean",
      +  "object"
      +]
    • addedInput schema / properties / attachmentLimit / minimum
      Added value: +0
    • changedInput schema / properties / attachmentLimit / type
      Previous value: -"number"New value: +"integer"
    • removedInput schema / properties / deduplicateSimilarURLs
      Removed value: -{
      -  "type": "boolean"
      -}
    • addedInput schema / properties / limit / minimum
      Added value: +1
    • changedInput schema / properties / limit / type
      Previous value: -"number"New value: +"integer"
    • addedInput schema / properties / maxConcurrency / minimum
      Added value: +1
    • changedInput schema / properties / maxConcurrency / type
      Previous value: -"number"New value: +"integer"
    • addedInput schema / properties / maxDepth
      Added value: +{
      +  "description": "Firecrawl v1 name; only checked against the limit, maxDiscoveryDepth sets the depth.",
      +  "minimum": 0,
      +  "type": "integer"
      +}
    • addedInput schema / properties / maxDiscoveryDepth / minimum
      Added value: +0
    • changedInput schema / properties / maxDiscoveryDepth / type
      Previous value: -"number"New value: +"integer"
    • removedInput schema / properties / prompt
      Removed value: -{
      -  "type": "string"
      -}
    • addedInput schema / properties / scope
      Added value: +{
      +  "description": "How far the crawl may leave the seed's directory.",
      +  "enum": [
      +    "auto",
      +    "exact",
      +    "parent",
      +    "domain"
      +  ],
      +  "type": "string"
      +}
    • addedInput schema / properties / scrapeOptions / additionalProperties
      Added value: +false
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / all
      Added value: +{
      +  "type": "boolean"
      +}
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / format
      Added value: +{
      +  "type": "string"
      +}
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / landscape
      Added value: +{
      +  "type": "boolean"
      +}
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / milliseconds / minimum
      Added value: +0
    • changedInput schema / properties / scrapeOptions / properties / actions / items / properties / milliseconds / type
      Previous value: -"number"New value: +"integer"
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / quality
      Added value: +{
      +  "maximum": 100,
      +  "minimum": 1,
      +  "type": "integer"
      +}
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / scale
      Added value: +{
      +  "type": "number"
      +}
    • changedInput schema / properties / scrapeOptions / properties / actions / items / properties / type / enum
      Previous value: -[
      -  "wait",
      -  "screenshot",
      -  "scroll",
      -  "scrape",
      -  "click",
      -  "write",
      -  "press",
      -  "executeJavascript",
      -  "generatePDF"
      -]New value: +[
      +  "wait",
      +  "click",
      +  "write",
      +  "press",
      +  "scroll",
      +  "screenshot",
      +  "scrape",
      +  "executeJavascript",
      +  "pdf"
      +]
    • addedInput schema / properties / scrapeOptions / properties / actions / items / properties / viewport
      Added value: +{
      +  "properties": {
      +    "height": {
      +      "type": "integer"
      +    },
      +    "width": {
      +      "type": "integer"
      +    }
      +  },
      +  "required": [
      +    "width",
      +    "height"
      +  ],
      +  "type": "object"
      +}
    • addedInput schema / properties / scrapeOptions / properties / blockAds
      Added value: +{
      +  "description": "Block ads when rendering. Defaults to true.",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / scrapeOptions / properties / chunking
      Added value: +{
      +  "description": "Split markdown into chunks, e.g. {\"type\": \"recursive\", \"size\": 400}.",
      +  "properties": {
      +    "type": {
      +      "type": "string"
      +    }
      +  },
      +  "required": [
      +    "type"
      +  ],
      +  "type": "object"
      +}
    • addedInput schema / properties / scrapeOptions / properties / discoverScripts
      Added value: +{
      +  "description": "With engine http and the links format: also list same-origin URLs found in external scripts.",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / scrapeOptions / properties / egress
      Added value: +{
      +  "description": "Name of a configured [[egress]] to fetch through.",
      +  "type": "string"
      +}
    • changedInput schema / properties / scrapeOptions / properties / engine / description
      Previous value: -"pagesift extension: fetch with plain HTTP, a headless browser, or decide automatically (default)."New value: +"Fetch with plain HTTP, a headless browser, or decide automatically (default)."
    • addedInput schema / properties / scrapeOptions / properties / explain
      Added value: +{
      +  "description": "Record how the result was produced in metadata.pagesift.plan.",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / scrapeOptions / properties / exportImages
      Added value: +{
      +  "description": "Export the page's images (PDF embedded images, Office media, web images); returned as resource links.",
      +  "type": "boolean"
      +}
    • changedInput schema / properties / scrapeOptions / properties / formats / description
      Previous value: -"Output formats. Defaults to [\"markdown\"]."New value: +"Output formats. Defaults to [\"markdown\"]. Object form passes options: {\"type\": \"extract\", \"schema\": {\"baseSelector\": ..., \"fields\": [...]}} extracts structured data with CSS selectors (no LLM; selectors are relocated after a redesign when a store is open), {\"type\": \"json\", \"schema\": {...}, \"prompt\": \"...\"} asks the LLM."
    • addedInput schema / properties / scrapeOptions / properties / formats / items / anyOf
      Added value: +[
      +  {
      +    "enum": [
      +      "markdown",
      +      "html",
      +      "rawHtml",
      +      "links",
      +      "images",
      +      "extract"
      +    ],
      +    "type": "string"
      +  },
      +  {
      +    "additionalProperties": false,
      +    "properties": {
      +      "intent": {
      +        "description": "extract without a schema: what to extract; the LLM writes the schema.",
      +        "type": "string"
      +      },
      +      "modes": {
      +        "description": "changeTracking: diff modes.",
      +        "items": {
      +          "type": "string"
      +        },
      +        "type": "array"
      +      },
      +      "prompt": {
      +        "description": "json: what to extract.",
      +        "type": "string"
      +      },
      +      "schema": {
      +        "description": "extract: CSS schema {baseSelector, fields}; json: JSON Schema for the LLM.",
      +        "type": "object"
      +      },
      +      "type": {
      +        "enum": [
      +          "markdown",
      +          "html",
      +          "rawHtml",
      +          "links",
      +          "images",
      +          "extract"
      +        ],
      +        "type": "string"
      +      }
      +    },
      +    "required": [
      +      "type"
      +    ],
      +    "type": "object"
      +  }
      +]
    • removedInput schema / properties / scrapeOptions / properties / formats / items / enum
      Removed value: -[
      -  "markdown",
      -  "html",
      -  "rawHtml",
      -  "screenshot",
      -  "links",
      -  "images",
      -  "summary",
      -  "changeTracking",
      -  "branding",
      -  "json",
      -  "query",
      -  "audio"
      -]
    • removedInput schema / properties / scrapeOptions / properties / formats / items / type
      Removed value: -"string"
    • addedInput schema / properties / scrapeOptions / properties / headers
      Added value: +{
      +  "additionalProperties": {
      +    "type": "string"
      +  },
      +  "description": "Extra request headers.",
      +  "type": "object"
      +}
    • changedInput schema / properties / scrapeOptions / properties / ifModifiedSince / description
      Previous value: -"pagesift extension: HTTP date for a conditional request."New value: +"HTTP date for a conditional request."
    • changedInput schema / properties / scrapeOptions / properties / ifNoneMatch / description
      Previous value: -"pagesift extension: ETag for a conditional request."New value: +"ETag for a conditional request."
    • removedInput schema / properties / scrapeOptions / properties / jsonOptions
      Removed value: -{
      -  "description": "Options for the json format.",
      -  "properties": {
      -    "prompt": {
      -      "type": "string"
      -    },
      -    "schema": {
      -      "type": "object"
      -    }
      -  },
      -  "type": "object"
      -}
    • addedInput schema / properties / scrapeOptions / properties / keepSession
      Added value: +{
      +  "description": "Keep the browser page open and return its id in metadata.scrapeId, to continue with pagesift_interact. Only on pagesift_scrape. Close it with pagesift_interact_stop.",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / scrapeOptions / properties / location / additionalProperties
      Added value: +false
    • removedInput schema / properties / scrapeOptions / properties / lockdown
      Removed value: -{
      -  "type": "boolean"
      -}
    • addedInput schema / properties / scrapeOptions / properties / maxAge / minimum
      Added value: +0
    • changedInput schema / properties / scrapeOptions / properties / maxAge / type
      Previous value: -"number"New value: +"integer"
    • addedInput schema / properties / scrapeOptions / properties / minAge
      Added value: +{
      +  "description": "Only return a cached copy at least this many milliseconds old.",
      +  "minimum": 0,
      +  "type": "integer"
      +}
    • addedInput schema / properties / scrapeOptions / properties / mobile / description
      Added value: +"Emulate a phone when rendering."
    • addedInput schema / properties / scrapeOptions / properties / parsers / description
      Added value: +"PDF parser selection, e.g. [\"pdf\"] or [{\"type\": \"pdf\", \"maxPages\": 10}]."
    • removedInput schema / properties / scrapeOptions / properties / parsers / items
      Removed value: -{
      -  "enum": [
      -    "pdf"
      -  ],
      -  "type": "string"
      -}
    • removedInput schema / properties / scrapeOptions / properties / pdfOptions
      Removed value: -{
      -  "properties": {
      -    "maxPages": {
      -      "maximum": 10000,
      -      "minimum": 1,
      -      "type": "integer"
      -    }
      -  },
      -  "type": "object"
      -}
    • removedInput schema / properties / scrapeOptions / properties / profile
      Removed value: -{
      -  "properties": {
      -    "name": {
      -      "type": "string"
      -    },
      -    "saveChanges": {
      -      "type": "boolean"
      -    }
      -  },
      -  "required": [
      -    "name"
      -  ],
      -  "type": "object"
      -}
    • changedInput schema / properties / scrapeOptions / properties / proxy / description
      Previous value: -"Only basic and auto are supported; pagesift does not bypass bot protection."New value: +"\"basic\", \"auto\", or an http/https/socks5 proxy URL. pagesift does not solve CAPTCHAs; a block or challenge page is reported as an error."
    • removedInput schema / properties / scrapeOptions / properties / proxy / enum
      Removed value: -[
      -  "basic",
      -  "stealth",
      -  "enhanced",
      -  "auto"
      -]
    • addedInput schema / properties / scrapeOptions / properties / queryFilter
      Added value: +{
      +  "description": "Keep only the blocks relevant to a query: a string, or {query, threshold, relative, topK, maxTokens, minTokens, headings}.",
      +  "type": [
      +    "string",
      +    "object"
      +  ]
      +}
    • removedInput schema / properties / scrapeOptions / properties / queryOptions
      Removed value: -{
      -  "description": "Options for the query format.",
      -  "properties": {
      -    "mode": {
      -      "default": "freeform",
      -      "enum": [
      -        "directQuote",
      -        "freeform"
      -      ],
      -      "type": "string"
      -    },
      -    "prompt": {
      -      "maxLength": 10000,
      -      "type": "string"
      -    }
      -  },
      -  "required": [
      -    "prompt"
      -  ],
      -  "type": "object"
      -}
    • removedInput schema / properties / scrapeOptions / properties / redactPII
      Removed value: -{
      -  "type": "boolean"
      -}
    • addedInput schema / properties / scrapeOptions / properties / respondTiming
      Added value: +{
      +  "description": "When a rendered page counts as ready: html does not run scripts; the others wait for visible content, DOM mutations to settle, scripts/XHR to go quiet, or media to load.",
      +  "enum": [
      +    "html",
      +    "visible-content",
      +    "mutation-idle",
      +    "resource-idle",
      +    "media-idle",
      +    "network-idle"
      +  ],
      +  "type": "string"
      +}
    • removedInput schema / properties / scrapeOptions / properties / screenshotOptions
      Removed value: -{
      -  "properties": {
      -    "fullPage": {
      -      "type": "boolean"
      -    },
      -    "quality": {
      -      "type": "number"
      -    },
      -    "viewport": {
      -      "properties": {
      -        "height": {
      -          "type": "number"
      -        },
      -        "width": {
      -          "type": "number"
      -        }
      -      },
      -      "required": [
      -        "width",
      -        "height"
      -      ],
      -      "type": "object"
      -    }
      -  },
      -  "type": "object"
      -}
    • removedInput schema / properties / scrapeOptions / properties / skipTlsVerification
      Removed value: -{
      -  "type": "boolean"
      -}
    • addedInput schema / properties / scrapeOptions / properties / storeSnapshot
      Added value: +{
      +  "description": "Keep a snapshot of this page for later change tracking.",
      +  "type": "boolean"
      +}
    • changedInput schema / properties / scrapeOptions / properties / timeout / minimum
      Previous value: -1New value: +1000
    • changedInput schema / properties / scrapeOptions / properties / waitFor / description
      Previous value: -"Milliseconds to wait before capturing the page."New value: +"Milliseconds to wait before capturing a rendered page."
    • addedInput schema / properties / scrapeOptions / properties / waitFor / minimum
      Added value: +0
    • changedInput schema / properties / scrapeOptions / properties / waitFor / type
      Previous value: -"number"New value: +"integer"
    • removedInput schema / properties / scrapeOptions / properties / zeroDataRetention
      Removed value: -{
      -  "type": "boolean"
      -}
    • addedInput schema / properties / webhook / description
      Added value: +"URL, or {url, headers, metadata, events}."
    • changedInput schema / properties / webhook / type
      Previous value: -"string"New value: +[
      +  "string",
      +  "object"
      +]
    • removedInput schema / properties / webhookHeaders
      Removed value: -{
      -  "additionalProperties": {
      -    "type": "string"
      -  },
      -  "type": "object"
      -}
  3. First observedv0.81.2

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations (readOnlyHint=false, openWorldHint=true, destructiveHint=false) only flag this as an open-world non-destructive write, so the description must carry the operational picture — and it does: it discloses the async nature (returns a job id right away, must poll), the default crawl envelope (sitemap inclusion, path scoping, unlimited depth, per-host concurrency), and failure handling (robots.txt blocks and failing pages land in the job's errors while the crawl continues). It also warns results can be large. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Five tight sentences, front-loaded with the what and immediately followed by sibling routing. Each sentence contributes a distinct fact (routing, defaults, failure behavior, async contract, sizing advice) with no repetition of the name or title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 20-parameter tool with nested scrapeOptions and an output schema, the description covers the pieces the schema cannot: that the call is asynchronous, how to retrieve results, what the defaults do, and how failures are surfaced. Return-value detail is correctly left to the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds meaning on top by explaining that maxDiscoveryDepth and maxConcurrency *lower* defaults rather than set absolute behavior, and by advising a conservative limit plus includePaths/excludePaths for topicality — guidance about parameter interaction that the flat schema does not convey.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource — 'Start a crawl job that follows links from a URL and scrapes every page found' — and immediately distinguishes itself from pagesift_scrape (one page) and pagesift_map (URLs only). An agent can choose among the three siblings without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It names the two alternatives and the exact conditions that select them ('when you need one page', 'when you only need URLs'), and it names pagesift_poll_job as the mandatory follow-up for retrieving results. It even closes with routing advice to constrain the crawl via limit/includePaths/excludePaths. This is explicit when-to-use guidance with alternatives, not mere implication.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.