Skip to main content
Glama

Crawl a page

crawl

Fetch a web page through a proxied browser and return its HTML -- or markdown, with output -- in content, optionally with an AI answer to ai_query in data. Spends credits; credits_charged says how many. If status is "queued" or "running", call crawl_poll with the job_id to collect it.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "crawl"
  }
}

Input Schema

NameRequiredDescriptionDefault
jsNoRender the page in a browser so javascript runs. Turn off for static pages to pay less.
urlYesThe page to fetch. http or https, with a host.
cityNoThe city inside that country to appear to browse from, as the pool spells it: letters, digits, hyphens or underscores, e.g. berlin or new_york. Requires country. Doubles the price of the crawl.
touchNoReport the screen as touch capable. Requires viewport.
imagesNoDownload the images on the page.
mobileNoTell the page it is on a phone. Requires viewport; pair it with user_agent for sites that sniff it.
outputNoHow the page comes back in `content`: "html" verbatim, or "markdown" -- smaller and easier to read. Free either way.html
adblockNoDrop ads and trackers before the page loads. Free either way.
countryNoThe country to appear to browse from, as a two-letter ISO 3166-1 code such as es, us or de. Free. Required to use city.
premiumNoFetch through the premium proxy pool instead of the standard one. Costs more; use it for targets that refuse ordinary requests.
ai_queryNoA question to answer about the page once it has been fetched, e.g. "What is the price?". Costs extra credits.
viewportNoThe window size as WIDTHxHEIGHT, e.g. 390x844. Required for device_scale_factor, mobile, landscape and touch.
landscapeNoReport the screen as landscape. Requires viewport.
user_agentNoThe user agent to send instead of the crawler's own.
wait_untilNoWhen the page counts as loaded. Use domcontentloaded or load for pages that keep a connection open.networkidle2
main_contentNoWith `output` "markdown" only: convert just the page's main block, without navigation, header, footer, sidebars or cookie banners. Falls back to the whole page when no main block with real text is found.
cookies_bannerNoHow to answer cookie banners: accept opts in, reject opts out where the banner allows it, none leaves them on the page.accept
device_scale_factorNoThe device pixel ratio, e.g. 3 for a recent phone. Requires viewport.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / properties / cookies_banner
      Added value: +{
      +  "default": "accept",
      +  "description": "How to answer cookie banners: accept opts in, reject opts out where the banner allows it, none leaves them on the page.",
      +  "enum": [
      +    "accept",
      +    "reject",
      +    "none"
      +  ],
      +  "type": "string"
      +}
  2. Changed2 schema fields changed
    • addedInput schema / properties / main_content
      Added value: +{
      +  "default": false,
      +  "description": "With `output` \"markdown\" only: convert just the page's main block, without navigation, header, footer, sidebars or cookie banners. Falls back to the whole page when no main block with real text is found.",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / output
      Added value: +{
      +  "default": "html",
      +  "description": "How the page comes back in `content`: \"html\" verbatim, or \"markdown\" -- smaller and easier to read. Free either way.",
      +  "enum": [
      +    "html",
      +    "markdown"
      +  ],
      +  "type": "string"
      +}
  3. Changed1 schema field changed
    • changedInput schema / properties / wait_until / default
      Previous value: -"networkidle0"New value: +"networkidle2"
  4. First observed

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare `openWorldHint: true` and `destructiveHint: false`; the description adds substantive context beyond that by disclosing that the call spends credits, that `credits_charged` reports the amount, and that the result may arrive asynchronously with a `job_id`. Only failure/error behavior is left unstated, keeping it from a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three dense sentences with the core purpose front-loaded and no filler. The return-field naming is earned given the absence of an output schema, though the sentence runs long with multiple clauses.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 18-parameter, no-output-schema tool the description covers the essentials: what comes back (`content`, `data`, `credits_charged`, `status`) and what to do when the job is not finished. It does not address error or failure responses, which is a modest gap rather than a blocking one.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so parameter mechanics are already fully documented and the baseline is 3. The description adds a little by tying `output` to the `content` field and `ai_query` to `data`, which is useful since there is no output schema, but it does not explain cost tradeoffs among the many proxy/device options beyond what the schema already states.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Fetch a web page through a proxied browser and return its HTML -- or markdown, with `output` -- in `content`" names a specific verb and resource, and the mention of `ai_query` answers and job polling distinguishes it from siblings like `screenshot` and `crawl_recipe`. An agent can tell what this tool produces without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit routing is given for the async case: if `status` is "queued" or "running", call `crawl_poll` with the `job_id`. That is a concrete when-to-use-this-vs-alternative instruction. It stops short of saying when to prefer `crawl` over `crawl_recipe` or `screenshot`, so it is not a full 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources