Skip to main content
Glama
Sign In
crawlbrulee

crawlbrulee-mcp

Official
by crawlbrulee
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

Scrape a URL asynchronously

scrape_async

Submit a scrape job asynchronously and get a job_id right away, so long-running scrapes with heavy JS rendering or full-page screenshots never block the connection. Poll with scrape_status and fetch results when done.

Instructions

Submit a scrape job to run ASYNCHRONOUSLY and return a job_id immediately, instead of holding the connection open. Use this for long-running scrapes (heavy JS rendering, full-page screenshots of long pages) — for a quick one-shot fetch prefer the synchronous scrape tool, which blocks and returns the page directly. Poll the job with scrape_status and fetch the page with scrape_result once done. Takes extract.elements like scrape (named values by CSS selector, no extra credits); the result carries them in elements. Rules: https://crawlbrulee.com/docs/scrape/elements. Optionally attach a per-job completion webhook: we deliver a single signed scrape.complete POST to your endpoint when the job finishes (HTTPS required in production), and echoes your opaque webhook.metadata back in the delivery (under data.metadata, alongside data.page_status_code (on success) and data.response_meta.usage). Set zero_data_retention: true to keep the result out of the shared cache; anything stored to deliver it is kept for 24 hours, then deleted. It adds 1 credit and must be enabled for the organization (else a zero_data_retention_not_enabled error, not billed). See https://crawlbrulee.com/docs/zero-data-retention.

Input Schema

NameRequiredDescriptionDefault
urlYesThe URL to scrape. Known tracking parameters are removed before the page is fetched, so they are neither sent to the target site nor part of the cache key. Every other query parameter is kept verbatim and is part of the cache key.
cacheNoCache settings for this request
proxyNoProxy tier to use for fetchingauto
cleanupNoWhat is removed from the page before any output is built. Applies to markdown, cleaned_html, links and images on every engine, and to the screenshot. Never applies to raw_html, which is always the page before we removed anything.
extractNoWhich content formats to extract. Defaults to metadata + cleaned_html.
webhookNoCompletion webhook delivered when this async job finishes. Async-only: the synchronous `scrape` tool does not accept it.
locationNoOptional locale + country emulation for the scrape
require_jsNoUse a headless browser to render JavaScript before scraping
zero_data_retentionNoKeeps the result out of the shared cache; anything stored to deliver it is kept for 24 hours, then deleted. Adds 1 credit. Must be enabled for your organization. See https://crawlbrulee.com/docs/zero-data-retention.

Output Schema

NameRequiredDescriptionDefault
job_idYesJob identifier — pass it to the `scrape_status` / `scrape_result` tools

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv1.3.3
    • changedInput schema / properties / cache / properties / max_age / anyOf
      Previous value: -[
-  {
-    "maximum": 9007199254740991,
-    "minimum": 0,
-    "type": "integer"
-  },
-  {
-    "format": "date-time",
-    "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d(?::[0-5]\\d(?:\\.\\d+)?)?(?:Z))$",
-    "type": "string"
-  }
-]New value: +[
+  {
+    "maximum": 9007199254740991,
+    "minimum": 0,
+    "type": "integer"
+  },
+  {
+    "format": "date-time",
+    "pattern": "^(?:(?:\\d\\d[2468][048]|\\d\\d[13579][26]|\\d\\d0[48]|[02468][048]00|[13579][26]00)-02-29|\\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\\d|30)|(?:02)-(?:0[1-9]|1\\d|2[0-8])))T(?:(?:[01]\\d|2[0-3]):[0-5]\\d:[0-5]\\d(?:\\.\\d+)?(?:Z))$",
+    "type": "string"
+  }
+]
  2. Changed4 schema fields changedv1.1.1
    • changedInput schema / properties / extract / properties / cleaned_html / description
      Previous value: -"Extract cleaned HTML (main content only)"New value: +"Extract cleaned HTML: the page body with scripts, styles, ads and cookie banners removed"
    • addedInput schema / properties / extract / properties / elements
      Added value: +{
+  "additionalProperties": {
+    "anyOf": [
+      {
+        "maxLength": 500,
+        "type": "string"
+      },
+      {
+        "additionalProperties": false,
+        "properties": {
+          "all": {
+            "default": false,
+            "description": "Return every match as a list instead of the first. Defaults to false",
+            "type": "boolean"
+          },
+          "attribute": {
+            "description": "The attribute to read when `output` is `attribute`. `href` and `src` come back as full urls",
+            "maxLength": 100,
+            "minLength": 1,
+            "type": "string"
+          },
+          "fields": {
+            "additionalProperties": {
+              "anyOf": [
+                {
+                  "maxLength": 500,
+                  "type": "string"
+                },
+                {
+                  "additionalProperties": false,
+                  "properties": {
+                    "all": {
+                      "default": false,
+                      "type": "boolean"
+                    },
+                    "attribute": {
+                      "maxLength": 100,
+                      "minLength": 1,
+                      "type": "string"
+                    },
+                    "fields": {
+                      "additionalProperties": {
+                        "anyOf": [
+                          {
+                            "maxLength": 500,
+                            "type": "string"
+                          },
+                          {
+                            "additionalProperties": false,
+                            "properties": {
+                              "all": {
+                                "default": false,
+                                "type": "boolean"
+                              },
+                              "attribute": {
+                                "maxLength": 100,
+                                "minLength": 1,
+                                "type": "string"
+                              },
+                              "fields": {
+                                "additionalProperties": {
+                                  "anyOf": [
+                                    {
+                                      "maxLength": 500,
+                                      "type": "string"
+                                    },
+                                    {
+                                      "additionalProperties": false,
+                                      "properties": {
+                                        "all": {
+                                          "default": false,
+                                          "type": "boolean"
+                                        },
+                                        "attribute": {
+                                          "maxLength": 100,
+                                          "minLength": 1,
+                                          "type": "string"
+                                        },
+                                        "output": {
+                                          "enum": [
+                                            "text",
+                                            "html",
+                                            "attribute"
+                                          ],
+                                          "type": "string"
+                                        },
+                                        "selector": {
+                                          "maxLength": 500,
+                                          "type": "string"
+                                        }
+                                      },
+                                      "required": [
+                                        "selector"
+                                      ],
+                                      "type": "object"
+                                    }
+                                  ]
+                                },
+                                "propertyNames": {
+                                  "type": "string"
+                                },
+                                "type": "object"
+                              },
+                              "output": {
+                                "enum": [
+                                  "text",
+                                  "html",
+                                  "attribute"
+                                ],
+                                "type": "string"
+                              },
+                              "selector": {
+                                "maxLength": 500,
+                                "type": "string"
+                              }
+                            },
+                            "required": [
+                              "selector"
+                            ],
+                            "type": "object"
+                          }
+                        ]
+                      },
+                      "propertyNames": {
+                        "type": "string"
+                      },
+                      "type": "object"
+                    },
+                    "output": {
+                      "enum": [
+                        "text",
+                        "html",
+                        "attribute"
+                      ],
+                      "type": "string"
+                    },
+                    "selector": {
+                      "maxLength": 500,
+                      "type": "string"
+                    }
+                  },
+                  "required": [
+                    "selector"
+                  ],
+                  "type": "object"
+                }
+              ]
+            },
+            "description": "Used instead of `output`: turns each match into an object of named values, each read inside that match only. Same shape as `elements`; nests up to 3 levels",
+            "propertyNames": {
+              "type": "string"
+            },
+            "type": "object"
+          },
+          "output": {
+            "description": "What to return for a match: `text` (default), `html` (the outer HTML) or `attribute`",
+            "enum": [
+              "text",
+              "html",
+              "attribute"
+            ],
+            "type": "string"
+          },
+          "selector": {
+            "description": "CSS selector, up to 500 characters",
+            "maxLength": 500,
+            "type": "string"
+          }
+        },
+        "required": [
+          "selector"
+        ],
+        "type": "object"
+      }
+    ]
+  },
+  "description": "Named values to read from the page by CSS selector, returned in the top-level `elements` field under the same names. Each value is either a selector string (the text of the first match) or an object with `selector`, `output`, `attribute` and `all`. `fields` (instead of `output`) turns each match into an object of named values read inside that match only, so a list of product cards can return a title, price and url per card. A name with no match is null, or [] with `all: true`. Read after cleanup; `<script>` and `<style>` tags are not part of it. An invalid selector is a 400. No extra credits, also when the page comes from cache. Limits and selector rules: https://crawlbrulee.com/docs/scrape/elements.",
+  "propertyNames": {
+    "type": "string"
+  },
+  "type": "object"
+}
    • changedInput schema / properties / extract / properties / metadata / description
      Previous value: -"Extract page metadata (title, description, OG tags, etc.). Read from the page head, which is capped at 2,000,000 characters — a larger head is truncated and the response carries a `metadata_truncated` warning."New value: +"Extract page metadata (title, description, OG tags, etc.) from the page head"
    • addedInput schema / properties / zero_data_retention
      Added value: +{
+  "description": "Keeps the result out of the shared cache; anything stored to deliver it is kept for 24 hours, then deleted. Adds 1 credit. Must be enabled for your organization. See https://crawlbrulee.com/docs/zero-data-retention.",
+  "type": "boolean"
+}
  3. First observedv1.0.3

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full behavioral burden and does so thoroughly: immediate job_id return, webhook delivery as a single signed POST with signature header and metadata echo, zero_data_retention storage deleted after 24 hours, the 1-credit cost and organization-enablement prerequisite with named error `zero_data_retention_not_enabled` and 'not billed' clarification. These are material behaviors not present in the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core routing information and async-vs-sync distinction are front-loaded, and most sentences add value. It is still a dense block with a long zero_data_retention paragraph that could be trimmed, but the structure remains logical and readable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given nine nested parameters and an output schema, the description covers the essential workflow: submission, immediate job_id, polling via `scrape_status`, retrieval via `scrape_result`, optional webhook completion, and retention semantics. An agent has enough to invoke and follow up correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description touches `extract.elements` and `webhook.metadata` but largely restates what the schema already documents in detail; it does not explain other significant parameters such as `cache`, `proxy`, `cleanup`, `location`, or `require_js`.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource ('Submit a scrape job') and immediately states the defining behavioral contrast: asynchronous with an immediate job_id instead of holding the connection open. It explicitly distinguishes itself from the synchronous `scrape` sibling, so an agent can route correctly without inspecting either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit when-to-use guidance ('long-running scrapes — heavy JS rendering, full-page screenshots of long pages') and when-not ('for a quick one-shot fetch prefer the synchronous `scrape` tool'). It also names the follow-up tools `scrape_status` for polling and `scrape_result` for fetching, completing the workflow.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools