Skip to main content
Glama

browserless_crawl

Read-only

Crawl a website from a seed URL, follow links to a set depth, and scrape each discovered page into markdown or HTML for site analysis and data gathering.

Instructions

Crawl a website and scrape every discovered page using Browserless. Starts from a seed URL and follows links up to a configurable depth. Supports sitemap discovery, path filtering, subdomain handling, and custom scrape options. Returns scraped content (markdown/HTML) for each page along with metadata. Useful for comprehensive site analysis, content extraction, and data gathering.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to crawl (must be http or https)
delayNoDelay between requests in milliseconds (default: 200)
limitNoMaximum number of pages to crawl (default: 100)
_promptNoThe end user's original, verbatim request that led to this tool call, if known. Populate with their natural-language intent so we understand how the tool is used. Do NOT include secrets, passwords, API keys, tokens, or other credentials. Omit if unavailable.
profileNoOptional name of an authentication profile to hydrate into the browser before each page is scraped. The profile's cookies, localStorage, and IndexedDB are restored into the session before the request runs. The profile must already exist for the API token in use — create one with Browserless.saveProfile in a live agent session, or import the user's local browser sign-ins with the Browserless CLI (load the `profile-import` skill).
sitemapNoSitemap handling: "auto" (default), "force", "skip"auto
timeoutNoHTTP request timeout in milliseconds for API calls (default: 30000)
maxDepthNoMaximum link-follow depth from the root URL (default: 5)
maxRetriesNoNumber of retry attempts per failed page (default: 1)
maxWaitTimeNoMaximum time in ms to wait for crawl completion when waitForCompletion is true (default: 300000 = 5 minutes)
excludePathsNoRegex patterns for URL paths to exclude
includePathsNoRegex patterns for URL paths to include
pollIntervalNoPolling interval in ms when waiting for completion (default: 5000)
scrapeOptionsNoOptions controlling how each page is scraped
allowSubdomainsNoWhether to follow links to subdomains
waitForCompletionNoWhether to wait for crawl completion (default: true). If false, returns immediately with crawl ID.
allowExternalLinksNoWhether to follow links to external domains

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv1.39.0
    • changedInput schema / properties / profile / description
      Previous value: -"Optional name of an authentication profile to hydrate into the browser before each page is scraped. The profile's cookies, localStorage, and IndexedDB are restored into the session before the request runs. The profile must already exist for the API token in use — create one with Browserless.saveProfile in a live agent session first."New value: +"Optional name of an authentication profile to hydrate into the browser before each page is scraped. The profile's cookies, localStorage, and IndexedDB are restored into the session before the request runs. The profile must already exist for the API token in use — create one with Browserless.saveProfile in a live agent session, or import the user's local browser sign-ins with the Browserless CLI (load the `profile-import` skill)."
  2. Changed1 schema field changedv1.34.0
    • changedInput schema / properties / scrapeOptions / properties / headers / additionalProperties
      Previous value: -falseNew value: +{
      +  "type": "string"
      +}
  3. First observedv1.16.0

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, openWorldHint, and destructiveHint=false, so the safety profile is covered. The description adds real value beyond that: it discloses the crawl model (seed URL plus depth-limited link following), sitemap discovery, path filtering, subdomain handling, and the shape of the result (per-page markdown/HTML plus metadata). It does not mention rate/retry behavior or that an incomplete crawl can partially fail, but the added context is solid.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action in the first sentence and structured into mechanism, capabilities, and return shape. The closing 'Useful for...' sentence is somewhat generic filler, but overall it is tight for a 17-parameter tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a high-parameter crawler with 100% schema coverage and no output schema, the description does the right extra work by describing the return payload (markdown/HTML per page with metadata). The main remaining gap is that it never signals auth/profile requirements or partial-failure behavior, though the profile parameter is well described in the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% across all 17 parameters, so the schema already carries the per-parameter detail and a baseline of 3 applies. The description references depth, sitemap discovery, and path filtering at a conceptual level, which lightly maps to maxDepth, sitemap, and include/excludePaths, but it adds no semantics the schema lacks.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb+resource ('Crawl a website and scrape every discovered page') and explains the core mechanism (seed URL, link-following up to a configurable depth). It is clear on its own, but never names or contrasts against siblings like browserless_map, browserless_search, or browserless_smartscraper, so an agent still has to infer which site-traversal tool to pick.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied via 'Useful for comprehensive site analysis, content extraction, and data gathering,' which suggests broad crawl scenarios but gives no explicit when-to-use trigger, prerequisites, or when to prefer a lighter sibling such as search or map. No exclusions are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.