Skip to main content
Glama

Crawl a website

web_crawl_start

Crawl a website from a start URL, following links and reading pages in a real browser as markdown, text, or HTML. Use it to extract site content while respecting robots.txt and host limits.

Instructions

Crawl a website from a start URL (no session needed): follow its links, read each page in a real browser and return the pages as markdown, text or HTML. Respects robots.txt and stays on the start page's host unless sameHost is false. Waits up to waitSeconds (default 45) and returns the pages read so far; while it is still running, web_crawl_get gives the rest. Public pages only.

Input Schema

NameRequiredDescriptionDefault
urlYes
formatNo
excludeNoNever follow URLs that match one of these regular expressions
includeNoOnly follow URLs that match one of these regular expressions
maxDepthNoLink depth from the start page, 0 to 10 (default 3)
maxPagesNoPages to read, 1 to 1000 (default 20; the plan's crawl limit applies)
sameHostNoStay on the start page's host (default true)
waitSecondsNoHow long to wait for the crawl before answering (default 45)

Schema Changelog

Changes observed during successful MCP inspections.

  1. Addedv3.1.0

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare openWorldHint=true and destructiveHint=false, so the safety profile is partly covered; the description adds the non-obvious behaviors: robots.txt compliance, same-host scoping by default, a bounded wait of waitSeconds, and the asynchronous partial-result contract with web_crawl_get. Only missing detail is whether an in-flight crawl can be cancelled or resumed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, no filler, and the critical facts (what it does, default scope, partial-return behavior) are front-loaded. Every clause carries distinct information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 8-parameter asynchronous crawl tool with no output schema, the description covers scope, safety, timing, output formats, and the two-phase retrieval pattern. An agent has everything needed to start a crawl and know what to expect back.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 75%, and the description fills the gaps for the undescribed parameters: format (markdown/text/HTML), sameHost (host scoping), and waitSeconds (default 45). It adds no semantics for exclude/include/maxDepth/maxPages beyond what the schema already documents, but those are already well-described.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource (crawl a website from a start URL), names the output formats, and explicitly contrasts itself with the session-based siblings ("no session needed"). It also names web_crawl_get as the continuation path, so an agent can place it precisely among web_fetch/web_extract/web_crawl_get.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives real routing context: no session required, public pages only, and "while it is still running, web_crawl_get gives the rest" tells the agent exactly what to do next. It does not explicitly say when to prefer web_fetch or web_extract over a crawl, so it stops short of full when-not guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.