Skip to main content
Glama

Start a crawl

crawl

Starts a site-section sweep from a URL, following links in the start path's subtree up to page and depth limits, and returns a task ID to track progress.

Instructions

Start crawling a section of one site: from url, follow links in the start path's subtree on its host and www twin, up to maxPages pages and maxDepth links deep. Returns { taskId } at once (with maxPages when the default of 100 applied); wait_crawl waits for it, then get_crawl_pages reads the pages. How long the server keeps them is not stated. Each page takes scrape's page options (formats, onlyMainContent, includeTags, waitFor, timeout, verify and the rest; their descriptions are on scrape). Similar URLs fold into one page, and every collapsed or refused link is counted in get_crawl's discovery.

Input Schema

NameRequiredDescriptionDefault
urlYes
modeNostandard (default) or research; for signed-in pages send a batch in mode authed.
accessNostandard (default) or enhanced: also the paid providers the operator approved.
maxAgeNo
minAgeNo
mobileNo
verifyNoSame as scrape.
formatsNoSame as scrape.
headersNo
sitemapNoinclude (default): also queue the site's sitemap URLs. skip: no sitemap. only: the start URL and the sitemap entries.
timeoutNo
waitForNo
webhookNoWhere the job posts events, retried: a URL or { url, headers?, metadata?, events?, secretEnv? } (a W2L_WEBHOOK_SECRET_* variable signing them).
blockAdsNo
fastModeNo
lockdownNo
maxDepthNoLinks followed from the start URL at most. Omitted or null: no limit.
maxPagesNoPages fetched at most. Omitted: 100. null: the server's default (none on a local server; hosted servers cap it).
useCachedNo
excludeTagsNo
includeTagsNo
integrationNoYour workflow label (printable, no spaces); recorded, not sent to sites.
excludePathsNoPathname regexes that skip a discovered link; they win over includePaths.
includeLinksNo
includePathsNoPathname regexes a discovered link must match; the start URL is always fetched.
maxFileBytesNo
storeInCacheNo
idempotencyKeyNoMakes a retry safe: the same key and arguments return the first taskId. Kept 24 hours.
maxConcurrencyNoPages fetched at once, at most (only lowers the worker count).
regexOnFullURLNoMatch includePaths and excludePaths against each link's canonical URL instead of its pathname. Default false.
allowSubdomainsNoFollow links to every host under the start URL's apex. Default false.
ignoreRobotsTxtNoAlso fetch what robots.txt disallows (recorded, with a warning). Default false. Local server only.
onlyMainContentNo
crawlEntireDomainNoFollow links anywhere on the start host, not only in the start path's subtree. Default false.
allowExternalLinksNoFollow links to any host, each with its own robots.txt; maxDepth and maxPages bound the walk. Default false. Not with allowlistedDomains.
allowlistedDomainsNo
removeBase64ImagesNo
skipTlsVerificationNo
ignoreQueryParametersNoTreat URLs that differ only in their query as one page. Default false.
deduplicateSimilarURLsNoTreat /a and /a/, www and apex, http and https as one page. Default true.

Output Schema

NameRequiredDescriptionDefault
taskIdYes
replayedNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.7.0

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With annotations already declaring it non-read-only, open-world, non-idempotent, the description adds real behavioral context beyond them: the async taskId return, that the 100-page default is echoed back, that retention duration is unknown to the client, and that collapsed/refused links are still tallied in get_crawl's discovery. It does not discuss idempotencyKey semantics or rate/robots constraints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Five dense sentences with the traversal scope front-loaded and the deferred workflow immediately after; little padding. The 'Returns { taskId }' sentence slightly overlaps an existing output schema, but the appended maxPages detail makes it earn its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 40-parameter tool with annotations and an output schema, it covers the essentials: what it traverses, what it returns immediately, how to retrieve results, and where page-option semantics live. The remaining gaps (most boolean/array flags, retention policy) are acknowledged or handled by the schema rather than silently omitted.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 40 parameters at 50% schema coverage, the description can only partially compensate. It adds meaning for maxPages (the default comes back in the response), maxDepth, and URL folding, and cross-references scrape for the page options, but the bulk of the option surface is left to the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

It names a specific verb and resource ('Start crawling a section of one site') and bounds the scope precisely (start path's subtree, own host + www twin). That scope description implicitly separates it from scrape (one page) and map (URL discovery only), though it never names those siblings explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear operational context: it returns { taskId } at once, so wait_crawl, then get_crawl_pages, is the intended follow-up chain. It also routes page-option questions to scrape's parameter descriptions. It stops short of stating when NOT to crawl (e.g. use map for URL listing, scrape for a single page).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.