Skip to main content
Glama

Start a site crawl

crawl

Start an async crawl of one website: pages read as Markdown, following links up to max_depth hops (and the site's sitemaps), honouring robots.txt. With query, the most relevant pages are read first. Returns a task id; call crawl_result with it. 1 credit per page read; limit credits are reserved and the rest refunded.

Input Schema

NameRequiredDescriptionDefault
urlYesStart page
limitNoMaximum pages (default 25)
queryNoRead the pages most relevant to this first
max_depthNoLink hops from url (default 2)
max_tokensNoCap per page
exclude_pathsNoRegular expressions; never follow URL paths matching one
include_pathsNoRegular expressions; follow only URL paths matching one

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds substantial behavior beyond the annotations: async execution, robots.txt compliance, sitemap following, Markdown page output, relevance ordering when query is set, and a credit model where limit credits are reserved and the remainder refunded. That cost/reservation detail is the kind of operational context annotations do not cover.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the core action, then the query/async behavior, then the return-and-cost contract. No filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter async job tool with no output schema, the description covers the essentials an agent needs: async nature, the task-id handoff to crawl_result, content format, crawl rules, and the credit/limit economics.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description does enrich query (relevance-first ordering) and limit (credit reservation/refund economics), but adds nothing about exclude_paths/include_paths or max_tokens beyond their schema text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource+scope: 'Start an async crawl of one website', with concrete mechanics (links to max_depth hops, sitemaps, robots.txt, Markdown). This distinguishes it from map/extract/webpage siblings that don't recursively follow links.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear operating context: it is async, returns a task id, and explicitly routes the agent to crawl_result to collect results, plus cost semantics for choosing a limit. It does not, however, state when to prefer crawl over map or extract.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.