Skip to main content
Glama
eitan3
by eitan3

start_crawl

Start a bounded breadth-first crawl from a starting URL and receive a job ID immediately, which you can use to poll crawl status and retrieve results asynchronously.

Instructions

Start a bounded breadth-first crawl. Returns a job_id immediately (non-blocking).

The crawl follows in-scope links from start_url up to max_pages / max_depth.

  • allow_patterns / deny_patterns: regexes a link URL must match / must not match.

  • same_domain (default true): restrict the crawl to start_url's host.

  • selectors: same format as the extract tool; when given, each crawled page yields the extracted data. Otherwise each page yields a short text excerpt.

  • obey_robots, download_delay: per-crawl overrides of the server defaults.

Poll progress with crawl_status(job_id) and read items with crawl_results(job_id).

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
max_depthNo
max_pagesNo
selectorsNo
start_urlYes
obey_robotsNo
same_domainNo
deny_patternsNo
allow_patternsNo
download_delayNo
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of disclosing behavior. It covers the non-blocking nature, bounded limits, same_domain default, robots/delay overrides, and selector semantics. It does not mention cancellation via cancel_crawl or potential rate limits, but the disclosed details are substantial and contextually useful.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, front-loaded with the most important behavioral fact (non-blocking, returns job_id), and uses bullet-like separations for parameters. Every sentence adds value, and the overall length is appropriate for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex crawl tool with no output schema, the description covers the initiation, parameter semantics, and follow-up tools, which is largely complete. It omits details about cancellation or error handling, but the presence of sibling tools like crawl_status and cancel_crawl partially fills that gap. Given the high complexity, a 4 is appropriate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for all 9 parameters. It explains allow/deny_patterns as regex constraints, same_domain as host restriction, selectors as same format as extract tool, and obey_robots/download_delay as per-crawl overrides. This fully covers the non-obvious parameters, leaving little ambiguity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool starts a bounded breadth-first crawl and immediately returns a job_id, using a specific verb and resource. It distinguishes itself from sibling tools like crawl_status and crawl_results by focusing on initiation rather than status checking or retrieval.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context on how the crawl behaves and explicitly instructs to poll with crawl_status and read results with crawl_results. It does not explicitly contrast with alternatives like fetch_page or get_sitemap, but the follow-up workflow is clearly indicated, earning a 4 rather than a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server

Other Tools

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/eitan3/Scrapy_MCP_Scraper'

If you have feedback or need assistance with the MCP directory API, please join our Discord server