Skip to main content
Glama

Crawl Site

crawl_site

Crawl an entire website from a seed URL using DFS to extract markdown or JSON content, with configurable depth, page, and timeout limits.

Instructions

Crawl an entire website starting from a seed URL using DFS strategy.

Input Schema

NameRequiredDescriptionDefault
urlYesThe seed URL to start crawling from
timeoutNoOverall site crawl timeout in seconds (default: 120, must be >= 1)
max_depthNoMaximum depth to crawl (default: 2, 0 = seed page only)
max_pagesNoMaximum number of pages to crawl (default: 25)
dedup_modeNoMarkdown dedup mode - "exact" (default) or "off"exact
remove_linksNoRemove all links from the markdown output (default: false)
output_formatNoOutput format - "markdown" (default) or "json" - markdown: Clean concatenated markdown with URL headers and timestamps - json: Full JSON with metadata, references, and crawl statisticsmarkdown
storage_stateNoPath to Playwright storage_state JSON for authenticated crawling
include_subdomainsNoWhether to include subdomains in the crawl (default: false)

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.30.0

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It discloses the DFS traversal strategy, which is useful, but says nothing about rate limiting, scope restrictions, long-running behavior against the 120s default timeout, or authentication requirements despite a storage_state parameter.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no filler. It is efficient, though the extreme brevity borders on under-specification for a nine-parameter crawling tool rather than reflecting purposeful conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex, potentially long-running, resource-intensive crawler with nine parameters, no annotations, and no output schema, one sentence is insufficient. It omits operational limits, output characteristics, and safety considerations an agent needs before invoking it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with every parameter (timeout, max_depth, max_pages, dedup_mode, remove_links, output_format, storage_state, include_subdomains) fully documented in the schema. The description adds no parameter-level meaning beyond that, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Crawl) and resource (an entire website) along with the starting point (seed URL) and traversal strategy (DFS). The word 'entire' implies a multi-page operation distinct from the sibling 'crawl', but the differentiation from 'crawl' and 'search' is never made explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to choose this tool over the 'crawl' or 'search' siblings, nor any mention of prerequisites such as robots.txt compliance, politeness, or when authenticated crawling is warranted. Usage must be entirely inferred from the phrase 'entire website'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools