Skip to main content
Glama
ImYourBoyRoy

web-scraper-server

by ImYourBoyRoy

crawl_site

Crawl a site's sitemap to discover all pages and return a list of URLs.

Instructions

Crawl a site's sitemap to discover pages.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYes
timeout_profileNostandard

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
resultYes
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. The description only states that it crawls a sitemap to discover pages, but doesn't disclose whether it follows nested sitemaps, respects robots.txt, handles errors, or what the actual output looks like. This is completely non-transparent about side effects, rate limits, or read-only guarantees.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence, which is succinct but severely under-specified. It is not merely concise; it omits critical details about parameters, behavior, and output. The extreme brevity resembles a placeholder rather than a well-crafted tool description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has two parameters, no annotations, and an output schema, but the description fails to provide any additional context. It doesn't explain the expected input format, the nature of the output, possible limitations, or error scenarios. For a moderately complex crawling tool, this description is completely inadequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate by explaining the parameters. However, the description does not mention 'url' or 'timeout_profile' at all. It's ambiguous whether 'url' is the sitemap URL or the site's base URL, and 'timeout_profile' is entirely unexplained. The description adds no value beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('crawl') and the target ('a site's sitemap') with the purpose ('discover pages'). This distinguishes it from sibling tools like get_sitemap (which likely only retrieves the sitemap) and scrape_url (which scrapes a specific page). However, it doesn't specify the output format or mention whether it returns the discovered pages directly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. For example, it doesn't explain when to prefer crawl_site over get_sitemap or extract_links, nor does it mention any prerequisites or exclusions. The purpose is implied but no explicit context is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server

Other Tools

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/ImYourBoyRoy/WebScraperToolkit'

If you have feedback or need assistance with the MCP directory API, please join our Discord server