Skip to main content
Glama

Site Crawl to Markdown

site_crawl
Read-only

Website crawler: read a whole website or one section of it (for example every page under /docs) as clean Markdown: starts at a URL, follows same-site links breadth-first and returns up to 20 pages with title and description. Use for documentation, product catalogs or company sites. For one page use web-scrape, for a known list of URLs web-read-batch (up to 5) or web-scrape-batch (up to 20). Charged only if at least one page was read. Params: url (required), max_pages (1 to 20, default 5), max_chars per page (500 to 20000, default 4000), path_prefix (e.g. /docs). Price: $0.005 USDC per call, paid inside this MCP call via x402 (USDC on Base, Solana, Polygon or Arbitrum). x402 MCP clients such as @x402/mcp pay automatically. Free sample: call with the example input {"url":"https://agentsvc.io/docs","max_pages":3,"max_chars":2000}.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYes
max_charsNo
max_pagesNo
path_prefixNoOnly follow links whose path starts with this, e.g. /docs

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare readOnly/openWorld/destructive=false, but the description adds substantial context beyond them: breadth-first traversal of same-site links, a 20-page ceiling, that results carry title and description, the billing rule ('charged only if at least one page was read'), and the full x402 payment model with chains and price. This is exactly the behavioral disclosure the annotations cannot supply.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose and alternatives, then params, then pricing. Dense but nearly every clause is load-bearing for a paid tool (payment path, free sample). It is longer than typical, but little is pure filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, yet the description explains the return ('up to 20 pages with title and description'), the traversal scope, the cost/billing condition and how payment is settled. Combined with usage routing and param guidance, an agent has everything needed to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is only 25% (just path_prefix), so the description must compensate, and it does: it marks url as required and enumerates max_pages (1-20, default 5), max_chars per page (500-20000, default 4000) and path_prefix with an example. It clarifies that max_chars is per page and supplies the '/docs' style example, though the numeric ranges largely duplicate the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('website crawler: read a whole website or one section of it... as clean Markdown') and immediately names what it is not: the alternatives web-scrape, web-read-batch and web-scrape-batch. An agent can distinguish it from every sibling without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit when-to-use cases ('documentation, product catalogs or company sites') plus explicit routing rules to alternatives ('For one page use web-scrape, for a known list of URLs web-read-batch (up to 5) or web-scrape-batch (up to 20)'). Both the positive selection condition and the alternatives are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources