Skip to main content
Glama
jakobautomation

agentsvc-mcp

site_crawl

Crawl a website or one section into clean Markdown by following same-site links from a URL, returning up to 20 pages with titles and descriptions for docs, catalogs, or company sites.

Instructions

Site Crawl to Markdown — Website crawler: read a whole website or one section of it (for example every page under /docs) as clean Markdown: starts at a URL, follows same-site links breadth-first and returns up to 20 pages with title and description. Use for documentation, product catalogs or company sites. For one page use web-scrape, for a known list of URLs web-read-batch (up to 5) or web-scrape-batch (up to 20). Charged only if at least one page was read. Params: url (required), max_pages (1 to 20, default 5), max_chars per page (500 to 20000, default 4000), path_prefix (e.g. /docs). Cost: $0.005 USDC per call. Latency p99: 60000ms.

Input Schema

NameRequiredDescriptionDefault
urlYes
max_charsNo
max_pagesNo
path_prefixNoOnly follow links whose path starts with this, e.g. /docs

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv1.0.0

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well: it discloses crawl strategy (breadth-first, same-site links), the maximum result count, returned fields (title and description), billing condition ('charged only if at least one page was read'), cost ($0.005 USDC) and p99 latency (60000ms). It does not cover failure/partial-crawl edge cases or whether robots.txt is respected, which keeps it just below a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core purpose and scope, followed by alternatives, billing and parameter summary, then cost/latency. Dense but every sentence carries information; minor repetition of schema values keeps it a 4 rather than a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with no annotations and no output schema, the description supplies behavior, return shape (pages with title/description), limits, billing semantics, cost and latency. Reasonably complete, with only edge-case behavior (failed pages, crawl cutoff) missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 25% (only path_prefix is described), so the description must compensate, and it does: it lists url (required), max_pages (1–20, default 5), max_chars per page (500–20000, default 4000), and path_prefix with an example. It adds the useful semantic that max_chars applies per page, meaning all four parameters are understandable without opening the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (crawl a website/section) and resource (pages as Markdown), with scope details like breadth-first same-site following and a 20-page cap. It explicitly distinguishes itself from siblings: 'For one page use web-scrape, for a known list of URLs web-read-batch... or web-scrape-batch'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Names concrete use cases (documentation, product catalogs, company sites) and explicit alternatives with their limits (web-scrape for one page, web-read-batch up to 5, web-scrape-batch up to 20). The agent knows exactly when this tool is the right choice versus siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools