Skip to main content
Glama

site-crawl-markdown

Crawls one public website breadth-first from a start URL, staying on the same hostname, and returns each page as clean Markdown (up to 20,000 chars per page). Reads robots.txt and skips disallowed paths, fetches at most 2 pages at a time with about 300ms between requests, and identifies as OrchardBot/1.0. Blocks localhost and private addresses, including after redirects. Answers within 45 seconds; if time runs out it returns the pages it has with partial=true. Returns JSON with start_url, pages_crawled, partial, pages[{url,title,markdown}] and skipped[{url,reason}]. max_pages is 1-20 (default 10). Paid only: $0.25 per call via x402 at https://tools.orchardshop.xyz/t/site-crawl-markdown.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>",
      "max_pages": "<max_pages>"
    },
    "name": "site-crawl-markdown"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYese.g. https://example.com
max_pagesYese.g. 3

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With zero annotations, the description carries the entire burden and does so thoroughly: robots.txt compliance, 2 concurrent fetches, ~300ms delay, OrchardBot/1.0 identity, localhost/private-address blocking including post-redirect, and a 45-second cap that returns partial=true rather than failing. Cost and payment mechanism are also disclosed, which is behavior an agent must know before invoking.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense but ordered: what it does, how it behaves, its limits, what it returns, then the parameter range, then pricing. Every sentence adds a fact an agent needs, and the return shape is stated compactly as a single field list.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, yet the return JSON (start_url, pages_crawled, partial, pages[{url,title,markdown}], skipped[{url,reason}]) is spelled out, along with the partial-result behavior and the 20,000-char per-page cap. Nothing material for correct invocation is missing even for a fairly complex crawl tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real meaning: max_pages is constrained to 1-20 with a default of 10, and the url parameter implicitly carries a same-hostname crawl scope that the schema's bare example does not convey. Those additions go beyond the schema's 'e.g. https://example.com' and 'e.g. 3'.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific verb and resource ('Crawls one public website breadth-first from a start URL') and immediately fixes the scope ('staying on the same hostname'), which separates it from the single-page siblings like page-to-markdown and bulk-page-to-markdown without naming them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear context for when this applies: one public site, same hostname, max_pages 1-20, paid at $0.25 per call, 45-second budget. It stops short of explicitly routing the agent to a sibling for other tasks (e.g. auditing or single-page conversion) or stating when not to use it, so it is strong but not exhaustive.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources