site-crawl-markdown
Crawls one public website breadth-first from a start URL, staying on the same hostname, and returns each page as clean Markdown (up to 20,000 chars per page). Reads robots.txt and skips disallowed paths, fetches at most 2 pages at a time with about 300ms between requests, and identifies as OrchardBot/1.0. Blocks localhost and private addresses, including after redirects. Answers within 45 seconds; if time runs out it returns the pages it has with partial=true. Returns JSON with start_url, pages_crawled, partial, pages[{url,title,markdown}] and skipped[{url,reason}]. max_pages is 1-20 (default 10). Paid only: $0.25 per call via x402 at https://tools.orchardshop.xyz/t/site-crawl-markdown.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>",
"max_pages": "<max_pages>"
},
"name": "site-crawl-markdown"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | e.g. https://example.com | |
| max_pages | Yes | e.g. 3 |