crawl
Extract pages from a site starting at a URL and return them as Markdown, following links breadth-first within depth, page, and domain limits while listing skipped URLs with reasons.
Instructions
Crawl a site starting at url and return each page as Markdown.
Follows links breadth-first up to max_depth link hops (limit 5) and max_pages pages (limit 100). same_domain=True stays on the starting host. Skipped URLs (robots.txt, blocked, errors) are listed with the reason. Long crawls stop at a time/size budget and return the pages collected so far.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "crawl"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| max_depth | No | ||
| max_pages | No | ||
| render_js | No | ||
| same_domain | No |