Crawl website
crawl_siteMap a website by discovering its pages and returning each URL, HTTP status, and title while honoring robots.txt; use it to find broken pages, list blog posts, or pick URLs to fetch.
Instructions
Discover the pages of a website and return each page's URL, HTTP status and . Reads sitemap.xml (and sitemap indexes) when present, otherwise follows same-origin links breadth-first. Always honours robots.txt (disallowed URLs are listed under skipped) and Crawl-delay. Use it to map a site, find broken pages, list blog posts, or pick URLs to read with fetch_page. Page bodies are not returned — call fetch_page for content.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "crawl_site"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start URL. Only pages on the same origin (scheme + host + port) are crawled. | |
| strategy | No | "auto" uses sitemap.xml when available and falls back to following links; "sitemap" or "links" force one method. | auto |
| max_depth | No | Link depth from the start URL (only used by the "links" strategy). | |
| max_pages | No | Maximum pages to fetch (hard cap 200). | |
| path_prefix | No | Only crawl URLs whose path starts with this, e.g. "/blog/". |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| pages | Yes | ||
| stats | Yes | ||
| robots | Yes | ||
| sitemap | No | ||
| skipped | Yes | ||
| startUrl | Yes | ||
| strategy | Yes | ||
| truncated | Yes | ||
| durationMs | Yes |