Web Crawl
web_crawlCrawl one site's same-host pages starting from a URL (depth 1-2, at most 20 pages) and return each page's title, URL and readable text. When to use: you need several pages from one site (docs section, news index, product list) and web_map alone is not enough. Do not use: to mirror a whole site (hard cap 20 pages, same host), or for one page (web_fetch). Returns: {ok, url, final_url, count, pages:[{url, title, status, chars, content}], cached, took_ms, _metering}. Respects robots.txt and SSRF guards; partial page failures are skipped. Metered: shares the web free bucket (100 calls/day per IP shared by all web tools), then 3 credits per call over x402 - no API key.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start URL (same host is crawled). | |
| depth | No | Link depth 1-2. Default 2. | |
| max_chars | No | Per-page content cap 200-4000. Default 1000. | |
| max_pages | No | Max pages 1-20. Default 20. |