crawl
crawlRead-only
Crawl a website into clean Markdown: follows same-site links from a start page (optionally under a path prefix), up to 100 pages, respecting robots.txt. Returns each page's URL, title and text. $0.005 + $0.001 per page. Add store=true (+$0.001) to get a 7-day share link instead of the data inline. Costs $0.025 USDC per call (x402).
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start page | |
| store | No | Return a 7-day share link (+$0.001) | |
| max_pages | No | 1-100, default 20 | |
| path_prefix | No | Only follow links under this path, e.g. /docs/ |