Read a small site
telos.crawl.siteCrawl several pages of one site read-only: reads sitemaps and feeds first, follows links within depth/page limits, obeys robots.txt, returns JSON summaries and a receipt.
Instructions
Use when an agent needs several pages of one site; sitemaps and feeds are read before links, inside a depth (0 to 3) and page budget (1 to 50). Read-only and zero-auth; obeys robots.txt and Crawl-delay. Reaches the network: one honest Telos User-Agent, per-host pacing with backoff on 429 and 503, no login, no cookies. Returns JSON page summaries and a receipt.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | start URL | |
| depth | No | link depth | |
| same_host | No | ||
| page_budget | No | maximum pages |