May a crawler fetch this URL?
robots_checkReads the site's robots.txt and says whether a crawler may fetch a URL (RFC 9309: the longest matching rule wins, Allow wins a tie), with the matching rule, crawl-delay and sitemaps. No AI involved; if robots.txt cannot be read you are not charged. Price: $0.01 USDC per call (x402).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL you want to fetch | |
| user_agent | No | Your crawler's name, for example MyBot (default: any crawler) |