May this user-agent crawl this URL? (RFC 9309)
robots_checkCrawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Page URL to test | |
| user_agent | No | Crawler product token, e.g. GPTBot or MyAgent. Default * |