Is a URL allowed for a crawler by robots.txt? Rules, crawl-delay and sitemaps
web_robotsIs a URL allowed for a crawler by robots.txt? Rules, crawl-delay and sitemaps. Fetches the site's /robots.txt and evaluates a URL path for a user-agent with RFC 9309 matching (longest match wins, * and $ wildcards). Returns allowed/disallowed, the deciding rule, crawl-delay, declared sitemaps and the agent groups present. Price: $0.005 USDC per call via x402 (eip155:8453).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Page URL (its origin's robots.txt is read) | |
| userAgent | No | * |