can_fetch
Does robots.txt allow fetching this exact URL? Fetches the live robots.txt and applies the longest-match rule for a user agent (default: * plus GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot). Returns allowed or not, the rule that decided it, and any crawl delay. A courtesy check, not a legal opinion.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The full URL you want to fetch | |
| user_agent | No | optional, e.g. GPTBot |