Check robots.txt
robots_checkFetch a site's robots.txt and test paths against it the way Google does (most specific user-agent group, longest rule wins, Allow wins a tie, * and $ wildcards), for Googlebot, Bingbot and AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) or the user agents you pass. Also reports the sitemaps it declares and what a missing or failing robots.txt means.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| site | Yes | Any URL on the site. | |
| paths | No | Paths or URLs to test, up to 100. | |
| user_agents | No |