Check robots.txt for AI crawlers
check_robotsRead a site's robots.txt and report, for 17 search and AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, CCBot, Google-Extended and more), whether each may fetch the home page and an optional path, and which rule decides. Uses RFC 9309 matching. Works without an API key (limited per day).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Site or page address, e.g. example.com or https://example.com/pricing | |
| path | No | Optional path to check as well, e.g. /blog/post. Defaults to the path in url. |