check_ai_access
Check if AI crawlers like GPTBot, ClaudeBot, and PerplexityBot are allowed to access a URL by reviewing robots.txt rules and fetching the page with each bot's user-agent.
Instructions
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) may access a URL: robots.txt decision per bot plus the live HTTP status when the page is fetched with each bot's user-agent string. Read-only; one request per bot.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL of the page to check. | |
| bots | No | Bot tokens to check, or omit for the default 15. Known: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Googlebot, Bingbot, Applebot, meta-externalfetcher, DuckAssistBot, Amazonbot, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit, Slackbot. | |
| http | No | Also fetch the page with each bot's user agent (default true). false checks robots.txt only. |