AI crawler access check
ai_crawler_accessCheck AI and search bot access by analyzing robots.txt and live-fetching the page with each bot's user-agent to identify blocks that prevent citations.
Instructions
Check whether AI and search crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, Applebot, Amazonbot, Meta, CCBot, Bytespider...) can reach a page: robots.txt rules for the site root and for the URL, plus a live request with each bot's User-Agent to detect UA-based blocks (e.g. Cloudflare 'block AI bots', WAF rules). Being blocked from OAI-SearchBot, PerplexityBot or Claude-SearchBot means the site cannot be cited by those assistants.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | A representative page, e.g. the homepage or an important article. | |
| liveFetch | No | Also request the page with each bot UA (one request per bot). |