AI crawler access check
ai_crawler_accessCheck if AI and search crawlers can reach a page by testing robots.txt rules and live bot User-Agent requests, so you can detect blocks that prevent citation.
Instructions
Check whether AI and search crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, Applebot, Amazonbot, Meta, CCBot, Bytespider...) can reach a page: robots.txt rules for the site root and for the URL, plus a live request with each bot's User-Agent to detect UA-based blocks (e.g. Cloudflare 'block AI bots', WAF rules). Being blocked from OAI-SearchBot, PerplexityBot or Claude-SearchBot means the site cannot be cited by those assistants.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "ai_crawler_access"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | A representative page, e.g. the homepage or an important article. | |
| liveFetch | No | Also request the page with each bot UA (one request per bot). |