AI crawler access
ayrank_ai_crawler_accessCheck if GPTBot, ClaudeBot, PerplexityBot and Google-Extended may crawl a homepage via robots.txt, meta robots and X-Robots-Tag; returns allowed/blocked per bot with the deciding line.
Instructions
Use to check whether GPTBot, ClaudeBot, PerplexityBot and Google-Extended may crawl a site's homepage under its robots.txt, plus the homepage's meta robots and X-Robots-Tag rules for them. Returns allowed or blocked per bot with the deciding line. Not for one specific page or another user agent (use ayrank_robots_txt_test), and it cannot see CDN or firewall rules that fire only for the real bots, because it sends an AY Rank user agent. Free AY Rank API, no key. JavaScript is not run, so content added by scripts is not seen.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"domain": "<domain>"
},
"name": "ayrank_ai_crawler_access"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | The site, like example.com. A full URL works; only its host is used. |