Check AI crawler access in robots.txt
check_ai_crawler_accessChecks whether 14 AI crawlers may fetch a page. Use it when the user asks whether AI crawlers can read a website or page, for example "can AI crawlers read example.com?", "is GPTBot blocked in my robots.txt?" or "which AI bots are allowed on https://example.com/blog?". Pass the page address. Returns, for 14 AI crawlers, whether robots.txt allows the page and the rule that decided it, plus the sitemaps, the redirect chain, and the page's X-Robots-Tag header and robots meta tag. It requests the page as ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and a browser, and says if ChatGPT can read it in one verdict (readable, blocked by CDN or WAF, blocked by robots, needs JavaScript, redirect problem, needs sign-in) with the fix. That test is by user agent from Agent Tools servers, not from OpenAI's addresses. Do not use it for private or local addresses, for security scans, or to collect a site's content.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "check_ai_crawler_access"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Address of a public web page, for example https://example.com/pricing. A bare domain such as example.com means https. Only public http and https sites on the standard ports are checked. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The address checked, after a bare domain became https. | |
| notes | No | Plain-language caveats to pass on with the result. | |
| notice | Yes | As-of caveat: what the site sent to Agent Tools at that moment, not what every crawler sees. | |
| robots | No | ||
| source | Yes | Where the data came from: the website itself. | |
| status | No | HTTP status code of the final page. | |
| failure | No | Why the page was not reached or read, such as a timeout or a refused redirect. | |
| crawlers | No | ||
| finalUrl | No | Address of the final page after redirects. | |
| reachable | Yes | False when no page answered; failure then says why. | |
| redirects | No | Redirect chain from the address to the page. | |
| metaRobots | No | The page's robots meta tag. | |
| xRobotsTag | No | The page's X-Robots-Tag response header. | |
| blockedCrawlers | No | Tokens of the crawlers robots.txt blocks on this page. |