Check AI crawler access
check_ai_crawler_accessChecks whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot may read one page, from robots.txt, the robots meta tags (including ones naming the crawler), the X-Robots-Tag header and the HTTP status. Use it when the person asks whether ChatGPT, Claude, Perplexity or Google can see a page, or whether blocked crawlers explain why the site is missing from AI answers. result.bots lists each crawler with verdict: readable, readable_not_for_index (it may fetch the page but is asked not to index it), blocked_by_robots_txt or page_unavailable, plus robotsTxtAllows and the pageDirectives found. Firewalls that block crawlers by user agent or IP are not visible from here. These are deterministic checks of public pages fetched by Agent Ready Website Audit, with no AI model and no account. They do not cover rankings, keyword volumes, backlinks or how AI answers mention the brand. Returns JSON: summary (one sentence), reportUrl (a shareable report page for the person, kept 30 days; null for tools that keep none) and result (the details below). Errors come back as one plain sentence. Recorded: the checked domain and path (never the query string), the tool, which checks failed and the app name; never an IP address or page text.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "check_ai_crawler_access"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| bot | No | One robots.txt product token, such as GPTBot, ClaudeBot or PerplexityBot. Defaults to every AI crawler Agent Ready tracks. | |
| url | Yes | The page to ask about: an absolute http or https URL, such as https://example.com/pricing. |