Skip to main content
Glama

Agent Ready Website Audit

Check AI crawler access

check_ai_crawler_access
Read-only

Checks whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot may read one page, from robots.txt, the robots meta tags (including ones naming the crawler), the X-Robots-Tag header and the HTTP status. Use it when the person asks whether ChatGPT, Claude, Perplexity or Google can see a page, or whether blocked crawlers explain why the site is missing from AI answers. result.bots lists each crawler with verdict: readable, readable_not_for_index (it may fetch the page but is asked not to index it), blocked_by_robots_txt or page_unavailable, plus robotsTxtAllows and the pageDirectives found. Firewalls that block crawlers by user agent or IP are not visible from here. These are deterministic checks of public pages fetched by Agent Ready Website Audit, with no AI model and no account. They do not cover rankings, keyword volumes, backlinks or how AI answers mention the brand. Returns JSON: summary (one sentence), reportUrl (a shareable report page for the person, kept 30 days; null for tools that keep none) and result (the details below). Errors come back as one plain sentence. Recorded: the checked domain and path (never the query string), the tool, which checks failed and the app name; never an IP address or page text.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "check_ai_crawler_access"
  }
}

Input Schema

NameRequiredDescriptionDefault
botNoOne robots.txt product token, such as GPTBot, ClaudeBot or PerplexityBot. Defaults to every AI crawler Agent Ready tracks.
urlYesThe page to ask about: an absolute http or https URL, such as https://example.com/pricing.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A4.1/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnly/openWorld annotations, it discloses the exact evidence sources checked, the full verdict vocabulary with the meaning of the ambiguous readable_not_for_index state, the limitation that user-agent/IP firewall blocks are invisible, that checks are deterministic with no model and no account, and that reportUrl is retained 30 days. It also discloses its recording/privacy behavior (domain and path only, never query string, IP or page text), which an agent would otherwise have no way to know.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and trigger are front-loaded in the opening two sentences, and the length is justified by the absence of an output schema. The middle section is dense but readable; the only minor inefficiency is that return-shape detail and scope-boundary detail are interleaved rather than grouped.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description correctly takes on the return-value burden, spelling out summary, reportUrl (with 30-day retention and null case) and the result.bots verdicts plus robotsTxtAllows and pageDirectives. Combined with annotations covering the safety profile, an agent has everything needed to select and call this tool and interpret the response.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both parameters (url as an absolute http/https page URL, bot as a single robots.txt product token defaulting to all tracked AI crawlers) are already fully documented in the schema. The description reinforces the 'one page' semantics and that result.bots enumerates every crawler, but adds no format, validation or defaulting detail beyond the structured fields.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first sentence gives a specific verb (checks), a precise resource (whether AI crawlers may read one page) and enumerates the exact evidence sources: robots.txt, robots meta tags, X-Robots-Tag header and HTTP status. The 'one page' scope implicitly separates it from site-level siblings like check_robots_sitemap_llms_txt, but no sibling is ever named, so differentiation is inferred rather than stated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit trigger condition ('Use it when the person asks whether ChatGPT, Claude, Perplexity or Google can see a page, or whether blocked crawlers explain why the site is missing from AI answers') and clear exclusions: firewalls that block by user agent or IP are not visible, and rankings, keyword volumes, backlinks and AI-answer mentions are out of scope. It stops short of naming which sibling tool should be used instead for those out-of-scope questions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources