Skip to main content
Glama

Atalaya AI crawler check

Check AI crawler access

check_ai_crawler_access
Read-onlyIdempotent

Checks whether AI crawlers can reach and read a public website or web app: which AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more) robots.txt allows and the exact rule for each, how much text the homepage HTML carries for crawlers that do not run JavaScript (flags client-rendered app shells, common in apps built with Lovable, Bolt, Base44 or Replit), title/meta description/H1, llms.txt, sitemap, and the platform or host setting to change (Cloudflare, Vercel, Netlify). Use it when someone asks why their site or app does not show up in ChatGPT, Claude, Perplexity or other AI answers.

Input Schema

NameRequiredDescriptionDefault
langNoLanguage of the verdict text (default en)
domainYesPublic domain or URL, for example example.com

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and open-world, so the safety profile is covered. The description adds real context beyond that: it only works on public sites, it inspects robots.txt, non-JS HTML content, metadata, llms.txt and sitemap, and it identifies which platform knob (Cloudflare/Vercel/Netlify) to turn. It stops short of stating prerequisites (site must be live and publicly reachable), latency, or whether a blocked crawler check is best-effort.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Effectively two sentences: the capability enumeration, then the trigger condition. It is front-loaded and purposeful, but the long parenthetical catalogues of crawler names, framework names and hosting platforms read as keyword padding rather than essential instruction.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, and the description compensates well by itemizing each category of finding the verdict returns. For a two-parameter read-only diagnostic, an agent has enough to call it and interpret results, though it lacks any note on failure modes (unreachable domain, blocked bot, rate limits).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 100% schema description coverage, the schema already explains both 'domain' (with an example) and 'lang' (enum en/es with default). The description adds nothing about either parameter — no note on URL vs bare domain handling, no mention of the language option. Baseline 3 applies since the schema carries the load.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Checks whether AI crawlers can reach and read a public website or web app') and enumerates exactly what the verdict covers (robots.txt rules per crawler, JS-free HTML text volume, title/meta/H1, llms.txt, sitemap, host-level setting). An agent immediately knows what it gets back. It does not differentiate itself from the only sibling, list_ai_crawlers, which is an obvious ambiguity risk.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use it when someone asks why their site or app does not show up in ChatGPT, Claude, Perplexity or other AI answers' gives a concrete triggering situation. There are no explicit exclusions and the sibling list_ai_crawlers is never named as the alternative for the simpler 'what crawlers exist' question, so the routing guidance is incomplete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources