Atalaya AI crawler check
Server Details
Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
- Status
- Healthy
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-11-25
- URL
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 2 tools
The two tools have clearly distinct purposes: one performs a diagnostic check on a site, the other returns a static reference list of crawler user-agent tokens. An agent can easily tell which to use based on whether it needs site analysis or crawler identifiers.
Both names use snake_case with a verb-first pattern (check_ai_crawler_access, list_ai_crawlers) and share consistent terminology around AI crawlers. The convention is predictable and readable.
The server's purpose is narrow, so two tools are defensible, but two is on the thin side and the surface feels minimal for a diagnostic server. Each tool earns its place, though the set could benefit from one or two more operations.
The diagnostic tool covers a wide range of relevant checks (robots.txt rules, rendered text, metadata, llms.txt, sitemap, platform settings) and the reference tool complements it well. Minor gaps exist, such as no tool to validate or generate a robots.txt or to run checks across multiple pages, but the core domain is well covered.
Available Tools
2 toolscheck_ai_crawler_accessCheck AI crawler accessARead-onlyIdempotentInspect
Checks whether AI crawlers can reach and read a public website or web app: which AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more) robots.txt allows and the exact rule for each, how much text the homepage HTML carries for crawlers that do not run JavaScript (flags client-rendered app shells, common in apps built with Lovable, Bolt, Base44 or Replit), title/meta description/H1, llms.txt, sitemap, and the platform or host setting to change (Cloudflare, Vercel, Netlify). Use it when someone asks why their site or app does not show up in ChatGPT, Claude, Perplexity or other AI answers.
| Name | Required | Description | Default |
|---|---|---|---|
| lang | No | Language of the verdict text (default en) | |
| domain | Yes | Public domain or URL, for example example.com |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, non-destructive and open-world, so the safety profile is covered. The description adds real context beyond that: it only works on public sites, it inspects robots.txt, non-JS HTML content, metadata, llms.txt and sitemap, and it identifies which platform knob (Cloudflare/Vercel/Netlify) to turn. It stops short of stating prerequisites (site must be live and publicly reachable), latency, or whether a blocked crawler check is best-effort.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Effectively two sentences: the capability enumeration, then the trigger condition. It is front-loaded and purposeful, but the long parenthetical catalogues of crawler names, framework names and hosting platforms read as keyword padding rather than essential instruction.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, and the description compensates well by itemizing each category of finding the verdict returns. For a two-parameter read-only diagnostic, an agent has enough to call it and interpret results, though it lacks any note on failure modes (unreachable domain, blocked bot, rate limits).
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 100% schema description coverage, the schema already explains both 'domain' (with an example) and 'lang' (enum en/es with default). The description adds nothing about either parameter — no note on URL vs bare domain handling, no mention of the language option. Baseline 3 applies since the schema carries the load.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Checks whether AI crawlers can reach and read a public website or web app') and enumerates exactly what the verdict covers (robots.txt rules per crawler, JS-free HTML text volume, title/meta/H1, llms.txt, sitemap, host-level setting). An agent immediately knows what it gets back. It does not differentiate itself from the only sibling, list_ai_crawlers, which is an obvious ambiguity risk.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Use it when someone asks why their site or app does not show up in ChatGPT, Claude, Perplexity or other AI answers' gives a concrete triggering situation. There are no explicit exclusions and the sibling list_ai_crawlers is never named as the alternative for the simpler 'what crawlers exist' question, so the routing guidance is incomplete.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_ai_crawlersList known AI crawlersARead-onlyIdempotentInspect
Lists the AI crawler user-agent tokens Atalaya checks, by vendor and purpose (training, search index, user-triggered fetch), so you can write a robots.txt that keeps AI search while opting out of training.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and openWorldHint=false, so safety and determinism are covered structurally. The description adds real content-level context by naming the classification axes (vendor, purpose) the response uses, but says nothing about response shape (flat list vs grouped) or ordering — minor for a zero-parameter lookup.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence in which every clause earns its place: the resource, the two classification axes with their enumerated purposes, and the concrete payoff. Nothing is repeated from the title or annotations.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, read-only lookup with no output schema, the description supplies both the result content (tokens by vendor and purpose) and the reason to call it. Only the exact return structure is unspecified, which is a small gap given there is no output schema to lean on.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes no parameters, so per the baseline the schema carries no semantics to supplement. The description correctly spends no words on argument handling and instead explains what the fixed result set contains.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb+resource: "Lists the AI crawler user-agent tokens Atalaya checks," plus the grouping dimensions (vendor, purpose: training, search index, user-triggered fetch). It does not name the sibling check_ai_crawler_access, so the boundary between listing tokens and checking a specific crawler's access is left implicit rather than stated.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit downstream use case — writing a robots.txt that admits AI search but opts out of training — which is enough context to know when to call it. It stops short of an explicit when-not or a pointer to check_ai_crawler_access for per-crawler verification.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
- First observed
check_ai_crawler_access - First observed
list_ai_crawlers
Related MCP Connectors
Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
See what GPTBot, ClaudeBot and other crawlers get from a site: robots.txt checks and AI scans.
Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.
Check declared robots.txt permissions for six AI crawlers at URL paths
Related MCP Servers
- AlicenseAqualityCmaintenanceAnalyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.51MIT
- AlicenseAqualityDmaintenanceAudits AI-bot visibility: robots.txt per-bot for 22 AI user-agents (GPTBot/ClaudeBot/PerplexityBot/etc), Cloudflare flags, JSON-LD, sitemap, llms.txt, SPA shell, plus cross-model brand mentions via Perplexity + OpenRouter. 0-100 score. SSRF-guarded, spend-capped.41MIT
- AlicenseAqualityCmaintenanceEnables AI agents to audit and improve a site's visibility in AI search by checking robots.txt against 17 major AI crawlers, detecting llms.txt, and returning a 0-100 visibility score with per-crawler verdicts; it also lists known AI crawlers and their purposes.2MIT

QueryRev AI Readinessofficial
AlicenseAqualityCmaintenanceChecks a site for AI crawlers: which of 11 AI crawlers its robots.txt lets in, whether its llms.txt follows llmstxt.org, and what its home page says without scripts. Findings with evidence; no scores, no model calls, no telemetry.579 npmMIT
Glama MCP Gateway
Add one secure layer between your agents and this server.