Skip to main content
Glama

Atalaya AI crawler check

Server Details

Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.

If you are the author of this connector, you can claim ownership by verifying the domain or GitHub account it belongs to. Claimed connector authors can inspect health checks, view analytics, and manage their listing.
Status
Healthy
Last Tested
Transport
Streamable HTTP · MCP 2025-11-25
URL
ChatGPT
–
Claude
–

TDQS

A4/5.0

Scored across 2 tools

Disambiguation5/5

The two tools have clearly distinct purposes: one performs a diagnostic check on a site, the other returns a static reference list of crawler user-agent tokens. An agent can easily tell which to use based on whether it needs site analysis or crawler identifiers.

Naming Consistency5/5

Both names use snake_case with a verb-first pattern (check_ai_crawler_access, list_ai_crawlers) and share consistent terminology around AI crawlers. The convention is predictable and readable.

Tool Count3/5

The server's purpose is narrow, so two tools are defensible, but two is on the thin side and the surface feels minimal for a diagnostic server. Each tool earns its place, though the set could benefit from one or two more operations.

Completeness4/5

The diagnostic tool covers a wide range of relevant checks (robots.txt rules, rendered text, metadata, llms.txt, sitemap, platform settings) and the reference tool complements it well. Minor gaps exist, such as no tool to validate or generate a robots.txt or to run checks across multiple pages, but the core domain is well covered.

Available Tools

2 tools
check_ai_crawler_accessCheck AI crawler accessA
Read-onlyIdempotent
Inspect

Checks whether AI crawlers can reach and read a public website or web app: which AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more) robots.txt allows and the exact rule for each, how much text the homepage HTML carries for crawlers that do not run JavaScript (flags client-rendered app shells, common in apps built with Lovable, Bolt, Base44 or Replit), title/meta description/H1, llms.txt, sitemap, and the platform or host setting to change (Cloudflare, Vercel, Netlify). Use it when someone asks why their site or app does not show up in ChatGPT, Claude, Perplexity or other AI answers.

NameRequiredDescriptionDefault
langNoLanguage of the verdict text (default en)
domainYesPublic domain or URL, for example example.com

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and open-world, so the safety profile is covered. The description adds real context beyond that: it only works on public sites, it inspects robots.txt, non-JS HTML content, metadata, llms.txt and sitemap, and it identifies which platform knob (Cloudflare/Vercel/Netlify) to turn. It stops short of stating prerequisites (site must be live and publicly reachable), latency, or whether a blocked crawler check is best-effort.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Effectively two sentences: the capability enumeration, then the trigger condition. It is front-loaded and purposeful, but the long parenthetical catalogues of crawler names, framework names and hosting platforms read as keyword padding rather than essential instruction.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, and the description compensates well by itemizing each category of finding the verdict returns. For a two-parameter read-only diagnostic, an agent has enough to call it and interpret results, though it lacks any note on failure modes (unreachable domain, blocked bot, rate limits).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 100% schema description coverage, the schema already explains both 'domain' (with an example) and 'lang' (enum en/es with default). The description adds nothing about either parameter — no note on URL vs bare domain handling, no mention of the language option. Baseline 3 applies since the schema carries the load.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Checks whether AI crawlers can reach and read a public website or web app') and enumerates exactly what the verdict covers (robots.txt rules per crawler, JS-free HTML text volume, title/meta/H1, llms.txt, sitemap, host-level setting). An agent immediately knows what it gets back. It does not differentiate itself from the only sibling, list_ai_crawlers, which is an obvious ambiguity risk.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use it when someone asks why their site or app does not show up in ChatGPT, Claude, Perplexity or other AI answers' gives a concrete triggering situation. There are no explicit exclusions and the sibling list_ai_crawlers is never named as the alternative for the simpler 'what crawlers exist' question, so the routing guidance is incomplete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_ai_crawlersList known AI crawlersA
Read-onlyIdempotent
Inspect

Lists the AI crawler user-agent tokens Atalaya checks, by vendor and purpose (training, search index, user-triggered fetch), so you can write a robots.txt that keeps AI search while opting out of training.

NameRequiredDescriptionDefault

No parameters

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and openWorldHint=false, so safety and determinism are covered structurally. The description adds real content-level context by naming the classification axes (vendor, purpose) the response uses, but says nothing about response shape (flat list vs grouped) or ordering — minor for a zero-parameter lookup.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence in which every clause earns its place: the resource, the two classification axes with their enumerated purposes, and the concrete payoff. Nothing is repeated from the title or annotations.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter, read-only lookup with no output schema, the description supplies both the result content (tokens by vendor and purpose) and the reason to call it. Only the exact return structure is unspecified, which is a small gap given there is no output schema to lean on.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes no parameters, so per the baseline the schema carries no semantics to supplement. The description correctly spends no words on argument handling and instead explains what the fixed result set contains.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb+resource: "Lists the AI crawler user-agent tokens Atalaya checks," plus the grouping dimensions (vendor, purpose: training, search index, user-triggered fetch). It does not name the sibling check_ai_crawler_access, so the boundary between listing tokens and checking a specific crawler's access is left implicit rather than stated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit downstream use case — writing a robots.txt that admits AI search but opts out of training — which is enough context to know when to call it. It stops short of an explicit when-not or a pointer to check_ai_crawler_access for per-crawler verification.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updates
    • First observedcheck_ai_crawler_access
    • First observedlist_ai_crawlers

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.
    5
    1
    MIT
  • A
    license
    A
    quality
    D
    maintenance
    Audits AI-bot visibility: robots.txt per-bot for 22 AI user-agents (GPTBot/ClaudeBot/PerplexityBot/etc), Cloudflare flags, JSON-LD, sitemap, llms.txt, SPA shell, plus cross-model brand mentions via Perplexity + OpenRouter. 0-100 score. SSRF-guarded, spend-capped.
    4
    1
    MIT
  • A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to audit and improve a site's visibility in AI search by checking robots.txt against 17 major AI crawlers, detecting llms.txt, and returning a 0-100 visibility score with per-crawler verdicts; it also lists known AI crawlers and their purposes.
    2
    MIT
  • A
    license
    A
    quality
    C
    maintenance
    Checks a site for AI crawlers: which of 11 AI crawlers its robots.txt lets in, whether its llms.txt follows llmstxt.org, and what its home page says without scripts. Findings with evidence; no scores, no model calls, no telemetry.
    5
    79 npm
    MIT
Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources