Skip to main content
Glama

Check which AI crawlers can reach a site

check_ai_crawlers
Read-onlyIdempotent

Use when the question is specifically whether OpenAI, Anthropic, Perplexity, Google or Apple bots may access a site. Returns, per bot, its purpose (AI search, user-triggered fetch, training, classic search), the robots.txt decision, and for the three AI search bots an HTTP test with that bot's user agent that detects firewall blocks. Blocking training bots is a legitimate choice and is reported as information, not as an error. Fetches up to 5 URLs.

Input Schema

NameRequiredDescriptionDefault
urlYesPublic http(s) URL or bare domain, for example https://example.com

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, idempotent, non-destructive and open-world, yet the description adds substantial context beyond them: the per-bot return shape (purpose category, robots.txt decision, HTTP test with that bot's user agent, firewall-block detection) and the important semantic that blocking training bots is reported as information, not an error. It also discloses the 'up to 5 URLs' fetch limit, which is real behavioral context an agent cannot get from the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four sentences, front-loaded with the trigger condition, then return content, then the interpretive caveat, then the fetch limit — a logical and largely waste-free order. It is dense (purpose taxonomy and HTTP testing squeezed into one sentence) but nothing is redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description must carry the return-value burden, and it does: per-bot purpose, robots.txt decision, and HTTP firewall test results are all described. With a single required parameter and annotations covering the safety profile, an agent has everything needed to call it and interpret the result correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the url parameter is already documented with a format example, so the schema does most of the work. The description still adds meaningful semantics with 'Fetches up to 5 URLs', implying the single url argument can expand into multiple fetched targets (e.g. redirects/sitemap), which is not visible in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb+resource (check whether named AI crawler bots may access a site) and enumerates the exact bots covered, so the agent knows precisely what is inspected. It implicitly scopes itself away from the broader audit_ai_readiness via 'when the question is specifically whether...', but it never names a sibling tool, so sibling differentiation is only inferential.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Use when the question is specifically whether OpenAI, Anthropic, Perplexity, Google or Apple bots may access a site' gives a clear triggering condition and narrows the use case well. It does not state explicit exclusions or point to check_llms_txt / audit_ai_readiness as the alternative for adjacent questions, so routing between siblings is left partly to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources