Skip to main content
Glama

upGrowth Growth Toolkit

AI crawler access check

ai_crawler_access_check
Read-onlyIdempotent

Reads a website's public robots.txt and llms.txt and reports whether AI crawlers from OpenAI, Anthropic, Perplexity and Google (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and others) are allowed, restricted or blocked.

Input Schema

NameRequiredDescriptionDefault
domainYesPublic domain to check, for example example.com.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, openWorld and non-destructive, so the safety profile is covered. The description adds the useful detail that results are categorised as allowed/restricted/blocked and names the specific bots checked, but says nothing about failure modes such as a missing robots.txt, timeouts, or whether results are cached.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One sentence, front-loaded with the action and the data sources, with the crawler list kept compact in parentheses. Nothing is wasted, though the parenthetical enumeration leans slightly long for a single-parameter tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description helpfully supplies the result vocabulary (allowed/restricted/blocked) and the bot identities, which is what an agent needs to interpret output. It stops short of covering edge cases like absent files or non-crawlable domains, leaving a small gap for an openWorld tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single parameter has 100% schema description coverage ('Public domain to check, for example example.com'), so the schema carries the burden and baseline 3 applies. The description adds only marginal context by framing the input as a website whose robots.txt/llms.txt is fetched.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (reads) and resources (robots.txt, llms.txt) plus what it produces (allowed/restricted/blocked status per crawler). This clearly separates it from the sibling robots_txt_generator, which writes rather than inspects.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: the agent can infer you call it to audit crawler access for a domain. There is no explicit when-to-use/when-not statement, no mention of prerequisites (e.g., that the domain must be publicly reachable), and no named alternative.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.