Skip to main content
Glama

Check AI crawler access

check_ai_crawler_access
Read-only

Check whether AI crawlers (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot) may access a public URL according to its robots.txt, and score the page's machine readability (status, title, meta description, canonical, JSON-LD, server-rendered text). Returns a 0-100 score and a list of findings.

Input Schema

NameRequiredDescriptionDefault
urlYesPublic page URL or domain, e.g. "https://example.com/pricing".

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint and openWorldHint, and the description adds genuinely useful behavioral context: which access-control source is consulted (robots.txt), which signals drive the score, and that the score is 0-100 with a findings list. It stops short of noting rate limits, caching, or failure behavior on unreachable URLs.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One dense sentence that front-loads the core purpose and then the output shape; the parenthetical crawler/signal lists are long but each item is informative. No filler sentences, though the readability clause slightly crowds the primary crawler-access purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description correctly carries the return-value burden by naming the 0-100 score and findings list. Combined with a fully documented single parameter, an agent has enough to call it correctly; only failure/timeout behavior is unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single 'url' parameter is fully documented in the schema (100% coverage), including an example, so the baseline of 3 applies. The description adds only the constraint that the URL must be public, which is marginal beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource (check crawler access for a URL per robots.txt) and enumerates the exact crawlers and readability signals examined. It is clearly distinguishable from siblings like read_url (fetches content) and generate_ai_robots_txt (produces a file).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the name and content but never stated: no 'use this when...' guidance, no conditions for choosing it over read_url or ai_visibility_score. It also gives no prerequisites (e.g., that the page must be publicly reachable).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources