Skip to main content
Glama

Server Details

Can GPTBot, ClaudeBot and PerplexityBot reach a site? Real fetch per crawler, firewall/CDN blocks.

If you are the author of this connector, you can claim ownership by verifying the domain or GitHub account it belongs to. Claimed connector authors can inspect health checks, view analytics, and manage their listing.
Status
Healthy
Last Tested
Transport
Streamable HTTP · MCP 2025-06-18
URL

TDQS

A4.5/5.0

Scored across 1 tool

Disambiguation5/5

There is only one tool, so there is no possibility of an agent misselecting between overlapping capabilities. Its purpose is stated unambiguously in the description.

Naming Consistency5/5

The single tool uses a clear snake_case verb_noun pattern (check_ai_crawler_access) that is self-descriptive and idiomatic for MCP tooling. There are no competing conventions to introduce inconsistency.

Tool Count4/5

One tool is thin by general standards, but the server's scope is a single focused check, and the tool is substantive rather than trivial, returning a score, per-crawler verdicts, and fixes. It fully earns its place with no bloat.

Completeness4/5

The tool covers the core domain well: per-crawler reachability testing, firewall/CDN detection, noindex and llms.txt inspection, plus actionable output. Minor gaps exist, such as no batch/site-list checking, no custom user-agent parameter, or no way to retrieve raw fetch traces.

Available Tools

1 tool
check_ai_crawler_accessCheck AI crawler accessA
Read-onlyIdempotent
Inspect

Checks whether AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others) can actually reach a website. Goes beyond robots.txt: fetches the site as each crawler and detects firewall/CDN blocks (Cloudflare, Akamai, Imperva), challenge pages, noindex and llms.txt. Returns a 0-100 AI Access Score, a per-crawler verdict and the top issues with fixes. Takes 15-60 seconds.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesWebsite domain or URL, for example example.com

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds substantial behavioral detail beyond the annotations: it fetches the site as each crawler, detects firewall/CDN blocks, challenge pages, noindex and llms.txt, and returns a 0-100 score, per-crawler verdicts and top fixes. Runtime is disclosed, and the read-only/idempotent/non-destructive profile is consistent with the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core purpose, followed by scope, detection capabilities, return values and runtime in four efficient sentences. Every sentence adds useful information without repetition or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter diagnostic tool with no output schema, the description explains what is checked, what is returned, and approximately how long it takes. No critical invocation or interpretation detail is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage and already defines the single url parameter with an example. The description adds no additional parameter syntax, format rules or edge-case guidance, so the schema is doing the full parameter documentation job.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: checks whether named AI crawlers can actually reach a website. It immediately distinguishes itself from a plain robots.txt check by describing real crawler fetching and firewall/CDN block detection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clearly frames the use case: verifying real AI crawler access beyond robots.txt, including a useful runtime expectation of 15-60 seconds. It does not explicitly state when not to use it or name alternatives, but no sibling tools exist and the context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 1 tool update
    • First observedcheck_ai_crawler_access

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.
    5
    1
    MIT
  • A
    license
    A
    quality
    D
    maintenance
    Audits AI-bot visibility: robots.txt per-bot for 22 AI user-agents (GPTBot/ClaudeBot/PerplexityBot/etc), Cloudflare flags, JSON-LD, sitemap, llms.txt, SPA shell, plus cross-model brand mentions via Perplexity + OpenRouter. 0-100 score. SSRF-guarded, spend-capped.
    4
    1
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Lets an AI assistant check any URL to see how much of a page's main content is present in the raw HTML versus only after JavaScript runs, showing which text blocks, meta fields, canonical tags, JSON-LD and links AI crawlers like GPTBot and ClaudeBot cannot read. It also reports what robots.txt allows for 22 AI bot tokens and how the site responds to each bot's user-agent.
    AGPL 3.0
  • A
    license
    A
    quality
    C
    maintenance
    Checks a website's robots.txt and Cloudflare settings to identify AI crawler blocking. Also generates llms.txt content to improve visibility to AI answer engines.
    3
    49 npm
    MIT
Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources