Skip to main content
Glama

AI Crawler Permission Check

Server Details

Check a site's robots.txt rules for AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more).

Ownership verified
Status
Healthy
Uptime
100.0% over the last 24 hours
Last Tested
Transport
Streamable HTTP · MCP 2025-11-25
URL
ChatGPT
–
Claude
–

TDQS

A3.6/5.0

Scored across 2 tools

Disambiguation4/5

The two tools overlap because both inspect robots.txt permissions for crawler user-agents, but their boundaries are fairly clear: one checks all known AI crawlers, while the other checks an arbitrary supplied user-agent. An agent could still momentarily hesitate over which to use for a specific AI crawler, but the descriptions provide enough distinction.

Naming Consistency5/5

Both names use the same snake_case verb_noun convention with a clear 'check_' prefix. The naming is predictable and immediately readable.

Tool Count4/5

Two tools is slightly below the typical 3-15 range, but the server has a narrow, well-defined purpose. Both tools earn their place by covering aggregate AI-crawler checks and arbitrary user-agent checks.

Completeness4/5

The server covers the core permission-checking lifecycle for robots.txt, including all known AI crawlers and arbitrary user-agent tokens. Minor gaps such as listing known crawlers separately or checking multiple domains at once exist, but the essential capability is present.

Available Tools

2 tools
check_ai_crawlersWhich AI crawlers may fetch this siteA
Read-onlyIdempotent
Inspect

For a domain (and optional path), whether each known AI crawler/agent user-agent is allowed by robots.txt, with the deciding rule.

Inputs: site (required), path. Returns: agents

check_user_agentIs this user-agent allowed on a pathB
Read-onlyIdempotent
Inspect

Whether a given crawler user-agent token may fetch a path on a domain according to robots.txt.

Inputs: site (required), user_agent (required), path. Returns: allowed

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      D
      maintenance
      Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.
      5
      1
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Checks a site for AI crawlers: which of 11 AI crawlers its robots.txt lets in, whether its llms.txt follows llmstxt.org, and what its home page says without scripts. Findings with evidence; no scores, no model calls, no telemetry.
      5
      49 npm
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      Checks a website's robots.txt and Cloudflare settings to identify AI crawler blocking. Also generates llms.txt content to improve visibility to AI answer engines.
      3
      MIT
    Try in Browser

    Glama MCP Gateway

    Add one secure layer between your agents and this server.

    Resources