Skip to main content
Glama
Sign In

AI Crawler Checker — Robots.txt Policy for Six Bots

by io.github.kriton-dev
OverviewDiscussionsAdmin

Server Details

Check declared robots.txt permissions for six AI crawlers at URL paths

The last two health checks of this connector failed. If this server requires authentication, missing or expired test credentials may be the cause. A test profile lets Glama authenticate for health checks and discover tools; it is separate from your personal connections.

If you are the author, claim ownership, then add or update a test profile under Admin → Test Profile.

Status
Unhealthy
OAuth
Works in Glama
Last Tested
Transport
Streamable HTTP
URL
ChatGPT
–
Claude
–

Related MCP Connectors

Related MCP Servers

  • QueryRev AI Readiness by QueryRevAI

    QueryRev AI Readinessofficial

    QueryRevAI
    A
    license
    A
    quality
    C
    maintenance
    Checks a site for AI crawlers: which of 11 AI crawlers its robots.txt lets in, whether its llms.txt follows llmstxt.org, and what its home page says without scripts. Findings with evidence; no scores, no model calls, no telemetry.
    5
    79 npm
    MIT

  • robotstxt-ai-mcp

    sharozdawa
    A
    license
    A
    quality
    C
    maintenance
    Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.
    5
    1
    MIT

  • ai-visibility-mcp

    bestaiinsider
    A
    license
    A
    quality
    D
    maintenance
    Audits AI-bot visibility: robots.txt per-bot for 22 AI user-agents (GPTBot/ClaudeBot/PerplexityBot/etc), Cloudflare flags, JSON-LD, sitemap, llms.txt, SPA shell, plus cross-model brand mentions via Perplexity + OpenRouter. 0-100 score. SSRF-guarded, spend-capped.
    4
    1
    MIT

  • OpenGEO

    rohanjha2019
    A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to audit and improve a site's visibility in AI search by checking robots.txt against 17 major AI crawlers, detecting llms.txt, and returning a 0-100 visibility score with per-crawler verdicts; it also lists known AI crawlers and their purposes.
    2
    MIT
Try in Browser
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources