Skip to main content
Glama

Server Details

Identify AI crawlers by user-agent or IP, verify them against operator ranges, build robots.txt.

If you are the author of this connector, you can claim ownership with GitHub, an HTTP challenge, or a DNS record. Claimed connector authors can inspect health checks, view analytics, and manage their listing.
Status
Not tested
Last Tested
Never
Transport
Streamable HTTP
URL

Tool Definition Quality

Score is being calculated. Check back soon.

Frequently Asked Questions

Discussions

No comments yet. Be the first to start the discussion!

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.
    5
    1
    MIT
  • A
    license
    A
    quality
    C
    maintenance
    Checks a website's robots.txt and Cloudflare settings to identify AI crawler blocking. Also generates llms.txt content to improve visibility to AI answer engines.
    3
    43
    MIT
  • A
    license
    A
    quality
    D
    maintenance
    Audits AI-bot visibility: robots.txt per-bot for 22 AI user-agents (GPTBot/ClaudeBot/PerplexityBot/etc), Cloudflare flags, JSON-LD, sitemap, llms.txt, SPA shell, plus cross-model brand mentions via Perplexity + OpenRouter. 0-100 score. SSRF-guarded, spend-capped.
    4
    1
    MIT
  • A
    license
    A
    quality
    A
    maintenance
    Enables AI agents to check whether a public website is crawlable, understandable, and ready for AI search workflows through local-only audits of robots.txt, sitemaps, metadata, and llms.txt.
    3
    51
    1
    MIT

View all MCP Servers

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources