ai-crawler-readiness
Server Details
Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
Glama couldn't complete the latest health check. If this server requires authentication, missing or expired test credentials may be the cause. A test profile lets Glama authenticate for health checks and discover tools; it is separate from your personal connections.
If you are the author, claim ownership, then add or update a test profile under Admin → Test Profile.
- Status
- Unhealthy
- Uptime
- 4.5% over 21 days
- Last Tested
- Transport
- Streamable HTTP
- URL
TDQS
Scored across 5 tools
Most tools are clearly distinct: domain_readiness covers archived per-domain status, measure_domain covers fresh measurement, edge_blocking covers vendor attribution, recent_changes covers temporal flips, and web_openness covers aggregate statistics. The only close pair is domain_readiness and measure_domain, but the explicit 'right now' vs 'archive' distinction prevents serious confusion.
All names are readable snake_case, but they do not follow a consistent pattern: domain_readiness and web_openness are noun compounds, edge_blocking is noun+gerund, measure_domain is verb+noun, and recent_changes is adjective+noun. The mixed conventions are still predictable enough to infer intent, but they are not uniform.
Five tools is well-scoped for an analytics and measurement server. Each tool covers a distinct question a user would reasonably ask: one domain now, one domain from archived data, which edge vendor is responsible, what changed recently, and overall web openness. There is no obvious redundancy or bloat.
The tool surface covers the core workflows: per-domain readiness, live remeasurement, aggregate openness, vendor-level edge blocking, and change detection. A minor gap is the lack of a direct way to enumerate all domains that currently block a specific crawler, but this does not undermine the main use cases.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
- Added
recent_changes
4 tool updates
- First observed
domain_readiness - First observed
edge_blocking - First observed
measure_domain - First observed
web_openness
Related MCP Connectors
Can GPTBot, ClaudeBot and PerplexityBot reach a site? Real fetch per crawler, firewall/CDN blocks.
Bing Webmaster, IndexNow and Google Search Console for AI: rankings, indexing gaps, AI crawlers.
1Free AI-readiness audit of any URL: AI crawler rules, JS-free text, JSON-LD, llms.txt. Tool catalog.
Scan and fix your site's AI discoverability: crawler access, llms.txt, JSON-LD. Free.
Related MCP Servers
- AlicenseAqualityDmaintenanceAudits AI-bot visibility: robots.txt per-bot for 22 AI user-agents (GPTBot/ClaudeBot/PerplexityBot/etc), Cloudflare flags, JSON-LD, sitemap, llms.txt, SPA shell, plus cross-model brand mentions via Perplexity + OpenRouter. 0-100 score. SSRF-guarded, spend-capped.41MIT
- AlicenseNot gradedqualityCmaintenanceLets an AI assistant check any URL to see how much of a page's main content is present in the raw HTML versus only after JavaScript runs, showing which text blocks, meta fields, canonical tags, JSON-LD and links AI crawlers like GPTBot and ClaudeBot cannot read. It also reports what robots.txt allows for 22 AI bot tokens and how the site responds to each bot's user-agent.AGPL 3.0
- AlicenseAqualityCmaintenanceAnalyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.51MIT
- AlicenseAqualityCmaintenanceChecks a website's robots.txt and Cloudflare settings to identify AI crawler blocking. Also generates llms.txt content to improve visibility to AI answer engines.343 npmMIT
Glama MCP Gateway
Add one secure layer between your agents and this server.