AI Crawler Permission Check
Server Details
Check a site's robots.txt rules for AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more).
- Status
- Healthy
- Uptime
- 100.0% over the last 24 hours
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-11-25
- URL
- ChatGPT
- –
- Claude
- –
TDQS
Scored across 2 tools
The two tools overlap because both inspect robots.txt permissions for crawler user-agents, but their boundaries are fairly clear: one checks all known AI crawlers, while the other checks an arbitrary supplied user-agent. An agent could still momentarily hesitate over which to use for a specific AI crawler, but the descriptions provide enough distinction.
Both names use the same snake_case verb_noun convention with a clear 'check_' prefix. The naming is predictable and immediately readable.
Two tools is slightly below the typical 3-15 range, but the server has a narrow, well-defined purpose. Both tools earn their place by covering aggregate AI-crawler checks and arbitrary user-agent checks.
The server covers the core permission-checking lifecycle for robots.txt, including all known AI crawlers and arbitrary user-agent tokens. Minor gaps such as listing known crawlers separately or checking multiple domains at once exist, but the essential capability is present.
Available Tools
2 toolscheck_ai_crawlersWhich AI crawlers may fetch this siteARead-onlyIdempotentInspect
For a domain (and optional path), whether each known AI crawler/agent user-agent is allowed by robots.txt, with the deciding rule.
Inputs: site (required), path. Returns: agents
check_user_agentIs this user-agent allowed on a pathBRead-onlyIdempotentInspect
Whether a given crawler user-agent token may fetch a path on a domain according to robots.txt.
Inputs: site (required), user_agent (required), path. Returns: allowed
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Related MCP Connectors
See what GPTBot, ClaudeBot and other crawlers get from a site: robots.txt checks and AI scans.
Check which AI bots a website or a pasted robots.txt lets in, grouped by purpose.
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Checks if GPTBot, ClaudeBot, PerplexityBot, Googlebot and 9 more AI crawlers can reach a page.
Related MCP Servers
- AlicenseAqualityDmaintenanceAnalyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.51MIT

QueryRev AI Readinessofficial
AlicenseAqualityCmaintenanceChecks a site for AI crawlers: which of 11 AI crawlers its robots.txt lets in, whether its llms.txt follows llmstxt.org, and what its home page says without scripts. Findings with evidence; no scores, no model calls, no telemetry.549 npmMIT- AlicenseNot gradedqualityBmaintenanceEnables users to check whether GPTBot, ClaudeBot, PerplexityBot, Googlebot and other AI/search crawlers can reach a specified page, distinguishing robots.txt blocks from server or firewall refusals.MIT
- AlicenseAqualityDmaintenanceChecks a website's robots.txt and Cloudflare settings to identify AI crawler blocking. Also generates llms.txt content to improve visibility to AI answer engines.3MIT
Glama MCP Gateway
Add one secure layer between your agents and this server.