check_bots
Probes a URL with AI crawler user agents, using the operators' current documented strings (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, GoogleOther, Google-Agent, CCBot, Applebot, meta-externalagent, meta-externalfetcher, Amzn-SearchBot, MistralAI-User, DuckAssistBot) in parallel, plus a regular Chrome browser request as a baseline, and returns per-bot reachability (status, key WAF/CDN headers, verdict). Reveals WAF/CDN-layer blocks that robots.txt audits miss. Verdicts: reachable, blocked-403, blocked-4xx, challenged-by-cdn (including 200 challenge pages), rate-limited-429, redirected-to-block-page, inconclusive-not-bot-specific (the browser baseline was refused too, so the block is not aimed at the bot), not-found, server-error-5xx, timeout, fetch-error. summary.blocked counts only bot-specific blocks; timeouts and errors are counted separately. Requests come from Cloudflare's network, not the bot operators' IPs (see probe.note). Google-Extended and Applebot-Extended are robots.txt tokens, not user agents, and are not probed.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute http(s) URL to probe | |
| timeoutMs | No | Per-fetch timeout in ms (default 5000) | |
| userAgents | No | Optional UA strings to test. Defaults to the bots listed above (robots.txt tokens such as Google-Extended and Applebot-Extended are not probed). |