check_ai_crawlers
Check AI crawler access on any public domain. Returns allow/block verdict for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and others via robots.txt and meta tags analysis.
Instructions
Live check of AI crawler access via robots.txt and meta tags — covers GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and others. Returns per-crawler allow/block verdict. Works on any public domain (no project setup required). Read-only. Returns JSON object.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Domain to scan, e.g. example.com | |
| industry | No | Industry context for benchmark comparison. Current values: news-media, arts-entertainment, law-government, finance-healthcare, saas-tech, ecommerce, other. The backend may add new values over time; pass any of the listed strings (or a future one) and the API will validate. |