stillindexed_check_ai_crawlers
Read-onlyIdempotent
Check robots.txt for the 13 tracked AI crawlers (ChatGPT search, Claude, Perplexity, Gemini grounding, and the training crawlers) and report, per crawler, whether the given URL's path is allowed or blocked, naming the deciding group and rule. Also probes the edge once as each of the three AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) against a browser request. Costs up to 5 fetches. Same check as https://stillindexed.com/tools/ai-crawler-checker.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute http(s) URL, or a hostname (https is assumed). |