crawler_access_audit
Verifies AI crawler access to a URL by parsing robots.txt and performing live GETs, surfacing blocks and gating that impede citation.
Instructions
Verify that major AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, plus real-time fetch UAs) can fetch a URL. Parses robots.txt and does a live GET with each bot's User-Agent. Surfaces robots.txt blocks AND UA-based gating that breaks AI citation.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Page URL to test for AI crawler access. | |
| bots | No | Override the default bot list. Each entry is a User-Agent token (e.g. 'GPTBot', 'ClaudeBot'). | |
| fetch_with_ua | No | If true, do a live GET as each bot's User-Agent and report status. Disable to only parse robots.txt (no extra requests). |