Test what a domain actually serves AI crawlers
crawler_block_testWhat the domain's CDN really does, as opposed to what its robots.txt says. Sends each AI crawler's real user-agent at the homepage and compares the response against a browser: a site can allow GPTBot in robots.txt and still have its CDN answer it a 403, and robots.txt cannot tell you that. Verdicts per crawler: ok, blocked, throttled (429, which is "too fast" and not "not you"), degraded (a 200 with a fraction of the bytes, the shape of a JS-gated page), or error. When measurable is false the site refused the control request too and there is no verdict to give. Live probe, ~6 requests. Free. Use preflight for the declared policy; use this for the observed one.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes |