AI-crawler policy
dossier_ai_crawlersReport what a domain's robots.txt says to the major AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, meta-externalagent): allowed, blocked or unspecified for each. Use to answer whether a site lets AI models train on or retrieve its content. A missing robots.txt is data, not an error: every crawler is then unspecified. One fetch, 10 s timeout. Returns JSON with a status field: {status:"ok", data, fetchedAt} on success, {status:"not_applicable", reason} when the thing is genuinely absent, {status:"timeout", ms}, or {status:"error", message} when it could not be determined. Treat not_applicable as a finding and error as unknown.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Public domain name, e.g. example.com. IP addresses, ports, paths and protocol prefixes are rejected. |