dossier_ai_crawlers
Core dossier check: Report a domain's AI-crawler policy — whether robots.txt allows, blocks, or is silent on the six major AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, meta-externalagent). Use to answer "does this site let AI models train on / retrieve its content?" in a posture audit. Fetches https:///robots.txt, follows redirects (off-site redirect = no policy), 10s timeout; parses robots groups. Returns a CheckResult; on success { hasRobots, agents: { : 'allowed' | 'blocked' | 'unspecified' } }. A missing robots.txt is data, not an error (every agent unspecified).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Public FQDN, e.g. example.com. Must be resolvable on the public internet; IPs, ports, paths, and protocol prefixes are rejected. |