Draft an AI crawler policy
draft_ai_crawler_policyDrafts robots.txt groups for the AI crawlers Agent Ready tracks, and a Content-Signal line, from what the site owner wants: crawlers that index for AI search, crawlers that fetch a page because a user asked, and crawlers that collect training data, each allowed or blocked, with per-crawler overrides. It fetches nothing. Use it after check_site_files or can_ai_fetch shows crawlers blocked or allowed against the owner's wishes. result has robotsTxt (the lines to merge), contentSignal, decisions (each crawler with its role and choice) and unknownOverrides (tokens not recognised). Returns JSON: summary (one sentence), reportUrl (a shareable report page for the person, kept 30 days; null for tools that keep none) and result (the details below). Errors come back as one plain sentence.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| search | No | Whether AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) may read the site: allow or block. Defaults to allow. | |
| training | No | Whether training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) may read it: allow or block. Defaults to allow. | |
| overrides | No | Per-crawler choices that win over the role choices, by robots.txt token, such as {"GPTBot": "block"}. Defaults to none. | |
| userFetch | No | Whether crawlers fetching for a user (ChatGPT-User, Claude-User) may read it: allow or block. Defaults to allow. |