AI crawler robots.txt rules
ai_crawler_policyBuilds robots.txt lines that allow or block AI crawlers from BACKLNK's reviewed crawler catalog, with what each line does and warnings. Use a preset (block-training: Block AI training, stay in AI search; block-ai: Block AI crawlers, keep search engines; allow-all: Allow every crawler) or decide per purpose and per crawler token.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| preset | No | ||
| crawlers | No | Per-crawler overrides by token, e.g. { "GPTBot": "allow" }. | |
| purposes | No | Decision per purpose; purposes not named stay allowed when no preset is given. |