Write robots.txt rules for AI crawlers
ai_crawler_policyWrite robots.txt rules for AI crawlers based on a policy, and merge them into the site's current robots.txt. Policies: allow_all, search_only (allow AI search and user-requested fetches, block model training), block_training (same as search_only), block_all. Returns the full new robots.txt and what changes.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Any page of the site. | |
| policy | Yes | Which AI crawlers to allow. |