robots.txt from a log
robots_from_logA robots.txt naming only the crawlers in your log, each with its request count and cost of blocking, plus a warning for any that do not documentably obey it — there the file is a request, not enforcement. Example: log='412 GPTBot/1.2', stance='block-ai-training' blocks GPTBot only.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| log | Yes | The log text. | |
| stance | No | Default block-ai-training. |