Write a cited robots.txt
suggest_robotsWrite a robots.txt from this dataset for the kinds you allow or block. Each group cites the vendor page. effect request_only means the vendor does not say the crawler obeys that line. control_token is not training. The file has no User-agent: * group. This tool does not fetch any website. Do not send url, host, target, website, robots_url, or fetch.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | Path the Allow or Disallow names. Defaults to /. | |
| allow | No | Kinds to Allow, such as search. | |
| block | No | Kinds to Disallow, such as training. | |
| policy | No | Same choice in words, such as "allow search, block training". |