generate_robots
Generate a robots.txt to control web crawlers by defining allowed and disallowed paths and listing sitemap URLs.
Instructions
Serialize a robots.txt for crawl control. Pass allow/disallow (string or array) and one or more sitemap URLs. Index control belongs in the head robots directive, not here.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| options | Yes |