robots
May you fetch this URL? Reads the site's robots.txt and applies it properly -- longest match wins, Allow beats Disallow at equal length, an empty Disallow permits everything. Same address guard as ping.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL you intend to fetch. | |
| user_agent | No | The agent name to evaluate as. Default coolant.run. |