lint_robots_txt
Lint a robots.txt for AI-crawler-specific mistakes. Pass url to fetch it live, or robots_txt to lint pasted text directly.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | A site whose /robots.txt should be fetched and linted | |
| robots_txt | No | Raw robots.txt text to lint directly, instead of fetching url |