stillindexed_check_robots
Validate a robots.txt with RFC 9309 and Google's parsing rules. Give url to fetch the live file, or content to test a pasted one, plus proposed to diff a second version. For each path or URL (up to 100, or use_sitemap to read them from the file's Sitemap line) it reports allowed or blocked for Googlebot, Googlebot-Image, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot, with the deciding rule and its line number, plus lints (unsupported fields, 500 KiB limit, BOM, HTTP status semantics). With proposed, changes lists only the verdicts that flip. Same check as https://stillindexed.com/tools/robots-txt-tester.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | Absolute http(s) URL, or a hostname (https is assumed). | |
| paths | No | Paths or full URLs to evaluate. Defaults to ["/"]. | |
| content | No | A robots.txt to test instead of fetching one. With url also set, url only names the host the paths belong to. | |
| proposed | No | A second robots.txt. The response lists the path x crawler verdicts that differ from the first. | |
| use_sitemap | No | Also test URLs from the first Sitemap: line (one level of sitemap index followed). Each sitemap fetch costs one unit of the hourly budget. |