robots_txt_test
Test whether a URL is allowed for a specific crawler using Google's robots.txt matching rules. Returns the deciding rule and line, plus optional stored-file comparison.
Instructions
Is this URL allowed for this crawler? Returns the deciding rule and line, using Google's matching rules.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Full URL to test | |
| check_id | No | Test against a stored version instead of the live file | |
| user_agent | No | Googlebot, Googlebot-Image, Googlebot-News, Googlebot-Video, Storebot-Google, AdsBot-Google, Google-Extended, or any token | Googlebot |
| show_content | No |