check_robots
Analyze robots.txt to verify crawl permissions, delay, request rate, and sitemap declarations.
Instructions
Inspect robots.txt permissions, crawl delay, request rate, and sitemap declarations.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||