analyze_robots_txt
Parse a webpage's robots.txt file to inspect crawl rules, sitemap references, and directives. Use it to verify SEO accessibility and search engine guidance.
Instructions
Analyze the robots.txt file of a webpage
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |