Analyze robots.txt
analyze_robots_txtRead-only
Fetch and parse the robots.txt of a site: rules per user agent, sitemap directives, syntax errors, and whether the file exists at all.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL or domain to fetch robots.txt from |