robots_txt
Fetch and parse a site's robots.txt to reveal user-agent rules, allowed/disallowed paths, sitemap directives, crawl-delay, and issues, so you can verify what search engines can crawl.
Instructions
Fetch and parse a site's robots.txt. Returns user-agent rules, allowed/disallowed paths, sitemap directives, crawl-delay, and issues. Use this to check what search engines can/cannot crawl.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Domain or URL (e.g. https://example.com). Will check /robots.txt at the root. |