Analyze robots.txt
robots_txt_analysisFetch and summarize an authorized origin's RFC 9309 robots.txt file to show crawl rules without crawling disclosed paths.
Instructions
Fetch and summarize the RFC 9309 robots.txt file for an authorized origin without crawling disclosed paths.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| groups | Yes | ||
| present | Yes |