netintel_robots_txt
Fetch and parse a domain's robots.txt to return crawl rules, sitemap URLs, crawl delays, and check if a path is allowed for a given bot.
Instructions
Fetch and parse a domain's robots.txt. Returns all crawl rules, sitemap URLs, crawl delay settings, and checks if a specific path is allowed for a given bot.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | ||
| domain | Yes | ||
| user_agent | No |