check_robots
Determines if a URL is crawlable by checking robots.txt, and returns the applicable crawl-delay, request rate, and sitemaps. Reports sites without robots.txt as allowed.
Instructions
Check the site's robots.txt: is url crawlable, and what crawl-delay applies?
user_agent: the agent to evaluate rules for (defaults to the server's user agent).
Returns {allowed, has_robots, crawl_delay, request_rate, sitemaps, robots_url}.
Sites with no robots.txt are reported as allowed (has_robots=false).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| user_agent | No |