robots.txt checker
robots-checkReads a site's robots.txt and says whether a crawler may fetch a path, with the matching rule, the major AI crawlers' access and the sitemaps. Returns { robots_url, robots_status, agent, path, allowed, rule, ai_crawlers, sitemaps } Price: 1 credit (1 credit = 1 CAD cent) per successful call; input that does not match the schema is rejected before billing.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL to fetch. Private and internal hosts are refused. | |
| agent | No | Crawler user agent, default *. |