Is a URL allowed by robots.txt?
robots_txt_checkIs a URL allowed by robots.txt? Check URL. Check whether a URL may be crawled according to the site's robots.txt for any user agent, with the matching rule, sitemaps and crawl-delay.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to check (e.g. https://www.google.com/search?q=test) | |
| agent | No | User agent (e.g. Googlebot) |