fetch_robots
Fetches a website's robots.txt and identifies which AI crawlers, such as GPTBot and ClaudeBot, are blocked or allowed.
Instructions
Fetch and analyze a robots.txt file from a URL. Returns which AI bots are blocked or allowed.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website URL to fetch robots.txt from (e.g. https://example.com) |