Check whether robots.txt allows a path for a crawler
robotsChecks whether a URL's path may be crawled under robots.txt (RFC 9309): the verdict for a user agent (default: *, Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot), the rule that decided it and its line, crawl-delay, sitemaps and the matching group's lines. Longest rule wins, allow wins a tie, * and $ work. No robots.txt means everything is allowed; a server error is a 503, not charged. Price: $0.002 per successful call, paid with x402 (USDC on Base, Solana and 10 more networks); failed calls are not charged.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "robots"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | A page URL on the site whose robots.txt is checked | |
| userAgent | No | One crawler name to check, e.g. Googlebot (optional; default lists the common crawlers) |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | ||
| note | No | ||
| path | No | ||
| origin | No | ||
| source | No | ||
| status | No | ||
| results | No | ||
| sitemaps | No | ||
| checkedAt | No |