Read and parse a website's robots.txt
robotsRead and parse a website's robots.txt: user-agent groups with allow/disallow rules, declared sitemaps, and which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot...) are blocked from the whole site. Fast (under 2 s). Free: 20 calls/day per user across free tools, no payment needed. Above that: $0.002 per call (x402, USDC on Base).
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "robots"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL | |
| payment | No | x402 payment: the base64 PAYMENT-SIGNATURE value signed for the requirements returned by an unpaid call (USDC on Base, gasless). Omit on the first call to get the price and requirements. |