Check which AI crawlers robots.txt allows on a page
ai_crawlersFor eleven AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others): allowed or blocked for the page's path under robots.txt, plus the noai and noimageai opt-outs in meta and X-Robots-Tag, and whether llms.txt exists. Reads the page as static HTML; no JavaScript runs. Price: $0.002 per successful call, paid with x402 (USDC on Base, Solana and 10 more networks); failed calls are not charged.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "ai_crawlers"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Page address, https://… |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | ||
| path | No | ||
| optOut | No | ||
| source | No | ||
| llmsTxt | No | ||
| crawlers | No | ||
| checkedAt | No | ||
| robotsTxt | No |