robots.txt checker
robots-checkReads a site's robots.txt and says whether a crawler may fetch a path, with the matching rule, the major AI crawlers' access and the sitemaps. Returns { robots_url, robots_status, agent, path, allowed, rule, ai_crawlers, sitemaps } Price: 1 credit (1 credit = 1 CAD cent) per successful call; input that does not match the schema is rejected before billing.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "robots-check"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL to fetch. Private and internal hosts are refused. | |
| agent | No | Crawler user agent, default *. |