Is a URL allowed for a crawler by robots.txt? Rules, crawl-delay and sitemaps
web_robotsIs a URL allowed for a crawler by robots.txt? Rules, crawl-delay and sitemaps. Fetches the site's /robots.txt and evaluates a URL path for a user-agent with RFC 9309 matching (longest match wins, * and $ wildcards). Returns allowed/disallowed, the deciding rule, crawl-delay, declared sitemaps and the agent groups present. Price: $0.005 USDC per call via x402 (eip155:8453).
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "web_robots"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Page URL (its origin's robots.txt is read) | |
| userAgent | No | * |