validate_robots
Check if a URL can be scraped by verifying robots.txt compliance to ensure web scraping follows site rules and avoids blocked content.
Instructions
Check robots.txt compliance for specific URLs
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "validate_robots"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to check robots.txt compliance for | |
| userAgent | No | User agent to check against (default: *) | * |