Validate robots.txt
validate_robotsAudit a robots.txt file to identify syntax errors, orphan rules, invalid paths, and AI-bot coverage gaps across 50+ known crawlers.
Instructions
Audit a robots.txt file line-by-line. Detects syntax errors, empty user-agent groups, orphan Allow/Disallow lines, non-slash paths, non-numeric Crawl-delay values, unofficial Noindex usage, and wildcard traps. Also summarizes AI-bot coverage across 50+ known AI crawlers. Provide EITHER url (to fetch and audit) OR text.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "validate_robots"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | URL of the site whose /robots.txt should be fetched and audited. | |
| text | No | Raw robots.txt text to audit directly (alternative to url). |