robots_txt_validate
Validate robots.txt rules by fetching and parsing a site's robots.txt. Optionally probe specific URLs against user-agent groups to determine allow or deny verdicts.
Instructions
Fetch /robots.txt for a site, parse user-agent groups + Allow/Disallow + Crawl-delay + Sitemap, and return the structured ruleset. Optionally probe a list of (user_agent, url) pairs and return per-probe allow/deny verdicts using RFC 9309 longest-match.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"site_url": "<site_url>"
},
"name": "robots_txt_validate"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| probes | No | Optional list of (user_agent, url) pairs to verdict against the rules. | |
| site_url | Yes | Any URL on the target host; /robots.txt is fetched relative to it. |