robots.txt check
robots_checkFetch a site's robots.txt, list its groups and sitemap lines, and test whether specific URLs are crawlable for a chosen user agent (default Googlebot) using longest-match rules.
Instructions
Fetch a site's robots.txt, show its groups and sitemap lines, and test whether specific URLs are crawlable for a given user agent (default Googlebot) using Google's longest-match rules.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"urls": "<urls>"
},
"name": "robots_check"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | URLs to test. robots.txt is fetched from the first URL's origin. | |
| userAgent | No | Googlebot |