stillindexed_check_robots
Read-onlyIdempotent
Fetch robots.txt for the host and report the effective allow/disallow for each path, wildcard group vs Googlebot separately.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute http(s) URL, or a hostname (https is assumed). | |
| paths | No | Paths to evaluate. Defaults to ["/"]. |