Site Robots
site_robotsFetch and parse robots.txt policy from any public HTTP(S) URL. Use to determine allowed and disallowed paths for web crawling.
Instructions
Fetch and parse the origin robots.txt policy. Use only for public HTTP(S) resources; it does not execute JavaScript or bypass access controls. Pass url as an absolute public HTTP(S) URL. Keep fresh=false to allow cache reuse; set fresh=true only when a new upstream fetch is required.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute public HTTP or HTTPS URL to inspect. Private, loopback, link-local, metadata-service and otherwise SSRF-sensitive destinations are rejected. | |
| fresh | No | Set true only when a new upstream fetch is required; false allows the bounded cached result and is cheaper for the origin. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| data | Yes | ||
| tool | Yes | ||
| quality | No | ||
| url_final | Yes | ||
| checked_at | Yes | ||
| from_cache | Yes | ||
| request_id | Yes | ||
| content_hash | Yes | ||
| url_requested | Yes |