robots-check
robots.txt (RFC 9309) rule reading with llms.txt and AI-use signals — Reads a supplied robots.txt the RFC 9309 way for an agent deciding whether to fetch up to 25 paths as one product-token user agent: group selection (merged groups, "" fallback), longest-match with the Allow-wins-tie rule, "" and "$" patterns, percent-encoding normalization, fetch-status handling (4xx = no rules, 5xx = disallow all), the rule and line that decided each path. It also lints a suppl
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"userAgent": "<userAgent>",
"paths": "<paths>"
},
"name": "robots-check"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| paths | Yes | Path plus optional query beginning with /, or a full http(s) URL (path and query are used). | |
| headers | No | Response headers of the page: X-Robots-Tag, tdm-reservation, tdm-policy. | |
| llmsTxt | No | ||
| purpose | No | ai-training also treats noai, noimageai and tdm-reservation: 1 as restrictions. Default crawl. | |
| metaTags | No | HTML meta elements: robots, googlebot, tdm-reservation, tdm-policy. | |
| robotsTxt | No | robots.txt body, at most 512,000 UTF-8 bytes (500 KiB). | |
| userAgent | Yes | Product token only (letters, underscore, hyphen), e.g. "Googlebot"; no version. | |
| fetchStatus | No | HTTP status the robots.txt fetch ended with (after at most 5 redirects). Omit when robotsTxt is supplied and was a 2xx. |