Check a URL Against Robots Rules
agent_access_check_urlWhat a public institution's published robots rules say for one URL and one product token, from Closed to Machines, a MORVS register of which public institutions serve an automated client that names itself and gives a contact address, what their robots rules say for each path checked, and where the official route to their data is (US federal agency hosts from the Federal Register's agency list, the 50 state legislatures, the G20 members' national government portals and national law sites, and MORVS's own two hosts; checked weekly by two clients, curl and Python urllib, from one residential US vantage, both sending the same identified user agent). Takes url (http or https) and optional product_token (a robots.txt product token such as GPTBot, ClaudeBot or Googlebot; MORVS when omitted). The rules are evaluated under RFC 9309 against the robots.txt text the register archived in its latest cycle, at the exact path and query of the URL; this server never requests the site. Returns robots.result (disallow_rule_matches, allow_rule_matches, no_rule_matches, no_rules_html_served, no_rules_rfc_unavailable, complete_disallow_rfc_unreachable, undetermined_edge_refused, or unknown), the decisive rule with its line, crawl_delay_s and max_requests_per_day where the file sets a delay for the token, the edge observation (the row's result class, status label, streak_weeks and each dated check with its responder), the official machine route where one was reviewed (URL, type, when it answered without a key, the route path's own robots result for the token, terms URL and status) or none_found or not_reviewed, and next_step (use_official_route, stop, no_rule_against_path_check_terms or unknown). The same host gives different verdicts for different paths. A host the register does not cover returns unknown with the absence note. Nothing here grants permission: a robots rule is a request a site publishes, not a licence, and a page that answered certifies no deeper path. A refusal is the edge's response to this client at a stated time; the vendor named is the responder seen in headers, not the institution's policy, and no record shows a reason. "No answer" (DNS, connection, TLS or timeout) is never a refusal; none_found (reviewed, nothing found where listed) and not_reviewed are kept apart, and neither shows that no route exists. Each answer leads with uncertainty[], then as_of (the cycle date), vantage, the source page on morvs.ai and the licence, and carries the question, vantage, label, robots-not-licence and absence notes (and the self-row note for MORVS's own hosts) and the claim rules. Data more than 9 days old makes each robots verdict and next step unknown, with a note. A failed or malformed read of the register is reported as temporarily unavailable, never as an empty answer. Answers are kept under about 25 KB. Organisations only; officials appear only by office and no person is named.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Required http or https URL (at most 2,048 characters), for example https://www.ilga.gov/legislation/; its exact path and query are evaluated. | |
| product_token | No | Optional robots.txt product token, for example GPTBot, ClaudeBot or Googlebot; MORVS when omitted. An unlisted token is evaluated without a group. |