Would this crawler fetch this path?
check_path_allowedEvaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| paths | No | Path or paths to test, e.g. "/blog/post". Defaults to "/". Max 200. | |
| robots_txt | Yes | The contents of the robots.txt file. | |
| user_agent | Yes | The crawler's product token, e.g. GPTBot, Googlebot, ClaudeBot. |