Would this crawler fetch this path?
check_path_allowedFIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree. Example: user_agent='GPTBot', paths=['/', '/blog'], with your robots_txt pasted in. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-allowed?robots_txt=&ua=GPTBot&path=/blog&s=client-dossiers
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| paths | No | Path or paths to test, e.g. "/blog/post". Defaults to "/". Max 200. | |
| robots_txt | Yes | The contents of the robots.txt file. | |
| user_agent | Yes | The crawler's product token, e.g. GPTBot, Googlebot, ClaudeBot. |