Skip to main content
Glama

Would this crawler fetch this path?

check_path_allowed
Read-onlyIdempotent

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree. Example: user_agent='GPTBot', paths=['/', '/blog'], with your robots_txt pasted in. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-allowed?robots_txt=&ua=GPTBot&path=/blog&s=client-dossiers

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
pathsNoPath or paths to test, e.g. "/blog/post". Defaults to "/". Max 200.
robots_txtYesThe contents of the robots.txt file.
user_agentYesThe crawler's product token, e.g. GPTBot, Googlebot, ClaudeBot.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations (readOnlyHint=true, idempotentHint=true, destructiveHint=false) already establish safety, and the description adds substantial beyond-annotation behavior: exact RFC 9309 matching semantics (longest token match for the group, longest path pattern for the rule, Allow tie-break, * and $ support), the return shape (allowed/disallowed per path with the exact deciding line), and the parser-divergence flagging. It also discloses a non-MCP invocation path. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core description is informative, but it is front-loaded with a large JSON-RPC invocation blob for no_arguments_lint_this_hosts_robots_txt that does not describe this tool and costs tokens. The RFC 9309 detail, example, and HTTP fallback each earn their place, but the overall structure is cluttered and the key purpose statement is buried after irrelevant content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description correctly carries the return-value burden ('Returns allowed/disallowed per path with the exact line that decided it') and even covers edge-case behavior (merge-groups vs first-group-wins disagreement). Combined with 100% schema parameter coverage, an agent has what it needs to call correctly. Only the misplaced opening block and lack of explicit sibling routing keep this from a 5.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds value beyond the schema by providing example user_agent tokens (GPTBot, Googlebot, ClaudeBot), clarifying that paths can be a single path or multiple ('one or more paths'), and explaining how user_agent drives group selection via RFC 9309 longest token match. This gives the agent semantic context the schema alone does not convey.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource: 'Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309', which distinguishes it from siblings like lint_robots_txt (syntax validation) and am_i_allowed (hosted-policy checks). The title 'Would this crawler fetch this path?' reinforces intent. Deducted because the description opens with a confusing JSON-RPC snippet for a different tool (no_arguments_lint_this_hosts_robots_txt) that clouds the purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied through the example (user_agent='GPTBot', paths=['/', '/blog'], with your robots_txt pasted in) and the repeated 'pasted robots.txt' wording, which signals this tool takes content rather than a URL. The HTTP alternative is disclosed as a fallback invocation path. However, there is no explicit when-to-use vs alternatives (e.g., when to prefer am_i_allowed or lint_robots_txt), and the opening 'FIRST CALL' instruction ambiguously routes the agent toward a sibling tool instead of clarifying this tool's role.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources