Skip to main content
Glama

Robots Policy Lint

Would this crawler fetch this path?

check_path_allowed
Read-onlyIdempotent

Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
pathsNoPath or paths to test, e.g. "/blog/post". Defaults to "/". Max 200.
robots_txtYesThe contents of the robots.txt file.
user_agentYesThe crawler's product token, e.g. GPTBot, Googlebot, ClaudeBot.

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnly and idempotent annotations, the description reveals substantial behavioral detail: it returns allowed/disallowed per path, cites the exact deciding line, and flags disagreements between merge-groups and first-group-wins parsers. It also specifies the RFC 9309 matching rules including wildcard support, giving an agent a clear model of how the tool behaves.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single dense sentence that front-loads the core action ('Evaluate a pasted robots.txt') and then adds high-value specifics: RFC 9309 semantics, wildcard support, exact return behavior, and parser-disagreement flags. Every clause earns its place, and there is no filler or repetition of the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Although there is no output schema, the description compensates by explicitly stating what is returned: allowed/disallowed status per path, the exact deciding line, and parser-disagreement cases. For a read-only, idempotent evaluation tool, this is complete enough for an agent to invoke it and interpret results correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already covers all three parameters with 100% descriptive coverage, including examples and defaults, so the description does not need to add parameter-level meaning. The description's algorithm detail indirectly clarifies how user_agent and paths are interpreted, but it does not add new parameter semantics beyond what the schema provides. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb ('Evaluate') and resource ('a pasted robots.txt') for a precise task: deciding whether given paths are allowed for one crawler under RFC 9309. It also clearly distinguishes itself from the sibling tools by focusing on per-path allow/disallow decisions rather than auditing, linting, diffing, or merging policies.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'Evaluate a pasted robots.txt for one crawler and one or more paths' gives clear context for when to use this tool: when you have robots.txt content and need path-level decisions. It does not explicitly name alternative tools or state when not to use it, but the usage context is unmistakable and sufficient for an agent to select it among siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4.5/5.0
Disambiguation5/5

Each tool has a distinct job: linting syntax, auditing AI crawler access, checking a specific path, diffing two versions, and merging a policy stance. Although audit_ai_access and check_path_allowed both evaluate robots.txt access, one is crawler-level across the whole file and the other is path-level for a single crawler, so they are not easily confused.

Naming Consistency5/5

All tool names follow a consistent snake_case verb_noun pattern: audit_, check_, diff_, lint_, merge_. The objects are clear and the verbs accurately signal the operation, so the naming scheme is predictable and uniform.

Tool Count5/5

Five tools is a well-scoped size for a robots.txt linting and policy analysis server. Each tool covers a meaningful operation without redundancy, and the count feels neither thin nor bloated.

Completeness5/5

The tool set covers the core lifecycle of robots.txt analysis: linting for errors, auditing AI access, evaluating specific paths, diffing version changes, and merging maintained policy stances. No obvious dead ends or missing operations exist for the stated domain.

Resources