Skip to main content
Glama

Robots Policy Lint

Which AI crawlers does this file actually stop?

audit_ai_access
Read-onlyIdempotent

Evaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and category. Also names the tokens in your file that match no known crawler (a typo blocks nothing) and separates the crawlers that document obedience from the ones observed ignoring robots.txt, which need an IP or WAF rule instead.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
pathNoPath to test the verdict at. Defaults to "/".
robots_txtYesThe contents of the robots.txt file.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With readOnlyHint and idempotentHint already present, the description adds meaningful behavioral detail: it evaluates against 'every AI crawler in this index', flags tokens matching no known crawler, and separates obedient crawlers from those observed ignoring robots.txt. This goes beyond the annotations and clarifies the tool's analytical behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no filler. The primary purpose is front-loaded, and the second sentence packs additional valuable features (unknown tokens, obedience classification) without bloat. Every clause earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Even without an output schema, the description specifies the shape and scope of results (blocked/allowed lists per operator and category, unknown tokens, obedience split). It covers the key context an agent needs to call the tool and interpret results; only a precise schema-like format description is missing, which is not essential here.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% (both robots_txt and path have descriptions), so the baseline is 3. The description itself does not add parameter-specific syntax or semantics beyond 'pasted robots.txt', which restates the schema. No extra value is contributed to parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('Evaluate') and resource (pasted robots.txt against the AI crawler index), and clearly states the two outputs (blocked and allowed lists per operator and category). It also differentiates itself from sibling tools like lint_robots_txt or check_path_allowed by focusing on AI-crawler coverage and unknown-token identification, so an agent can pick it unambiguously.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a clear usage context: auditing a pasted robots.txt for AI crawler coverage and determining which crawlers need extra IP/WAF rules. It implicitly distinguishes itself from alternative tools by offering a comprehensive index-based audit, though it stops short of explicitly naming siblings or stating when not to use them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4.5/5.0
Disambiguation5/5

Each tool has a distinct job: linting syntax, auditing AI crawler access, checking a specific path, diffing two versions, and merging a policy stance. Although audit_ai_access and check_path_allowed both evaluate robots.txt access, one is crawler-level across the whole file and the other is path-level for a single crawler, so they are not easily confused.

Naming Consistency5/5

All tool names follow a consistent snake_case verb_noun pattern: audit_, check_, diff_, lint_, merge_. The objects are clear and the verbs accurately signal the operation, so the naming scheme is predictable and uniform.

Tool Count5/5

Five tools is a well-scoped size for a robots.txt linting and policy analysis server. Each tool covers a meaningful operation without redundancy, and the count feels neither thin nor bloated.

Completeness5/5

The tool set covers the core lifecycle of robots.txt analysis: linting for errors, auditing AI access, evaluating specific paths, diffing version changes, and merging maintained policy stances. No obvious dead ends or missing operations exist for the stated domain.

Resources