Skip to main content
Glama
Grubbomatic

Crawl Readiness MCP Server

Validate robots.txt

validate_robots
Read-only

Audit a robots.txt file to identify syntax errors, orphan rules, invalid paths, and AI-bot coverage gaps across 50+ known crawlers.

Instructions

Audit a robots.txt file line-by-line. Detects syntax errors, empty user-agent groups, orphan Allow/Disallow lines, non-slash paths, non-numeric Crawl-delay values, unofficial Noindex usage, and wildcard traps. Also summarizes AI-bot coverage across 50+ known AI crawlers. Provide EITHER url (to fetch and audit) OR text.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlNoURL of the site whose /robots.txt should be fetched and audited.
textNoRaw robots.txt text to audit directly (alternative to url).

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.2.3

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish readOnlyHint=true and openWorldHint=true. The description adds behavioral context beyond that by explaining that a URL is fetched and audited, while raw text is audited directly, and by listing the specific issues detected. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no filler. The first sentence states the action and scope, the second lists detection categories and input requirements. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers input modes, audit scope, and specialized AI-bot coverage, which is enough for an agent to invoke the tool correctly. The only minor gap is the lack of an explicit output format, but no output schema exists and the description still conveys that results are audit findings and a coverage summary.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description adds value by stating the either/or relationship between url and text, and by clarifying that url means 'fetch and audit' while text means raw input, which goes beyond the schema property descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Audit'), names the resource ('robots.txt'), and enumerates concrete checks such as syntax errors, empty user-agent groups, orphan Allow/Disallow lines, and AI-bot coverage. This clearly distinguishes it from siblings like validate_schema and check_ai_readiness.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear context: use this tool when you need to audit a robots.txt file, and it explains the two input modes ('Provide EITHER url ... OR text'). It does not explicitly name alternative tools or when not to use them, so it stops short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.