Skip to main content
Glama

Validate robots.txt text

validate_robots
Read-onlyIdempotent

Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
pathNoURL path to test, starting with "/" (default "/").
robots_txtYesThe robots.txt file content.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
infoNo
pathNo
statsNo
validYes
errorsYes
warningsYes
ai_crawlersYes
diagnosticsNo
personal_agentsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, closed-world, idempotent behavior, so the bar is lower, yet the description still adds non-obvious context: no network access, same engine as POST /api/validate, and the 50,000-character / 64 KB JSON-RPC payload ceiling. These operational constraints are not derivable from the annotations alone.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tightly packed sentences, front-loaded with the core purpose, followed by safety posture and the size/alternative constraint. No sentence is redundant and nothing important is buried.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description needn't explain return values, and it still covers limits, alternatives, and engine parity. Nothing an agent needs to call this correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents both parameters and the 50,000 maxLength. The description nonetheless explains the rationale behind the limit (the whole JSON-RPC request must fit in 64 KB) and clarifies that 'path' tests exactly one URL path, adding meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Validate robots.txt text against RFC 9309') plus the exact outputs (errors, warnings, per-path crawler allow/block verdicts). This is clearly distinguishable from the sibling tools generate_robots and list_crawlers, which produce rather than validate content.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit conditional alternative: 'for larger files use POST /api/validate or check_site,' naming a sibling tool by name. It does not spell out when-not to use validation at all, but the escape hatch for oversized inputs is clear and actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources