Skip to main content
Glama

Revisar reglas de robots.txt

robots-txt-check
Read-onlyIdempotent

Decide si un agente puede acceder a una ruta según un robots.txt dado (RFC 9309) e indica la regla decisiva. No descarga nada. Cuándo: hay que leer, construir o revisar con exactitud una URL, una cabecera HTTP, un token, una versión, un rango de IP, un documento HTML o Markdown o un archivo para agentes. No para: informa, para un robots.txt dado y 21 rastreadores de IA conocidos (robots-ai-crawlers). Efectos secundarios: ninguno (solo lectura, sin red, sin estado guardado). Determinista: la misma entrada da la misma salida. Límites: entrada JSON de hasta 64 KB, 1 s por llamada, resultado de hasta 1 MB.

EN: Decides whether a user agent may fetch a path under a supplied robots.txt (RFC 9309: most specific group, longest match, Allow wins ties, * and $ wildcards) and reports the deciding rule. Nothing is fetched. When: a URL, HTTP header, token, version string, IP range, HTML or Markdown document or agent-readiness file has to be parsed, built or checked exactly. [EN completo / full EN: inputSchema.description]

Input Schema

NameRequiredDescriptionDefault
pathNoURL path (with query) (string; optional; default "/"; up to 4096 chars)./
robots_txtYesrobots.txt content (string; required).
user_agentYesUser-agent product token or full UA (string; required; up to 500 chars).

Output Schema

NameRequiredDescriptionDefault
ruleNo
groupNo
allowedNo
sitemapsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / properties / user_agent / examples
      Added value: +[
      +  "Googlebot"
      +]
  2. Changed1 schema field changed
    • addedInput schema / description
      Added value: +"EN: Decides whether a user agent may fetch a path under a supplied robots.txt (RFC 9309: most specific group, longest match, Allow wins ties, * and $ wildcards) and reports the deciding rule. Nothing is fetched. When: a URL, HTTP header, token, version string, IP range, HTML or Markdown document or agent-readiness file has to be parsed, built or checked exactly. Not for: reports,ing for a given robots.txt and 21 known AI crawlers and agent fetchers (robots-ai-crawlers). Side effects: none (read-only, no network access, no stored state). Deterministic: the same input gives the same output. Limits: JSON input up to 64 KB, 1 s per call, result up to 1 MB."
  3. First observed

TDQS

A3.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnly/idempotent/non-destructive annotations, it discloses no network access, no stored state, determinism (same input, same output), and concrete limits (64 KB JSON input, 1 s per call, 1 MB result). It also names the RFC 9309 matching semantics applied (most specific group, longest match, Allow wins ties, wildcards), which the agent could not infer from structured data.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness2/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The same content is duplicated in Spanish and English, and the English half simply points at inputSchema.description rather than adding value. The 'When' block is a long list of unrelated items that does not earn its place. The genuinely useful facts (RFC semantics, no network, limits) are buried among noise.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With annotations, a full-coverage schema and an output schema in place, the description supplies what remains: matching semantics, side-effect profile and resource limits. Only the misleading 'Cuando' enumeration keeps completeness below a 5.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

All three parameters are fully documented in the schema (100% coverage) and the tool has an output schema, so the schema carries the semantics. The description only generically refers to a 'supplied robots.txt' and a user agent, adding nothing about the path parameter, its default of '/', or the UA token format. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence gives a specific verb+resource+scope: decide whether a user agent may fetch a path under a supplied robots.txt per RFC 9309, plus it reports the deciding rule and fetches nothing. That clearly separates it from siblings like robots-ai-crawlers and llms-txt-lint. The purpose is nevertheless diluted by the generic 'Cuando/When' list of unrelated artifacts (HTTP headers, IP ranges, HTML documents), which obscures the actual scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It uses an explicit 'Cuando / When' and 'No para / Not for' structure and names the alternative sibling (robots-ai-crawlers) for AI-crawler lookups. But the when-list is garbled and off-topic, claiming the tool is for parsing URLs, HTTP headers, tokens, IP ranges and HTML/Markdown documents, which this tool does not do. The routing is present but partly misleading.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources