Skip to main content
Glama

check_robots

Evaluate robots.txt content offline to determine whether specified AI crawlers can access a URL path, using default or custom user-agent tokens.

Instructions

Evaluate robots.txt text (no network) for AI crawlers.

Args: robots_txt: full robots.txt content. path: URL path to test, e.g. "/" or "/blog/post". user_agents: crawler tokens to test; defaults to the built-in AI crawler list.

Input Schema

NameRequiredDescriptionDefault
pathNo/
robots_txtYes
user_agentsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It usefully discloses that no network access occurs and that a built-in AI crawler list is applied by default, but it never says what the evaluation returns (allow/deny per agent, matched rule) nor how conflicting directives are resolved.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One-line purpose statement followed by a compact Args block where every line carries information not present in the schema. No filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description should sketch the return shape, and it does not — an agent cannot tell whether it receives a boolean, a per-agent map, or matched rules. Parameter coverage is adequate, but the result contract is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: robots_txt is defined as full file content, path gets concrete example values ("/", "/blog/post"), and user_agents is defined as crawler tokens with a stated default. Only minor ambiguity remains, such as whether tokens match case-insensitively or accept full UA strings.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: evaluate robots.txt text for AI crawlers, with the key qualifier that it performs no network fetch. This clearly differentiates it from a live site check, but the siblings (list_ai_crawlers, check_site) are never named to make the boundary explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"no network" and "for AI crawlers" imply the usage context (testing already-fetched robots.txt content against AI crawler tokens), but there is no explicit when-to-use statement or routing to check_site for cases where the caller doesn't already have the text.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools