Skip to main content
Glama

stillindexed_check_robots

Read-onlyIdempotent

Validate a robots.txt with RFC 9309 and Google's parsing rules. Give url to fetch the live file, or content to test a pasted one, plus proposed to diff a second version. For each path or URL (up to 100, or use_sitemap to read them from the file's Sitemap line) it reports allowed or blocked for Googlebot, Googlebot-Image, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot, with the deciding rule and its line number, plus lints (unsupported fields, 500 KiB limit, BOM, HTTP status semantics). With proposed, changes lists only the verdicts that flip. Same check as https://stillindexed.com/tools/robots-txt-tester.

Input Schema

NameRequiredDescriptionDefault
urlNoAbsolute http(s) URL, or a hostname (https is assumed).
pathsNoPaths or full URLs to evaluate. Defaults to ["/"].
contentNoA robots.txt to test instead of fetching one. With url also set, url only names the host the paths belong to.
proposedNoA second robots.txt. The response lists the path x crawler verdicts that differ from the first.
use_sitemapNoAlso test URLs from the first Sitemap: line (one level of sitemap index followed). Each sitemap fetch costs one unit of the hourly budget.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed6 schema fields changed
    • addedInput schema / properties / content
      Added value: +{
      +  "description": "A robots.txt to test instead of fetching one. With url also set, url only names the host the paths belong to.",
      +  "type": "string"
      +}
    • changedInput schema / properties / paths / description
      Previous value: -"Paths to evaluate. Defaults to [\"/\"]."New value: +"Paths or full URLs to evaluate. Defaults to [\"/\"]."
    • addedInput schema / properties / paths / maxItems
      Added value: +100
    • addedInput schema / properties / proposed
      Added value: +{
      +  "description": "A second robots.txt. The response lists the path x crawler verdicts that differ from the first.",
      +  "type": "string"
      +}
    • addedInput schema / properties / use_sitemap
      Added value: +{
      +  "description": "Also test URLs from the first Sitemap: line (one level of sitemap index followed). Each sitemap fetch costs one unit of the hourly budget.",
      +  "type": "boolean"
      +}
    • changedInput schema / required
      Previous value: -[
      -  "url"
      -]New value: +[]
  2. First observed

TDQS

A4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/openWorld/non-destructive, and the description adds substantial context on top: the 100-path cap, the full list of evaluated crawlers, the lint categories (unsupported fields, 500 KiB limit, BOM, HTTP status semantics), the diff-only output shape when 'proposed' is set, and a concrete cost disclosure ('Each sitemap fetch costs one unit of the hourly budget'). The rate-limit/budget statement is exactly the kind of behavior an agent cannot get from annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is front-loaded in sentence one, followed by input modes and then the return shape, so an agent can stop reading once it has what it needs. The paragraph is dense and packs several unrelated concerns (crawler list, lints, cost, diff behavior) into one block, but no sentence is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and five parameters, the description fully compensates by describing the return payload (per-crawler verdicts, deciding rule and line number, lint list, flipped-verdict-only changes) and the fetching behavior. An agent knows what it will get back and what each mode costs without inspecting anything else.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter is already documented with its interaction rules (content vs url, proposed semantics, use_sitemap cost). The description largely restates those same rules rather than adding new meaning; the only marginal addition is 'up to 100, or use_sitemap to read them from the file's Sitemap line', which is already implied by the schema. Baseline 3 applies when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Validate a robots.txt with RFC 9309 and Google's parsing rules') and enumerates the scope of the output: per-path verdicts for eleven named crawlers, the deciding rule and line number, and lints. The robots.txt focus is distinctive enough to separate it from most siblings, though it never explicitly contrast itself with stillindexed_check_ai_crawlers or stillindexed_crawler_diff, which overlap on the crawler-verdict angle.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains invocation modes clearly ('Give url to fetch the live file, or content to test a pasted one, plus proposed to diff a second version'), which tells an agent how to drive the parameters. It gives no guidance on when to prefer this tool over stillindexed_check_ai_crawlers, stillindexed_crawler_diff or stillindexed_check_url, leaving the sibling selection to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources