Skip to main content
Glama

Check robots.txt

robots_check
Read-onlyIdempotent

Fetch a site's robots.txt and test paths against it the way Google does (most specific user-agent group, longest rule wins, Allow wins a tie, * and $ wildcards), for Googlebot, Bingbot and AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) or the user agents you pass. Also reports the sitemaps it declares and what a missing or failing robots.txt means.

Input Schema

NameRequiredDescriptionDefault
siteYesAny URL on the site.
pathsNoPaths or URLs to test, up to 100.
user_agentsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, openWorld, non-destructive), and the description goes further by disclosing the matching semantics (most specific UA group, longest rule wins, Allow wins ties, * and $ wildcards). It also notes that missing or failing robots.txt behavior is reported, adding real behavioral context beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense paragraph that front-loads the core action and matching semantics before covering inputs and secondary outputs. Every clause carries information, though the parenthetical agent list makes it slightly heavy for a one-sentence structure.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description correctly takes on the burden of describing return content: path test results, declared sitemaps, and missing/failing robots.txt interpretation. It is nearly complete for a read-only open-world tool, with only the exact result format left implicit.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema documents 'site' and 'paths' but leaves 'user_agents' empty; the description compensates by enumerating the supported agents (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended) and noting custom agents can be passed. This adds meaning the 67%-coverage schema does not carry.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Fetch a site's robots.txt and test paths against it') and immediately distinguishes itself from siblings like llms_txt or sitemap_check by naming the exact matching algorithm used. An agent knows exactly what this tool does without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains what gets tested and which user agents are available, which implies the use case, but never states when to reach for this tool versus related siblings such as sitemap_check, llms_txt or page_audit. No explicit when-to-use or when-not-to-use guidance is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources