Check robots.txt
check_robots_txtFetch and analyze a site's robots.txt and how it affects crawling.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to analyze. |
check_robots_txtFetch and analyze a site's robots.txt and how it affects crawling.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to analyze. |
Changes observed during successful MCP inspections.
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, openWorldHint=true, and destructiveHint=false, so safety is covered. The description adds that it analyzes crawling impact, but omits whether missing robots.txt is handled, auth/rate-limit needs, or return format. With annotations covering the safety profile, this is a modest addition.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One tightly written sentence with no filler; the core action is front-loaded. Every word contributes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter read tool with annotations, the description is minimally adequate, but with no output schema it does not explain what the analysis returns (e.g., directives, crawl delays, blocked paths) or when to reach for it over siblings. That leaves an agent guessing about the result shape.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single url parameter, and the description adds no syntax, format, or validation details beyond what the schema provides. Baseline 3 applies when the schema fully documents parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (fetch and analyze) and resource (robots.txt), and adds the effect on crawling. The resource name alone distinguishes it from sibling analyzers like analyze_meta_tags or check_hreflang, though it does not explicitly name alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus siblings such as run_seo_audit or check_hreflang. The description states capability only, leaving the agent to infer context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Add one secure layer between your agents and this server.