Skip to main content
Glama

Read robots.txt

check_robots
Read-only

Fetches a site's robots.txt and returns its status and contents. Free: needs an API key but no balance.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "check_robots"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesURL or domain, such as https://example.com/page.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish readOnlyHint and openWorldHint, so safety is covered, and the description adds useful context beyond them: it discloses both the return shape (status and contents) and the cost/auth model (an API key is required but no balance is consumed). It does not say what happens when robots.txt is missing or unreachable, which would matter for error handling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with no wasted words: the capability comes first, then the billing/auth caveat. The result is easy to parse, and the key operational constraint is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter read tool with no output schema, the description covers what it does, what it returns at a high level, and the auth/cost implications, so it is nearly complete. The only gaps are edge-case handling for missing robots.txt and guidance on choosing between this tool and check_sitemap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single url parameter has 100% schema description coverage, including an example, so the schema already carries the semantics. The description adds nothing about accepted formats, for example whether a full page URL versus a domain versus a bare host is acceptable.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource: it fetches a site's robots.txt and returns its status and contents. That is more precise than the title alone, but it does not distinguish this tool from the closely related sibling check_sitemap, so an agent must infer the difference.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The purpose strongly implies when this tool should be used, namely to inspect crawl directives before scraping. However, the description gives no explicit routing guidance, no mention of the sibling tools check_sitemap or page_audit, and no prerequisites or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.