Skip to main content
Glama

Read and parse a website's robots.txt

robots
Read-only

Read and parse a website's robots.txt: user-agent groups with allow/disallow rules, declared sitemaps, and which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot...) are blocked from the whole site. Fast (under 2 s). Free: 20 calls/day per user across free tools, no payment needed. Above that: $0.002 per call (x402, USDC on Base).

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "robots"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesPublic http(s) URL
paymentNox402 payment: the base64 PAYMENT-SIGNATURE value signed for the requirements returned by an unpaid call (USDC on Base, gasless). Omit on the first call to get the price and requirements.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=true, so the safety profile is covered. The description adds genuinely useful behavior beyond them: a latency bound (under 2 s), a free quota (20 calls/day per user across free tools), and the paid path ($0.002/call via x402, USDC on Base), which prepares the agent for a possible payment challenge. It stops short of describing response shape for unpaid calls.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose and return contents are front-loaded in one dense sentence, followed by latency and pricing. Every sentence earns its place, though "Fast (under 2 s)" is marginal filler and the pricing clause is comparatively heavy for a call-routing document.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter read tool with no output schema, the description covers purpose, returned data, performance, and the full cost/auth model needed to invoke it successfully, including what to do on an unpaid first call.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both parameters are already documented in the schema, establishing a baseline of 3. The description corroborates the payment mechanism (x402, USDC on Base, price) but adds no syntax or format detail about the `url` parameter beyond what the schema states.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb+resource ("Read and parse a website's robots.txt") and enumerates what it returns: user-agent groups with allow/disallow rules, declared sitemaps, and blocked AI crawlers. It never names a sibling tool, so differentiation (e.g. vs. sitemap) is left to inference, which keeps it from a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit when-to-use or when-not-to-use statement, nor any pointer to alternatives. Usage is only implied by the purpose statement plus the pricing context, which tells the agent about cost but not about tool selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.