Skip to main content
Glama

robots_check

Fetches robots.txt, evaluates allow/disallow rules for a user agent and path, and returns sitemaps and crawl-delay. 0.001 USDC per call via x402.

Input Schema

NameRequiredDescriptionDefault
urlYesURL to check
agentNoUser agent*
paymentProofNoPayment proof: USDC tx hash on Polygon or Base (0x...) or Solana tx signature. Omit on the first call to receive the 402 payment instructions, pay, then call again with the proof.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the behavioral burden. It does valuable work by disclosing the paid model (0.001 USDC per call via x402) and what is returned (sitemaps and crawl-delay), but it omits rate limits, timeout/error behavior when robots.txt is absent, and whether results are cached.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the core operation and followed by the cost constraint. Every clause carries information; nothing is filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 3-parameter tool with no output schema and no annotations, the description covers the essential shapes: what it fetches, what it computes, what it returns, and what it costs. The only gap is the unexplained 'path' versus the url-only schema and the absence of any note on behavior when robots.txt doesn't exist.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already explains url, agent (default '*'), and the paymentProof 402 retry flow, warranting the baseline 3. The description adds little parameter detail, and it names a 'path' input that has no corresponding schema property — the path must be derived from url, which is left unexplained.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific resource (robots.txt) and three concrete operations: fetch, evaluate allow/disallow for a user agent, and return sitemaps plus crawl-delay. This is far more specific than any sibling (sitemap_parse handles sitemaps, http_check handles status codes), so an agent can route correctly without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

When to call it is only implied by the purpose statement — an agent can infer 'use this to check crawl permissions,' but there is no explicit 'use this instead of X when...' and no statement of preconditions or failure cases (e.g., missing robots.txt, unreachable host). No exclusions or alternatives are offered despite several adjacent web-inspection siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.