Skip to main content
Glama

May a crawler fetch this URL?

robots_check
Read-only

Reads the site's robots.txt and says whether a crawler may fetch a URL (RFC 9309: the longest matching rule wins, Allow wins a tie), with the matching rule, crawl-delay and sitemaps. No AI involved; if robots.txt cannot be read you are not charged. Price: $0.01 USDC per call (x402).

Input Schema

NameRequiredDescriptionDefault
urlYesPublic http(s) URL you want to fetch
user_agentNoYour crawler's name, for example MyBot (default: any crawler)

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only and open-world nature, but the description adds genuinely useful behavioral facts: no AI is involved, the call is free if robots.txt cannot be read, and the price is $0.01 USDC via x402. That pricing/billing-exception detail is exactly the kind of context annotations cannot express.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core purpose, then behavioral and cost details in tight clauses. Slightly dense with parentheticals, but essentially every clause carries information an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description correctly names what comes back (matching rule, crawl-delay, sitemaps) as well as cost and the unreadable-robots.txt fallback. Complete enough to call correctly, missing only explicit alternative-tool routing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both parameters (url, user_agent) are fully documented there, including the default 'any crawler'. The description mentions matching rule, crawl-delay and sitemaps, which are return values rather than parameter guidance, so it adds no parameter meaning beyond the schema. Baseline 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a concrete verb and resource: reads the site's robots.txt and reports whether a crawler may fetch a URL. The RFC 9309 rule semantics (longest match wins, Allow wins ties) make it unambiguous versus siblings like domain_check or check_links.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the description (checking crawl permission for a URL), but it never says when to prefer this over sibling tools such as domain_check or check_links, nor any preconditions beyond the input URL being public http(s). Adequate but no explicit routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources