Skip to main content
Glama

Check whether robots.txt allows a path for a crawler

robots
Read-only

Checks whether a URL's path may be crawled under robots.txt (RFC 9309): the verdict for a user agent (default: *, Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot), the rule that decided it and its line, crawl-delay, sitemaps and the matching group's lines. Longest rule wins, allow wins a tie, * and $ work. No robots.txt means everything is allowed; a server error is a 503, not charged. Price: $0.002 per successful call, paid with x402 (USDC on Base, Solana and 10 more networks); failed calls are not charged.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "robots"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesA page URL on the site whose robots.txt is checked
userAgentNoOne crawler name to check, e.g. Googlebot (optional; default lists the common crawlers)

Output Schema

NameRequiredDescriptionDefault
urlNo
noteNo
pathNo
originNo
sourceNo
statusNo
resultsNo
sitemapsNo
checkedAtNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnlyHint and openWorldHint, and the description adds substantial behavior: longest-match rule ordering, allow-wins-tie, wildcard and $ support, implicit allow when no robots.txt exists, and 503 on server error with no charge. Pricing/paid-access terms (x402, USDC, per-call cost) are genuinely load-bearing context for an agent deciding to call. It stops short of describing the shape of the error/permission responses, but covers far more than the annotations do.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and semantics are front-loaded and every clause carries information — rule precedence, defaults, pricing. The single paragraph is dense and run-on, packing pricing and x402 details inline, but it is not padded; a light structural split would improve scanning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter, read-only check with a declared output schema, the definition supplies everything needed: what is evaluated, default agent behavior, rule resolution order, edge cases (missing robots.txt, server error), and cost. Return-value formatting can be delegated to the output schema, so nothing material is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description nevertheless adds meaning beyond the schema by enumerating the actual default crawler set (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) and the '*' default, clarifying exactly what omitting userAgent produces.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a precise verb and resource — 'Checks whether a URL's path may be crawled under robots.txt (RFC 9309)' — and immediately scopes what comes back (verdict, deciding rule, crawl-delay, sitemaps). An agent can distinguish this from a generic crawler or site-metadata tool without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage context is implied rather than stated: the tool is clearly meant to be consulted before crawling, and the optional userAgent default is explained. However, there is no explicit when-to-use/when-not guidance and no sibling alternative is named (e.g., ai_crawlers, llms_txt), so the agent must infer the boundary.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources