Skip to main content
Glama

Check which AI crawlers robots.txt allows on a page

ai_crawlers
Read-only

For eleven AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others): allowed or blocked for the page's path under robots.txt, plus the noai and noimageai opt-outs in meta and X-Robots-Tag, and whether llms.txt exists. Reads the page as static HTML; no JavaScript runs. Price: $0.002 per successful call, paid with x402 (USDC on Base, Solana and 10 more networks); failed calls are not charged.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "ai_crawlers"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesPage address, https://…

Output Schema

NameRequiredDescriptionDefault
urlNo
pathNo
optOutNo
sourceNo
llmsTxtNo
crawlersNo
checkedAtNo
robotsTxtNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A3.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnlyHint/openWorldHint annotations, the description discloses the cost model ($0.002 per successful call), the payment rail (x402, USDC on Base, Solana and 10 more networks), that failed calls are free, and the key limitation that the page is read as static HTML with no JavaScript execution. These are exactly the behavioral facts an agent needs before calling a paid, open-world tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the substantive capability and ending with the commercial terms in a separate sentence. It is dense but every clause carries a distinct fact (crawler coverage, opt-out sources, static rendering, pricing, payment rails); the crawler list could arguably be trimmed but it aids recognition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-value explanation is not required, and the description covers scope, rendering limits, and cost. Minor gaps remain: nothing about behavior on non-HTML pages, redirects, or fetch failures beyond the no-charge guarantee.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is only one parameter and the schema already documents it fully at 100% coverage, so the baseline is 3. The description adds mild context by clarifying that robots.txt evaluation is scoped to 'the page's path' and that retrieval is static HTML, but it adds no syntax or format guidance beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('allowed or blocked for the page's path under robots.txt') and enumerates the crawlers checked (GPTBot, ClaudeBot, PerplexityBot, Google-Extended), plus the meta/X-Robots-Tag opt-outs and llms.txt existence. That is far more specific than the bare name. It stops short of explicitly differentiating from the sibling 'robots' and 'llms_txt' tools, which overlap on llms.txt existence.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the framing — this is the tool for auditing AI-crawler access and noai/noimageai opt-outs on a given URL. There is no explicit when-to-use or when-not-to-use statement, and no routing away from the sibling 'robots' or 'llms_txt' tools despite the overlap in scope.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources