Skip to main content
Glama

Agent Web Census

site_ai_crawler_policy

Read-only

Which AI crawlers does a website block in robots.txt? Looks up a domain in our scan of the top 10,000 websites (Tranco list) and returns, per AI bot (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, ...), whether it is blocked, partly restricted or explicitly allowed, plus llms.txt presence.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "domain": "<domain>"
    },
    "name": "site_ai_crawler_policy"
  }
}

Input Schema

NameRequiredDescriptionDefault
domainYesDomain or URL, e.g. 'nytimes.com' or 'https://www.nytimes.com/section'.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnlyHint/openWorldHint annotations, the description discloses the data source and scope (a scan of the top 10,000 websites from the Tranco list) and the exact per-bot return values plus llms.txt presence. The one real gap is behavior for domains outside that scan, which is not stated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences with no wasted words: the purpose question comes first, followed by how the lookup works and what it returns. The rhetorical-question opener is slightly less direct than a declarative verb phrase but still front-loads the intent effectively.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has no output schema, and the description compensates by summarizing the return values: per-bot blocked, partly restricted or explicitly allowed, plus llms.txt presence. The only missing piece for an agent to call it safely is what happens with a domain absent from the top-10k scan.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is a single required 'domain' parameter whose schema description is 100% covered and already shows accepted forms ('nytimes.com' or a full URL). The description only says it looks up a 'domain' and adds no normalization or precedence rules, so the schema is doing the work here, matching the baseline for high coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource: it looks up a domain in a scan of the top 10,000 websites and reports robots.txt policy per AI bot. An agent can immediately tell what question this tool answers. It stops short of naming a sibling or drawing an explicit boundary against ai_crawler_stats, which keeps it at 4 rather than 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied by the opening question ('Which AI crawlers does a website block in robots.txt?'); there is no explicit when-to-use statement, no exclusion of cases where a different tool applies, and no mention of what to do when the domain is not in the Tranco scan. The context is inferable but not stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.