Skip to main content
Glama

Agent Web Census

ai_crawler_stats

Read-only

Aggregate AI-crawler blocking statistics across the top 10,000 websites: percentage of sites with a reachable robots.txt that fully block, partly restrict or explicitly allow each AI bot. Give a bot name for one bot, or omit for the overall table.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "ai_crawler_stats"
  }
}

Input Schema

NameRequiredDescriptionDefault
botNoOptional bot name, e.g. 'GPTBot', 'ClaudeBot', 'Google-Extended' (case-insensitive).

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare a safe read (readOnlyHint=true) over a bounded corpus (openWorldHint=false). The description adds real context beyond them: the underlying data source (robots.txt), the sample frame (top 10,000 websites), and critically the 'reachable robots.txt' caveat that qualifies how percentages should be read. It omits data-freshness/snapshot timing, which keeps it from a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no filler: the first front-loads scope, source, and metric definition; the second resolves the only input decision. Nothing repeats the schema verbatim and every clause carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description must convey the return shape, and it does so reasonably by naming the three policy categories and framing results as percentages per bot. Missing is how results are organized (per-bot rows vs. a single figure) and any snapshot/recency note, both of which an agent would want to interpret the numbers.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already supplies the parameter name, format, examples, and case-insensitivity. The description adds only one semantic detail beyond it, the default behavior when the parameter is omitted (overall table), which warrants the baseline 3 rather than higher.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb and resource ('Aggregate AI-crawler blocking statistics across the top 10,000 websites') and defines the metric precisely (percentage of sites with a reachable robots.txt that fully block, partly restrict, or explicitly allow). It never names a sibling, but the word 'aggregate' plus the dataset scope implicitly separates it from the per-site tool site_ai_crawler_policy.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit conditional guidance for the single parameter: supply a bot name for one bot, omit it for the overall table. It does not say when an agent should prefer this aggregate view over the per-site alternative (site_ai_crawler_policy), which is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.