Skip to main content
Glama

Citemark

Check AI crawler access

check_ai_crawlers
Read-onlyIdempotent

Reads a website's robots.txt and reports which of 15 AI crawlers (from OpenAI, Anthropic, Google, Apple, Meta and others) may crawl its homepage, separating bots that fetch pages for AI answers from bots that collect training data, plus the robots.txt lines that would let blocked answer bots in.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "website": "<website>"
    },
    "name": "check_ai_crawlers"
  }
}

Input Schema

NameRequiredDescriptionDefault
websiteYesThe product's website, like acme.com (a full URL works too).

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, openWorld and non-destructive, so the safety profile is covered. Beyond that, the description discloses the data source (robots.txt), the crawler taxonomy, and that output includes concrete robots.txt lines that would unblock answer bots — real value with no output schema. It does not say what happens when robots.txt is missing or unparseable.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One front-loaded sentence with no filler; every clause contributes. It is dense and slightly run-on — the trailing 'plus the robots.txt lines…' clause would read better as its own sentence — but nothing is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description must convey what comes back, and it does: per-crawler verdicts, a functional categorization of bots, and concrete fix lines. Minor gaps remain around failure modes and whether only the homepage is checked, but an agent can call this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single 'website' parameter already documents accepted formats (bare domain or full URL). The description adds nothing about the parameter, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a concrete verb and resource — reads a site's robots.txt and reports per-crawler access for 15 named AI crawlers — and scopes it to the homepage. The result structure (answer bots vs training bots, plus remediation lines) further distinguishes it from siblings like check_ai_visibility or get_ai_picks.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description makes the use case obvious but never says when to pick this over check_ai_visibility or get_ai_picks, nor any prerequisite (e.g., run before start_tracking). Usage is implied by the content rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources