Skip to main content
Glama

check_ai_crawler_access

Read-only

Returns a site's robots.txt verdict for AI crawlers, including allowed and blocked bots, with a paid upgrade link. Use to inspect one public website's declared crawler policy. Not for proving that a CDN or firewall actually permits a bot, or for ongoing monitoring. Apify Actor links can be paid with x402 via Apify, no Apify account needed.

Input Schema

NameRequiredDescriptionDefault
urlYesPublic website URL or domain.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only and open-world behavior; the description adds real context beyond them: the result includes a paid upgrade link, and payment for Apify Actor links can run through x402 with no Apify account required. Billing and upgrade behavior would otherwise be invisible to the agent, though return format and rate-limit details are omitted.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core purpose and the main exclusion are front-loaded, and each sentence carries distinct information. The closing x402/Apify payment sentence is useful but slightly tangential to the tool's function.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and a single simple parameter, the description still conveys the shape of the return (allowed and blocked bots plus an upgrade link) and sets the scope limit (declared policy, not enforcement). That is nearly complete for a one-parameter read-only lookup; only pagination or edge-case behavior for missing robots.txt is unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only one parameter exists and schema coverage is 100%, so the baseline would be a 3; the description adds a little by framing the URL as a public website (or domain) whose declared policy is inspected, implying external accessibility is expected.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: returns a site's robots.txt verdict for AI crawlers, listing allowed and blocked bots. It also scopes the input to one public website, which separates it from broader siblings like seo_audit or seo_browser_audit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a clear use case (inspect one public website's declared crawler policy) and two explicit exclusions (proving a CDN/firewall actually permits a bot, ongoing monitoring). It stops short of naming an alternative tool for the network-level question, so it is strong but not a full when/when-not/alternative mapping.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources