Skip to main content
Glama

Check which AI bots a site allows

check_ai_bot_access
Read-onlyIdempotent

Reads the public robots.txt of a website (one request to https:///robots.txt, nothing else on the site) and says, for each AI crawler and AI agent of the migurei bot registry (training crawlers such as GPTBot and ClaudeBot, search bots, fetchers that act for a user, and others), whether that bot is allowed or disallowed for the path "/", which robots.txt rule decided it, and whether the file exists. Input is a domain or URL; only public domain names are accepted (no IP addresses, no ports, no private hosts). Read-only; the domain is not stored. robots.txt is a request to bots, not access control, and a User-Agent is only a claim.

Input Schema

NameRequiredDescriptionDefault
domainYesA public domain name or a URL, for example example.com or https://example.com/page. Only its host is used.

Output Schema

NameRequiredDescriptionDefault
botsYes
domainYes
methodYes
summaryYes
learn_moreYes
robots_txtYes
robots_urlYes
registry_versionYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, open-world, idempotent, non-destructive behavior, yet the description adds substantial context beyond them: exactly one HTTP request is made, the domain is not stored, robots.txt is advisory rather than access control, and a User-Agent is only a claim. These are real behavioral traits an agent could not infer from annotations alone.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the action and the request scope before detailing the per-bot output and input constraints. The final caveat about robots.txt not being access control is relevant but slightly lengthens an already dense definition; every clause is useful, though the description is near the upper bound of appropriate length.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-value explanation is unnecessary; the description covers scope, input restrictions, per-bot output semantics, and the advisory nature of robots.txt. For a single-parameter read-only tool, an agent has everything needed to call it correctly and interpret the result.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and already explains that the domain may be a URL and that only its host is used. The description adds meaning beyond the schema by excluding IP addresses, ports, and private hosts, which clarifies what values are valid. It stops short of describing length limits or URL parsing edge cases, so not a 5.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource (reads robots.txt) and precisely scopes the operation: one request to https://<host>/robots.txt, nothing else on the site. It also enumerates what the answer contains (per-bot allowed/disallowed, deciding rule, file existence), which lets an agent distinguish it from get_site_info without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear conditions for use: input is a domain or URL, only public domain names accepted, no IPs, ports, or private hosts. It does not explicitly route the agent to or away from the sibling get_site_info, so it falls short of full alternative guidance, but the scope constraints are strong enough to select the tool correctly.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources