Skip to main content
Glama

Read AI crawler access

read_ai_crawlability
Read-onlyIdempotent

Read whether the site's robots.txt lets each AI crawler in, for the latest or a chosen crawl: whether robots.txt was fetched (and its status), the sitemaps it lists, and per crawler (such as GPTBot or ClaudeBot) its operator, purpose, the matching rule, root access and how many sampled URLs it may not fetch. This is permission, not observed visits: use read_crawl_logs for crawlers that actually came. unavailable no_site_crawl means no crawl exists; robots_not_observed means the crawl could not read robots.txt.

Input Schema

NameRequiredDescriptionDefault
crawl_idNoA crawl id that read_site_health returned; omit for the latest crawl.
project_idYesA project id that list_projects returned.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • addedInput schema / properties / project_id / description
      Added value: +"A project id that list_projects returned."
  2. First observed

TDQS

A4.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already declare read-only, idempotent, non-destructive, closed-world behavior, so the safety profile is covered. The description still adds meaningful edge-case semantics that annotations cannot convey — 'unavailable no_site_crawl' means no crawl exists and 'robots_not_observed' means robots.txt could not be read — which is genuine behavioral value, though it stops short of describing ordering, sampling limits, or response shape beyond the field list.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core question and no filler; every sentence carries information, including the permission-vs-observed contrast and the status-code glossary. The first sentence is heavily nested with a long enumerative list, which slightly hurts scannability but does not waste space.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the return-value burden and does so: it lists the reported fields, the per-crawler attributes, and both failure statuses. For a two-parameter read tool with strong annotations, nothing an agent needs to select or interpret the call is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents both project_id and crawl_id (including the 'omit for latest' rule), making 3 the baseline. The description's 'latest or a chosen crawl' restates rather than extends the schema, adding no new syntax or format detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Read whether the site's robots.txt lets each AI crawler in') and enumerates exactly what is reported: fetch status, listed sitemaps, and per-crawler operator, purpose, matching rule, root access and blocked samples. It also names the sibling it is not (read_crawl_logs), so an agent can distinguish permission data from observed visits without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly routes the agent: use this for permission, use read_crawl_logs for crawlers that actually came. It also clarifies the crawl_id choice (latest vs. chosen crawl from read_site_health), so there is no ambiguity about when either parameter path applies.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources