Skip to main content
Glama

Agent Ready by WorkPods (hosted)

Can an AI crawler read this page?

can_ai_fetch
Read-only

Answers whether AI crawlers may read one URL, from robots.txt, the robots meta tags (including ones naming the crawler), the X-Robots-Tag header and the HTTP status. Use it when the person asks whether ChatGPT, Claude, Perplexity or Google can see a page. result.bots lists each crawler with verdict: readable, readable_not_for_index (it may fetch the page but is asked not to index it), blocked_by_robots_txt or page_unavailable, plus robotsTxtAllows and the pageDirectives found. Firewalls that block crawlers by user agent or IP are not visible from here. These are deterministic checks of public pages fetched by Agent Ready by WorkPods, with no AI model and no account. They do not cover rankings, keyword volumes, backlinks or how AI answers mention the brand. Returns JSON: summary (one sentence), reportUrl (a shareable report page for the person, kept 30 days; null for tools that keep none) and result (the details below). Errors come back as one plain sentence. Recorded: the checked domain and path (never the query string), the tool, which checks failed and the app name; never an IP address or page text.

Input Schema

NameRequiredDescriptionDefault
botNoOne robots.txt product token, such as GPTBot, ClaudeBot or PerplexityBot. Defaults to every AI crawler Agent Ready tracks.
urlYesThe page to ask about: an absolute http or https URL, such as https://example.com/pricing.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds substantial context beyond the readOnly/openWorld annotations: the checks are deterministic, involve no AI model, no account, and pages are fetched by Agent Ready by WorkPods. It discloses retention (reportUrl kept 30 days, null for tools that keep none), error format (one plain sentence), and exactly what is recorded (domain and path, never query string, IP or page text).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core question and the use-case trigger, then evidence sources, then caveats and return shape. It is dense but nearly every clause carries information; the verdict enumeration and privacy/recording tail make it longer than strictly needed for tool selection.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, and the description compensates fully by describing the return shape (summary, reportUrl, result with per-bot verdicts and robotsTxtAllows/pageDirectives) and the error format. Combined with the rich annotations, an agent has everything needed to call this correctly and set expectations for the person.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description clarifies scope that the schema does not: it is one URL at a time, and the bot token may name specific crawlers whose robots.txt rules are honored ('including ones naming the crawler'). The per-bot behavior is further implied by result.bots listing each crawler.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a precise verb and resource: 'Answers whether AI crawlers may read one URL.' It also names the exact evidence sources (robots.txt, robots meta tags, X-Robots-Tag, HTTP status), which separates it from siblings like check_page or check_site_files that operate on pages or whole sites.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a concrete trigger ('Use it when the person asks whether ChatGPT, Claude, Perplexity or Google can see a page') and clear exclusions (firewalls blocking by user agent or IP, rankings, keyword volumes, backlinks, brand mentions in AI answers). It stops short of naming a specific sibling tool to use instead for those out-of-scope questions, which keeps it below a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources