ai-crawler-readiness
Server Details
Can AI crawlers read this site? GPTBot/ClaudeBot verdicts, robots vs edge blocking, 47k-domain index
Glama couldn't complete the latest health check. If this server requires authentication, missing or expired test credentials may be the cause. A test profile lets Glama authenticate for health checks and discover tools; it is separate from your personal connections.
If you are the author, claim ownership, then add or update a test profile under Admin → Test Profile.
- Status
- Unhealthy
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-06-18
- URL
TDQS
Scored across 5 tools
Most tools are clearly distinct: domain_readiness covers archived per-domain status, measure_domain covers fresh measurement, edge_blocking covers vendor attribution, recent_changes covers temporal flips, and web_openness covers aggregate statistics. The only close pair is domain_readiness and measure_domain, but the explicit 'right now' vs 'archive' distinction prevents serious confusion.
All names are readable snake_case, but they do not follow a consistent pattern: domain_readiness and web_openness are noun compounds, edge_blocking is noun+gerund, measure_domain is verb+noun, and recent_changes is adjective+noun. The mixed conventions are still predictable enough to infer intent, but they are not uniform.
Five tools is well-scoped for an analytics and measurement server. Each tool covers a distinct question a user would reasonably ask: one domain now, one domain from archived data, which edge vendor is responsible, what changed recently, and overall web openness. There is no obvious redundancy or bloat.
The tool surface covers the core workflows: per-domain readiness, live remeasurement, aggregate openness, vendor-level edge blocking, and change detection. A minor gap is the lack of a direct way to enumerate all domains that currently block a specific crawler, but this does not undermine the main use cases.
Available Tools
5 toolsdomain_readinessAInspect
Can AI assistants actually read this domain? Returns the measured readiness score, the per-crawler verdict (served / blocked in robots.txt / refused by the edge despite robots.txt allowing it), and which edge vendor answers for the host. Measured with real HTTP requests, not guessed from robots.txt alone. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: they are flagged robotsOnly, carry no server verdict, and are excluded from every count about what the server did. Also returns fixes: the ordered, domain-specific list of what to change to let the blocked crawlers in, separating what is written in the site own robots.txt from what the CDN/WAF applies on top of it.
| Name | Required | Description | Default |
|---|---|---|---|
| host | Yes | A bare domain, e.g. "wikipedia.org" (no scheme, no path). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does substantial work: it discloses that the tool makes real HTTP requests, that Google-Extended and Applebot-Extended are treated as robots-only tokens with no server verdict, and that they are excluded from server-side counts. It does not mention failure modes, rate limits, or side effects, but the core behavior is clearly exposed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Every clause earns its place: it front-loads the tool's purpose, then explains the measurement method, the robots-token edge case, and the structure of `fixes`. Despite its length, nothing is redundant and the structure moves from output to method to caveats.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Without an output schema, the description must define the return contract, and it covers the score, per-crawler verdicts, edge vendor, and fixes with important caveats. It stops short of describing exact response formatting, error behavior, or what happens on unreachable hosts, but for a one-parameter measurement tool the essentials are present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already documents `host` clearly with the bare-domain requirement and an example, so schema coverage is 100%. The description adds domain-specific context but no new parameter semantics, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a concrete evaluation question and then enumerates specific outputs: readiness score, per-crawler verdicts with the three possible values, the answering edge vendor, and the `fixes` list. This makes the tool's function unambiguous and distinct from generic domain-measurement tools, even though sibling names are not mentioned.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool: when you need a measured readiness answer based on real HTTP requests rather than guesses from robots.txt alone. However, it never names the sibling tools or states when not to use this tool, leaving the routing decision to the agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
edge_blockingAInspect
Who is actually doing the blocking: for each CDN/WAF vendor, the share of (domain x crawler) pairs that robots.txt ALLOWS and the server refuses anyway — i.e. how much of the blocking is an infrastructure default rather than a decision the site owner made. Each vendor also comes broken down per crawler, which separates a blanket wall (same rate for every crawler) from a managed block list that names some AI user-agents and not others.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | How many vendors to return, highest contradiction rate first (default 12). Vendors with fewer than 200 allowed pairs are left out of the table rather than reported on thin evidence. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It adds rich interpretive context: the distinction between a 'blanket wall' and a 'managed block list', and the meaning of the contradiction rate as an infrastructure default. It does not mention read-only nature or side effects, but for an analytics tool this is adequately covered by the described output intent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences each earn their place: the first defines the core metric, the second explains the crawler breakdown and its analytical value. It is detailed but not bloated; however, the second sentence is somewhat dense and could be split without losing meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a single optional parameter and no output schema, the description covers the essential context: the variables involved (vendor, crawler, contradiction rate), the threshold logic (in schema), and the interpretation. It does not spell out the exact response structure, but the dimensions are clearly implied, making it complete enough for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The sole parameter `limit` has a 100% descriptive schema entry explaining its default and the 200-pair threshold. The main description adds no parameter-specific detail, so the baseline 3 for full schema coverage applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly defines the tool's output: per CDN/WAF vendor, the share of domain-crawler pairs that robots.txt allows but the server blocks, with a crawler-level breakdown. It goes beyond naming the metric by explaining how to interpret it ('infrastructure default vs. site owner decision'), making it unambiguous and distinct from sibling tools like domain_readiness or web_openness.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage by fully explaining what the tool measures, but it never explicitly states when to use this tool over siblings or when not to use it. There are no exclusions or alternative recommendations, so the agent must infer context from the metric description alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
measure_domainAInspect
Measure a domain RIGHT NOW instead of reading the archive: 9 real HTTP requests, one from a browser user-agent and one per AI crawler user-agent, plus robots.txt / llms.txt / sitemap. Use it for any site the index has not reached yet, or when the caller wants a fresh verdict after changing robots.txt or a WAF rule. Returns the same fixes list as domain_readiness, derived from the fresh measurement.
| Name | Required | Description | Default |
|---|---|---|---|
| host | Yes | A bare domain, e.g. "wikipedia.org" (no scheme, no path). | |
| force | No | Measure again even if the archive already has a verdict from the last 24 hours (default false). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the behavioral burden. It clearly discloses that the tool performs 9 real outbound HTTP requests, lists the user-agent composition and resource types, and indicates that results come from a fresh measurement rather than cached data. It does not mention potential latency, rate-limit considerations, or whether results are persisted, but the core behavioral profile is well disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each earning its place: the first defines the action precisely, the second gives usage direction, and the third explains the return value. There is no filler or duplication of schema content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of an output schema, the description adequately explains the return by referencing the same `fixes` list as domain_readiness. It also supplies the request composition, usage conditions, and the distinction from archive data. It could be more complete by describing what the fixes list contains, but it is sufficient for an agent to invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so both parameters (host and force) are already fully described in the input schema. The description adds useful context about when a fresh measurement matters (after robots.txt/WAF changes), but it does not materially extend the parameter semantics beyond what the schema already provides. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Measure a domain') and immediately differentiates the tool from its archive-based alternative ('instead of reading the archive'). It also enumerates the exact measurement behavior (9 HTTP requests, specific user-agents, and resource types), so an agent clearly knows what this tool does and how it differs from domain_readiness.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit when-to-use guidance: 'Use it for any site the index has not reached yet, or when the caller wants a fresh verdict after changing robots.txt or a WAF rule.' It also states the alternative behavior ('instead of reading the archive'), which helps the agent choose between this tool and the archive-based sibling.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
recent_changesAInspect
What CHANGED: the domains that recently started or stopped blocking a specific AI crawler, with the day the flip was observed and whether it happened in robots.txt or at the edge. This cannot be reconstructed after the fact from any public source — it exists only because the index made the same requests the day before and the day after. Use it to answer 'who just blocked/unblocked ChatGPT, Claude, Perplexity...' or to watch one domain over time.
| Name | Required | Description | Default |
|---|---|---|---|
| days | No | Look-back window in days (default 30, max 365). | |
| host | No | Restrict to one domain, e.g. "nytimes.com" (optional). | |
| limit | No | How many changes to return, most recent first (default 50, max 500). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavioral disclosure. It clearly explains that the data is a before/after diff from the index's own requests and cannot be reconstructed from public sources. This is meaningful provenance information beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact, front-loaded with the core 'what changed' purpose, and every sentence adds value: the uniqueness explanation and concrete use cases are both useful. No filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the output semantics well for a tool with no output schema: domains, flip day, and robots.txt/edge distinction. The only minor gap is that 'a specific AI crawler' is mentioned but the input schema has no crawler parameter, leaving some ambiguity about how the agent selects which crawler's changes are returned.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents days, host, and limit. The description adds no new parameter-level details, which is fine because the baseline of 3 applies when the schema covers all parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states exactly what the tool does: it reports domains that recently started or stopped blocking a specific AI crawler, when the flip was observed, and whether the change was in robots.txt or at the edge. This is specific enough to distinguish it from sibling tools focused on static readiness or blocking state.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit use cases: answering 'who just blocked/unblocked ChatGPT, Claude, Perplexity...' and watching a single domain over time. It does not explicitly name alternatives or exclusions, but the intended context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_opennessBInspect
How much of the most-visited web is readable by AI assistants right now: the share of measured domains that block at least one AI crawler, and the served / robots-blocked / edge-blocked breakdown per crawler.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of explaining behavior. It reveals that the tool reports aggregate shares and per-crawler breakdowns, and suggests a current-snapshot view with 'right now.' However, it does not clarify how 'served,' 'robots-blocked,' or 'edge-blocked' are determined, what data source or measurement window is used, or whether any operational constraints apply.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The entire description is one focused, front-loaded sentence. It packs the metric's scope, blocking criterion, and per-crawler breakdown without filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a parameterless metric tool, the description conveys the main output and scope, and no output schema exists to fill gaps. Still, it leaves the exact meaning of the breakdown categories and the relationship to sibling tools like domain_readiness and edge_blocking unspecified, so an agent may need additional inference.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and 100% schema coverage, so there is no parameter documentation burden. The description appropriately focuses on output semantics rather than parameter details, which matches the baseline for a parameterless tool.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the tool's subject: the share of measured top domains readable by AI assistants and a per-crawler breakdown of served, robots-blocked, and edge-blocked states. It is specific and distinct from the aggregate scope, though it lacks an explicit action verb like 'returns' or 'calculates'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is for assessing overall web openness across measured domains, but it never explicitly says when to choose this tool over siblings like domain_readiness or edge_blocking. There is no when-not-to-use guidance or named alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
- Added
recent_changes
4 tool updates
- First observed
domain_readiness - First observed
edge_blocking - First observed
measure_domain - First observed
web_openness
Related MCP Connectors
Free AI-readiness audit of any URL: AI crawler rules, JS-free text, JSON-LD, llms.txt. Tool catalog.
AI visibility tracking & website fixes for better AI search presence
Free AI visibility (GEO/AEO) audit: can ChatGPT, Claude and Perplexity find and cite your website?
Checks whether a website is readable and citable by AI systems (ChatGPT, Claude, Perplexity, etc.)
Related MCP Servers
- AlicenseAqualityDmaintenanceAudits AI-bot visibility: robots.txt per-bot for 22 AI user-agents (GPTBot/ClaudeBot/PerplexityBot/etc), Cloudflare flags, JSON-LD, sitemap, llms.txt, SPA shell, plus cross-model brand mentions via Perplexity + OpenRouter. 0-100 score. SSRF-guarded, spend-capped.41MIT
- AlicenseAqualityCmaintenanceAnalyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.51MIT
- AlicenseAqualityCmaintenanceChecks a website's robots.txt and Cloudflare settings to identify AI crawler blocking. Also generates llms.txt content to improve visibility to AI answer engines.331 npmMIT
- AlicenseAqualityBmaintenanceEnables inspection of any website's AI-search readiness, checking AI crawler blocks, llms.txt, schema markup, and indexing directives from MCP clients like Claude.428 npmMIT
Glama MCP Gateway
Add one secure layer between your agents and this server.