Check whether search engines and AI crawlers can read a site (paid)
check_crawlabilityCheck whether search engines and AI crawlers may read a page and how the site is mapped: robots.txt evaluated with Google's matching rules for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Applebot-Extended and others (with the rule that decided each), meta robots and X-Robots-Tag, canonical link, declared sitemaps with entry counts and freshness, and llms.txt, with a list of fixes. Costs 0.01 USDC per completed call, paid automatically via x402 if your client supports it. Never charged if the call fails.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL of the page or website |