AI Crawlability Check
Checks whether Amazon's AI crawlers (Amzn-SearchBot, Amzn-User, Amazonbot) may access a URL according to robots.txt, reporting allowed/blocked status and the deciding rule.
Checks access for Apple's AI crawlers (Applebot and Applebot-Extended) and identifies the robots.txt rule that allows or blocks them.
Checks whether ByteDance's Bytespider crawler is allowed by robots.txt for a URL and reports the deciding rule.
Checks whether DuckDuckGo's DuckAssistBot is allowed to fetch a URL according to robots.txt and reports the deciding rule.
Checks access for Google's crawlers and control tokens (Googlebot, Google-Agent, Google-Extended) and reports whether robots.txt allows or blocks them.
Checks access for Meta's AI crawlers (meta-webindexer, meta-externalfetcher, meta-externalagent) and reports allowed/blocked status with the deciding robots.txt rule.
Checks access for OpenAI's crawlers (OAI-SearchBot, ChatGPT-User, GPTBot) and reports whether robots.txt allows or blocks them for a URL.
Checks access for Perplexity's crawlers (PerplexityBot, Perplexity-User) and reports allowed/blocked status plus the deciding robots.txt rule.
This server has a remotely hosted endpoint. Open its connector listing for connection details, authentication requirements, and setup instructions:
AI Crawlability Check
AI Crawlability Check is a free Model Context Protocol (MCP) server that tells your AI assistant whether AI crawlers can reach a web page, whether the page can be indexed, and whether its content is quotable.
It checks robots.txt rules for 29 AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and runs 70 measured checks across four tools. It is hosted, needs no account and no API key, and stores nothing.
Endpoint:
https://mcp.milan-novotny.com/mcp(Streamable HTTP, no authentication)Documentation: https://www.milan-novotny.com/tools/ai-crawlability-mcp/
Privacy and terms: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/
MCP Registry name:
io.github.Milan344s/ai-crawlability-check
Connect
Claude (web and desktop). Settings, Connectors, "Add custom connector", then paste the endpoint URL.
Claude Code
claude mcp add --transport http ai-crawlability https://mcp.milan-novotny.com/mcpCursor and other clients with a JSON config
{
"mcpServers": {
"ai-crawlability": { "url": "https://mcp.milan-novotny.com/mcp" }
}
}Then ask in plain language:
"Are we blocking GPTBot, ClaudeBot or PerplexityBot on https://example.com/pricing?"
"Check whether this page can be indexed by AI search engines, and list the fixes."
"Run all the crawlability checks on our five most important URLs and give me one table."
Related MCP server: OnPage.dev MCP server
Tools
All five tools are read-only.
Tool | Input | What it returns |
|
| Allowed or blocked for 29 AI crawlers with the deciding robots.txt rule, plus 9 scored checks: robots.txt health, catch-all rules, server refusals, noindex, snippet rules, declared sitemaps, |
|
| 15 scored checks: HTTP status and redirects, noindex, canonical target, copy present in the raw HTML, JavaScript dependence, XML sitemap coverage, response time, HTML size, structured data, dates and language. |
|
| 22 scored checks on answer-first structure, question headings, lists and tables, statistics with sources, named entities, schema, author and freshness. Tuned for articles and guides. |
|
| 24 on-page SEO checks: title, meta description, H1, heading order, keyword use, slug, canonical, links, alt text and schema. |
|
| A review rubric as text, for the judgment a script cannot make. |
Every URL tool returns a 0 to 100 score, the measured value behind each check, the reason the check matters and a fix for each failure.
Example result
check_ai_crawler_access on a page whose robots.txt blocks most AI crawlers (shortened):
{
"tool": "check_ai_crawler_access",
"score": 45,
"verdict": "Weak",
"summary": "9 of 19 search and answer-time bots and 3 of 10 training bots may fetch this URL according to robots.txt.",
"top_fixes": [
{ "check": "answer_bots_allowed", "status": "fail", "fix": "Allow these user agents for this path in robots.txt: ..." }
],
"bots": [
{ "bot": "OAI-SearchBot", "owner": "OpenAI", "role": "answer", "allowed": false, "decided_by": "Disallow: / (group OAI-SearchBot)" }
]
}Crawlers checked
Names and purposes were verified against each vendor's own crawler documentation on 19 November 2026. Bytespider has no vendor documentation and is included because it appears in most block lists. Corrections are welcome as issues or pull requests against src/robots.js.
Search and answer-time crawlers
Crawler | Owner | What it is for | Vendor position on robots.txt |
| OpenAI | Surfaces sites in ChatGPT search | Honors |
| OpenAI | Live fetch when a ChatGPT user or custom GPT asks | May not apply (user-triggered) |
| Anthropic | Crawls to improve search results in Claude | Honors |
| Anthropic | Live fetch when a Claude user asks | Honors |
| Perplexity | Perplexity search index; not used for model training | Honors |
| Perplexity | Live fetch when a Perplexity user asks | Generally ignores (user-triggered) |
| Google Search, including AI Overviews and AI Mode | Honors | |
| Microsoft | Bing index, which also grounds Copilot | Honors |
| Apple | Search across Siri, Spotlight and Safari | Honors |
| DuckDuckGo | Real-time fetch for DuckDuckGo AI answers; no training | Honors |
| Meta | Meta AI search results and citations | Not stated |
| Meta | User-requested link fetches for Meta AI | May bypass (user-triggered) |
| Amazon | Search in Amazon products such as Alexa; no training | Honors |
| Amazon | Live fetch for Amazon user requests; no training | May not follow all rules (user-triggered) |
| Mistral | Mistral search index; no training | Not stated |
| Mistral | Live fetch when a Le Chat user asks | Honors |
| You.com | You.com search index | Honors |
| Google-hosted agents browsing for a user | Generally ignores (user-triggered) | |
| Anthropic | Hosted Claude browser acting for a user | Not stated |
Training crawlers and control tokens
Crawler | Owner | What it is for | Vendor position on robots.txt |
| OpenAI | OpenAI model training | Honors |
| Anthropic | Anthropic model training | Honors |
| Control token for Gemini training and grounding; no effect on Search | Honors | |
| Apple | Control token for Apple AI training on Applebot data | Honors |
| Meta | Training Meta AI models and improving products | Not stated |
| Amazon | Improves Amazon products; data may train Amazon AI models | Honors |
| Mistral | Mistral training datasets | Honors |
| Common Crawl | Open web archive, widely reused as AI training data | Honors |
| Allen Institute for AI | Training open language models | Not stated |
| ByteDance | ByteDance crawler, reported as model training | No vendor documentation found |
robots.txt matching follows RFC 9309: the most specific user-agent group wins, the longest matching rule wins, and Allow beats Disallow on a tie.
Limits
It queries no index. The scores say whether a page can be crawled, indexed and quoted. They do not say whether any AI engine cites it today.
Bot-specific firewall rules are invisible. The server reads robots.txt, meta tags and headers. It reports when its own request is refused, and it never sends another company's crawler user agent.
No JavaScript. Pages are read as raw HTML on purpose, because that is how most AI crawlers read them.
Sitemaps are sampled. At most 5 sitemap files are read per check.
Public pages only. Private, local and non-standard addresses are refused.
How it works
The server is one Cloudflare Worker with no dependencies, no storage and no language model. It implements the MCP Streamable HTTP transport statelessly: every POST to /mcp is a self-contained JSON-RPC exchange.
File | Purpose |
| MCP protocol handling, tool definitions, landing page |
| Crawler access and indexability tools |
| Citability and SEO tools, scoring |
| Crawler list and RFC 9309 matching |
| Dependency-free HTML extraction |
| Fetch guard: public hosts only, 1.5 MB cap, 12 second timeout |
| The two review rubrics |
| Protocol, unit and tool tests |
Run it yourself
Requires Node.js 22 or later.
npm testRun the URL tools against a live page:
node test/run.mjs https://example.com/ "optional keyword"Start a local server on port 8787:
npm run devTo deploy your own copy, remove the routes line from wrangler.toml (it points at this project's domain) and run npm run deploy.
Privacy
The server keeps no state. It has no database, no accounts and no cookies, and it does not log tool inputs. For a URL it requests that page and the same site's robots.txt, llms.txt, llms-full.txt and up to 5 sitemap files, with the user agent milan-novotny-mcp/1.0. Full policy: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/
Author and licence
Built by Milan Novotný as a personal project. Crawler and product names belong to their owners; this project is not affiliated with or endorsed by any of them.
Released under the MIT licence.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Audit any site's AI visibility from your assistant: crawler access, rendering, and schema.
Technical SEO and AI-readiness audits for any page, plus a check that your fix worked.
Reads a site as search engines and AI assistants do, ranks its fixes, and checks them live.
Reads a site as search engines and AI assistants do, ranks its fixes, and checks them live.
Related MCP Servers
- AlicenseAqualityBmaintenanceEnables AI assistants to audit a website's AI visibility by checking crawler access, identity files, and page signals, and running a full scored audit across ChatGPT, Claude, Gemini, and Perplexity.10106 npmMIT
- AlicenseNot gradedqualityBmaintenanceEnables AI assistants to scan pages and whole sites for SEO and AI-search visibility, generate ready-to-paste fixes and validated structured data, and verify HTML before deploy and live results after — all through stable issue codes so an agent can fix problems one by one and prove they are gone.4MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI assistants to analyze public websites, follow progress, retrieve shareable reports, compare multiple URLs, list checks and recent activity, and surface fixes ranked by impact.Apache 2.0
- AlicenseAqualityBmaintenanceEnables users to connect AI assistants or coding agents to scan pages for readiness to be cited by AI search engines and used by AI agents, then retrieve fix checklists, reports, domain data, and usage information.1272 npmMIT