Skip to main content
Glama
AI Crawlability Check

AI Crawlability Check

by Milan344s

AI Crawlability Check

AI Crawlability Check is a free Model Context Protocol (MCP) server that tells your AI assistant whether AI crawlers can reach a web page, whether the page can be indexed, and whether its content is quotable.

It checks robots.txt rules for 29 AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, and runs 70 measured checks across four tools. It is hosted, needs no account and no API key, and stores nothing.

Connect

Claude (web and desktop). Settings, Connectors, "Add custom connector", then paste the endpoint URL.

Claude Code

claude mcp add --transport http ai-crawlability https://mcp.milan-novotny.com/mcp

Cursor and other clients with a JSON config

{
  "mcpServers": {
    "ai-crawlability": { "url": "https://mcp.milan-novotny.com/mcp" }
  }
}

Then ask in plain language:

  • "Are we blocking GPTBot, ClaudeBot or PerplexityBot on https://example.com/pricing?"

  • "Check whether this page can be indexed by AI search engines, and list the fixes."

  • "Run all the crawlability checks on our five most important URLs and give me one table."

Related MCP server: OnPage.dev MCP server

Tools

All five tools are read-only.

Tool

Input

What it returns

check_ai_crawler_access

url

Allowed or blocked for 29 AI crawlers with the deciding robots.txt rule, plus 9 scored checks: robots.txt health, catch-all rules, server refusals, noindex, snippet rules, declared sitemaps, Content-Signal lines and llms.txt.

check_ai_indexability

url

15 scored checks: HTTP status and redirects, noindex, canonical target, copy present in the raw HTML, JavaScript dependence, XML sitemap coverage, response time, HTML size, structured data, dates and language.

check_ai_citability

url

22 scored checks on answer-first structure, question headings, lists and tables, statistics with sources, named entities, schema, author and freshness. Tuned for articles and guides.

check_seo_basics

url, keyword (optional)

24 on-page SEO checks: title, meta description, H1, heading order, keyword use, slug, canonical, links, alt text and schema.

get_review_rubric

type: geo or seo

A review rubric as text, for the judgment a script cannot make.

Every URL tool returns a 0 to 100 score, the measured value behind each check, the reason the check matters and a fix for each failure.

Example result

check_ai_crawler_access on a page whose robots.txt blocks most AI crawlers (shortened):

{
  "tool": "check_ai_crawler_access",
  "score": 45,
  "verdict": "Weak",
  "summary": "9 of 19 search and answer-time bots and 3 of 10 training bots may fetch this URL according to robots.txt.",
  "top_fixes": [
    { "check": "answer_bots_allowed", "status": "fail", "fix": "Allow these user agents for this path in robots.txt: ..." }
  ],
  "bots": [
    { "bot": "OAI-SearchBot", "owner": "OpenAI", "role": "answer", "allowed": false, "decided_by": "Disallow: / (group OAI-SearchBot)" }
  ]
}

Crawlers checked

Names and purposes were verified against each vendor's own crawler documentation on 19 November 2026. Bytespider has no vendor documentation and is included because it appears in most block lists. Corrections are welcome as issues or pull requests against src/robots.js.

Search and answer-time crawlers

Crawler

Owner

What it is for

Vendor position on robots.txt

OAI-SearchBot

OpenAI

Surfaces sites in ChatGPT search

Honors

ChatGPT-User

OpenAI

Live fetch when a ChatGPT user or custom GPT asks

May not apply (user-triggered)

Claude-SearchBot

Anthropic

Crawls to improve search results in Claude

Honors

Claude-User

Anthropic

Live fetch when a Claude user asks

Honors

PerplexityBot

Perplexity

Perplexity search index; not used for model training

Honors

Perplexity-User

Perplexity

Live fetch when a Perplexity user asks

Generally ignores (user-triggered)

Googlebot

Google

Google Search, including AI Overviews and AI Mode

Honors

Bingbot

Microsoft

Bing index, which also grounds Copilot

Honors

Applebot

Apple

Search across Siri, Spotlight and Safari

Honors

DuckAssistBot

DuckDuckGo

Real-time fetch for DuckDuckGo AI answers; no training

Honors

meta-webindexer

Meta

Meta AI search results and citations

Not stated

meta-externalfetcher

Meta

User-requested link fetches for Meta AI

May bypass (user-triggered)

Amzn-SearchBot

Amazon

Search in Amazon products such as Alexa; no training

Honors

Amzn-User

Amazon

Live fetch for Amazon user requests; no training

May not follow all rules (user-triggered)

MistralAI-Index

Mistral

Mistral search index; no training

Not stated

MistralAI-User

Mistral

Live fetch when a Le Chat user asks

Honors

YouBot

You.com

You.com search index

Honors

Google-Agent

Google

Google-hosted agents browsing for a user

Generally ignores (user-triggered)

Claude-Agent

Anthropic

Hosted Claude browser acting for a user

Not stated

Training crawlers and control tokens

Crawler

Owner

What it is for

Vendor position on robots.txt

GPTBot

OpenAI

OpenAI model training

Honors

ClaudeBot

Anthropic

Anthropic model training

Honors

Google-Extended

Google

Control token for Gemini training and grounding; no effect on Search

Honors

Applebot-Extended

Apple

Control token for Apple AI training on Applebot data

Honors

meta-externalagent

Meta

Training Meta AI models and improving products

Not stated

Amazonbot

Amazon

Improves Amazon products; data may train Amazon AI models

Honors

MistralAI-Training

Mistral

Mistral training datasets

Honors

CCBot

Common Crawl

Open web archive, widely reused as AI training data

Honors

AI2Bot

Allen Institute for AI

Training open language models

Not stated

Bytespider

ByteDance

ByteDance crawler, reported as model training

No vendor documentation found

robots.txt matching follows RFC 9309: the most specific user-agent group wins, the longest matching rule wins, and Allow beats Disallow on a tie.

Limits

  • It queries no index. The scores say whether a page can be crawled, indexed and quoted. They do not say whether any AI engine cites it today.

  • Bot-specific firewall rules are invisible. The server reads robots.txt, meta tags and headers. It reports when its own request is refused, and it never sends another company's crawler user agent.

  • No JavaScript. Pages are read as raw HTML on purpose, because that is how most AI crawlers read them.

  • Sitemaps are sampled. At most 5 sitemap files are read per check.

  • Public pages only. Private, local and non-standard addresses are refused.

How it works

The server is one Cloudflare Worker with no dependencies, no storage and no language model. It implements the MCP Streamable HTTP transport statelessly: every POST to /mcp is a self-contained JSON-RPC exchange.

File

Purpose

src/index.js

MCP protocol handling, tool definitions, landing page

src/crawl.js

Crawler access and indexability tools

src/citability.js, src/seo.js

Citability and SEO tools, scoring

src/robots.js

Crawler list and RFC 9309 matching

src/html.js

Dependency-free HTML extraction

src/fetchpage.js

Fetch guard: public hosts only, 1.5 MB cap, 12 second timeout

src/rubrics.js

The two review rubrics

test/run.mjs

Protocol, unit and tool tests

Run it yourself

Requires Node.js 22 or later.

npm test

Run the URL tools against a live page:

node test/run.mjs https://example.com/ "optional keyword"

Start a local server on port 8787:

npm run dev

To deploy your own copy, remove the routes line from wrangler.toml (it points at this project's domain) and run npm run deploy.

Privacy

The server keeps no state. It has no database, no accounts and no cookies, and it does not log tool inputs. For a URL it requests that page and the same site's robots.txt, llms.txt, llms-full.txt and up to 5 sitemap files, with the user agent milan-novotny-mcp/1.0. Full policy: https://www.milan-novotny.com/tools/ai-crawlability-mcp/privacy/

Author and licence

Built by Milan Novotný as a personal project. Crawler and product names belong to their owners; this project is not affiliated with or endorsed by any of them.

Released under the MIT licence.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      Enables AI assistants to audit a website's AI visibility by checking crawler access, identity files, and page signals, and running a full scored audit across ChatGPT, Claude, Gemini, and Perplexity.
      10
      106 npm
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI assistants to scan pages and whole sites for SEO and AI-search visibility, generate ready-to-paste fixes and validated structured data, and verify HTML before deploy and live results after — all through stable issue codes so an agent can fix problems one by one and prove they are gone.
      4
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI assistants to analyze public websites, follow progress, retrieve shareable reports, compare multiple URLs, list checks and recent activity, and surface fixes ranked by impact.
      Apache 2.0
    • A
      license
      A
      quality
      B
      maintenance
      Enables users to connect AI assistants or coding agents to scan pages for readiness to be cited by AI search engines and used by AI agents, then retrieve fix checklists, reports, domain data, and usage information.
      12
      72 npm
      MIT