Crawl Readiness MCP Server
This MCP server lets an AI client audit, validate, generate, and monitor website AI-readiness from inside supported MCP clients.
Check a site’s AI readiness with a 0–100 score, per-crawler access status, detected AI files, structured data, meta signals, and prioritized fixes.
Validate JSON-LD structured data on any URL and report required-field errors, warnings, and type-specific issues.
Audit robots.txt line-by-line, including syntax problems and AI-bot coverage.
Compare what human browsers see versus what AI crawlers see, flagging parity issues, JS-only shells, cloaking, or bot blocking.
Generate an llms.txt file and companion robots.txt snippet (requires API key).
Generate an AI-crawler-aware robots.txt with presets: allow-all, search-only, recommended, or block-all (requires API key).
Generate Organization, WebSite, Article, BreadcrumbList, and FAQPage JSON-LD snippets (requires API key).
Read existing LLM Monitor trend data, including brand mention share, provider breakdown, competitors, and weekly movement (requires API key).
Audits whether a website is accessible to Google AI crawlers, including robots.txt status and structured-data issues.
Audits whether a website is accessible to Perplexity's AI crawler, including robots.txt status and content parity.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Crawl Readiness MCP Server Is my site AI-readable? URL: mydomain.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Crawl Readiness MCP Server
An MCP server that checks whether a website is set up for AI the way its owner intends: can GPTBot, ClaudeBot, PerplexityBot and 50+ other AI crawlers read it, and what does its access policy allow? Then it generates the files (llms.txt, an AI-crawler robots.txt, JSON-LD structured data), reads your Answer Analysis, and tells you whether a page's published signals permit a use (search, answers, training, agent action), inside Claude Desktop, Cursor, Cline, Windsurf and other MCP clients.
Ask Claude "can AI read my site?" and it will actually check, running a live audit against 50+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google's and more) and reading the access policy the site states. Then ask it to fix what is blocked, and it can generate a proper llms.txt, robots.txt or JSON-LD schema and write it straight into your project.
No more copy-paste between your editor and a browser.
What it does
Nine tools your AI assistant can call:
Audit (no signup):
check_ai_readiness— Score a site 0–100 against 50+ AI crawlers, with a prioritized fix list; extras that don't affect the score are marked as suchvalidate_schema— Audit JSON-LD structured data on any URL (20+ types, per-type rules)validate_robots— Line-by-line robots.txt auditcheck_content_parity— Compare what humans see vs what AI crawlers see
Permission (no signup):
check_permission— Ask whether a page may be used for search, answers, training or agent action. Reads robots.txt, Content-Signal, Content-Usage, an RSL licence, the TDM reservation, ai.txt and the page's own directives: any reservation wins, and a source that could not be read makes the answer unknown, never allowed. Returns a dated receipt signed with Ed25519, with a hash of every file read. Documented at machinereadiness.com/permission
Generate (requires free API key):
generate_llms_txt— Auto-crawl a site and produce a properly formatted llms.txtgenerate_robots_txt— AI-crawler-aware robots.txt with preset policiesgenerate_schema— Organization / WebSite / Article JSON-LD ready to paste
Answer Analysis (read-only; needs an API key, and the analyses come with Monitor, Agency or the audit):
get_monitor_trend— Read your Answer Analysis: for the questions your customers ask, whether live AI search (Perplexity, Google AI Overviews) named your brand or a competitor, and the change run over run (reads existing analyses; doesn't trigger runs)
Related MCP server: @didyouseo/mcp
Install
Claude Desktop, one click
Download crawl-readiness-mcp-<version>.mcpb from the latest release and double-click it (or drag it into the Claude Desktop window). Claude Desktop shows what the extension does and asks for the optional API key. The bundle carries the server and its dependencies, so nothing else is installed.
Claude Code, as a plugin
This repository is also a Claude plugin: the .mcp.json at its root starts the server (pinned to this version), and the ai-readiness skill tells Claude when to use which tool and how to read the results. Add it from the Claude directory once it is listed there. To try it before then, clone the repository and start Claude Code with it loaded:
claude --plugin-dir ./crawl-readiness-mcpClaude Code asks for the optional API key when the plugin is enabled and keeps it in secure storage; the plugin's own files never contain it.
For Claude Desktop, by hand
Open your config file:
macOS:
~/Library/Application Support/Claude/claude_desktop_config.jsonWindows:
%APPDATA%\Claude\claude_desktop_config.json
Add this to the mcpServers block:
{
"mcpServers": {
"crawl-readiness": {
"command": "npx",
"args": ["-y", "crawl-readiness-mcp"]
}
}
}Restart Claude Desktop. You'll see "crawl-readiness" listed in the tools panel.
For Cursor
Open Cursor Settings → Features → Model Context Protocol → Add new global MCP server.
{
"crawl-readiness": {
"command": "npx",
"args": ["-y", "crawl-readiness-mcp"]
}
}For Cline / Windsurf / Continue
Same shape — add crawl-readiness to your MCP servers list with npx -y crawl-readiness-mcp as the command.
Unlock the generator tools
The five checks (the four audit tools and check_permission) work out of the box, no signup. The three generator tools need an API key — free at crawlreadiness.com/dashboard (50 checks/month on the free tier, no credit card). get_monitor_trend needs a key too, on a plan that includes Answer Analyses (Monitor, Agency or the audit).
Once you have a key, add it via env in the same config:
{
"mcpServers": {
"crawl-readiness": {
"command": "npx",
"args": ["-y", "crawl-readiness-mcp"],
"env": {
"CRAWL_READINESS_API_KEY": "cr_live_xxxxxxxxxxxx"
}
}
}
}Restart the client. All nine tools are now callable.
Example prompts
Once installed, try any of these in your AI client:
"Is my portfolio site AI-readable? URL: yourdomain.com"
"Check if ChatGPT can see acme-inc.com"
"Validate the JSON-LD on stripe.com — is anything missing?"
"Compare what humans see vs what AI crawlers see on producthunt.com"
"Generate an llms.txt for my site and save it to
/public/llms.txt""My site scored 68/100 — walk me through the fixes"
"May I train on https://example.com/research? Check what the site publishes."
"Before we build the RAG index, which of these pages reserve AI answers?"
Your AI can chain tools automatically — audit, identify issues, fix them, and write files to your project without you copy-pasting anything.
Configuration
Env var | Purpose | Default |
| API key (free) for the generators; lifts the daily limit on | none |
| Override the API base URL for every tool, the permission check included (for testing) |
|
What tools receive from the server
Each tool returns the full JSON response from the Crawl Readiness API — the same data the web dashboard renders. Your AI has access to:
check_ai_readiness: score, robots.txt status per crawler, structured-data flags, homepage signals, prioritized fix list (fixes marked
extra: trueare optional and not scored)validate_schema: per-block issues (errors, warnings, info), suggested missing types
validate_robots: line-by-line issues, AI-bot coverage summary
check_content_parity: verdict + per-crawler HTTP status, word overlap %, warnings
check_permission: verdict (
denied,allowed-under-terms,allowed,unknown) with the reason, every signal as it was read, the page's status, and the signed receipt with a hash of every file read; withobserve: true, also how the server treated a crawler's user-agent string sent from our serversgenerate_llms_txt: the file content + companion robots.txt snippet
generate_robots_txt: merged robots.txt with the AI policy applied
generate_schema: JSON-LD scripts for each detected schema type
get_monitor_trend: per analysis, the share of answers naming the brand, the per-provider split, the competitors named, the trend run over run, and short example answers
What this sends, and where
Every tool makes HTTPS requests to https://www.crawlreadiness.com, and check_permission to https://machinereadiness.com (the same service on the standard's own host), and nowhere else. What leaves your machine is the website address (and for the generators, the inputs) you asked about, the API key if you configured one, and a crawl-readiness-mcp/<version> user agent. The server stores nothing locally, runs no other programs, and reads no files; writing a generated file into your project is something your AI client does, with your say-so. The service's own data handling is described at crawlreadiness.com/privacy.
The plugin's .mcp.json starts the published npm package pinned to the same version as the repository (npx -y crawl-readiness-mcp@<version>), so what runs is the code in this repository at that tag.
Support
Docs: crawlreadiness.com/mcp
Email: support@crawlreadiness.com
About
Crawl Readiness checks whether a website is set up for AI the way its owner intends: what AI crawlers can read, what its access policy allows, and what is missing, with the files that fix it. Every generator tool in this MCP server is also available on the web at crawlreadiness.com/tools. The MCP server just makes the same tools callable from inside your AI assistant, so the "audit → fix → apply" loop happens in one conversation without leaving your editor.
Releasing (maintainers)
npm run release validates server.json and prints the publish steps. The last step uses the mcp-publisher CLI, which is not part of this repository: download it from the MCP Registry releases and keep it on your PATH or in the repo root (it is git-ignored there).
License
MIT © 2026 Crawl Readiness / Tundrastone
Available Tools
9 toolscheck_ai_readinessCheck AI ReadinessARead-only
Check whether AI crawlers (ChatGPT, Claude, Perplexity, Google AI, and 50+ others) can access a website, and what access policy it states. Returns a 0-100 AI readiness score, per-crawler access status, detected AI-specific files (llms.txt, agents.json), structured data presence, meta signals, and a prioritized fix list. The score counts the crawlers that feed AI search and answers: each entry in robots.results carries its purpose and whether it is scored, and training-only crawlers are listed but never counted, so blocking them does not lower the score. Fixes marked extra: true (llms.txt, agents.json, Content-Signal, a markdown copy, agent-ready forms) are informational 'Going further' items and do not affect the score; present them as optional. Use this as the first step whenever a user asks whether AI can read a site.
Inputs: url (required).
check_content_parityContent Parity CheckARead-only
Compare what human browsers see vs what AI crawlers see. Fetches a page four times in parallel — as Chrome, GPTBot, ClaudeBot, and PerplexityBot — and reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking.
Inputs: url (required).
check_permissionCheck Permission to Use a PageARead-only
Ask whether a page may be used for a purpose, from every machine-readable signal the site publishes: robots.txt per crawler, Cloudflare Content-Signal, the IETF Content-Usage directive and header, an RSL licence, a TDM reservation (header or /.well-known/tdmrep.json), ai.txt, and noai/noindex/nosnippet on the page itself. Purposes: 'search' (index the page and show links and excerpts, no AI summary), 'answer' (input to a generated answer, including retrieval for RAG), 'train' (train or fine-tune a model), 'agent' (fetch or act on the page for a user, live). Returns a verdict: 'denied' (a signal reserves this use; any reservation wins, a block on any one crawler that does the job included, and a robots.txt answering a server error counts as a full disallow), 'allowed-under-terms' (permitted on the conditions listed in terms, such as payment or a licence server), 'unknown' (a source that could carry a reservation could not be read: robots.txt answered 401 or 403, the page refused the request, or a licence could not be fetched; unknown outranks allowed) or 'allowed' (no signal reserves the use, which is not a licence). With observe: true it also requests the page once with a representative crawler's user-agent string, from Crawl Readiness's servers, and reports the status in observed: that shows how the server treats the string, not what the real crawler gets. Every source read is listed in signals with what was found and what it decided. The receipt is a dated record of all of this, with the SHA-256 of every file read, signed with Ed25519; /api/permission-key publishes the current and retired public keys, so the receipt can be verified later. Use it before fetching pages for RAG, a dataset or an agent action, and whenever the user asks whether they may use, quote, index, train on or act on a page. It is a reading of the signals the site published on that date, not legal advice. 25 calls a day per network without an API key. Documented at https://machinereadiness.com/permission.
Inputs: url (required), purpose (required), agent, observe.
generate_llms_txtGenerate llms.txtA
Generate a properly formatted llms.txt file for a website. Crawls the site's sitemap, groups pages by section, pulls page titles and descriptions, and produces both the llms.txt content and a companion robots.txt snippet. The AI client can then write the returned content to disk in the user's project. Requires an API key.
Inputs: url (required).
generate_robots_txtGenerate robots.txtA
Generate a complete, ready-to-save AI-crawler-aware robots.txt for a website. Fetches the existing robots.txt (if any) and returns the finished file in generated.robotsTxt — the existing rules with an AI-crawler policy section merged in — plus the per-bot allow/block breakdown. Presets: 'allow-all' (every AI crawler allowed), 'recommended' (blocks only the training-only crawlers; everything that feeds AI search and answers stays allowed), 'search-only' (allows only the crawlers that feed AI search and answers, the ones the readiness score counts, and blocks the rest), 'block-all'. A file from 'recommended' or 'search-only' loses no points for crawler access. Write generated.robotsTxt to the site's /robots.txt. Requires an API key.
Inputs: url (required), preset.
generate_schemaGenerate JSON-LD SchemaA
Generate JSON-LD structured data for a website. Detects the site's name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article schemas plus starter templates for BreadcrumbList and FAQPage. Returns each schema as a block ready to paste into the site's . Requires an API key.
Inputs: url (required).
get_monitor_trendRead the Answer AnalysisARead-only
Read the user's Answer Analysis: for the questions a brand's customers ask, how often live AI search (Perplexity, Google AI Overviews) named the brand, which competitors were named, and the change run over run. Call with NO argument to list the user's analyses with each brand's latest share of answers naming it and the direction; pass a brand name or project id to get that brand's full trend, per-provider breakdown, competitor comparison, average position, and short example answers. Reads data existing analyses have already collected — it does not trigger new runs, and it measures rather than promises. Read-only. Requires an API key on a plan that includes Answer Analyses (Monitor, Agency or the audit).
Inputs: project.
validate_robotsValidate robots.txtARead-only
Audit a robots.txt file line-by-line. Detects syntax errors, empty user-agent groups, orphan Allow/Disallow lines, non-slash paths, non-numeric Crawl-delay values, unofficial Noindex usage, and wildcard traps. Also summarizes AI-bot coverage across 50+ known AI crawlers. Provide EITHER url (to fetch and audit) OR text.
Inputs: url, text.
validate_schemaValidate JSON-LD SchemaARead-only
Validate all JSON-LD structured data on a URL. Extracts every block, runs each through a rules engine covering 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, Recipe, Event, etc.), and reports required-field errors, recommended-field warnings, and type-specific gotchas.
Inputs: url (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 9 tools
Most tools occupy clearly separate slots: generate vs. validate pairs for robots.txt and schema are unambiguous, and get_monitor_trend is a distinct read of existing Answer Analysis data. The main friction is the check_* cluster — check_ai_readiness (site-level score), check_permission (page-level verdict from many signals) and check_content_parity (browser vs. bot diff) all read robots.txt/crawler access, so an agent could pick the wrong one at the edges, though descriptions do differentiate them.
Every name follows verb_noun in snake_case, and the verbs map to a small predictable set (check_, generate_, validate_, get_) that mirrors the resource lifecycle. No camelCase or stylistic outliers.
Nine tools is well inside the sweet spot for a focused diagnostic/generation server. Each tool maps to a distinct artifact or check (readiness, robots, llms.txt, schema, permission, parity, monitor) with no filler.
Coverage is strong: diagnose, generate and validate across the two main artifact types (robots.txt, JSON-LD), plus llms.txt generation, per-page permission verdicts, and a crawler-parity view. The notable asymmetry is that llms.txt and agent-ready files (agents.json) can be generated but not validated, and there is no sitemap-inspection tool even though llms.txt generation depends on one.
Maintenance
Related MCP Connectors
6-category AI readiness, Schema.org, robots.txt, /llms.txt & 25k-token audit + auto-fix MCP server
Run SEO + AI-visibility (GEO) audits from Claude, Cursor & other AI clients.
Free MCP tools for AI-search visibility: crawler checks, page audits, and llms.txt generation.
Live AEO, GEO, JSON-LD Schema, robots.txt AI crawler, and Agent Readiness diagnostic auditor.
Related MCP Servers
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to crawl live websites, audit AEO readiness, generate Schema.org @graph JSON-LD, llms.txt, ai.txt, and robots.txt, inject structured data into HTML, validate optimizations, and retrieve framework-specific code snippets.MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI assistants to run SEO audits and use site tools for meta tags, robots.txt, schema validation, SERP previews, Core Web Vitals, Open Graph, hreflang, Search Console, GA4, keywords, and backlinks via the Model Context Protocol.73 npmMIT
- AlicenseNot gradedqualityCmaintenanceEnables AI assistants such as Claude, ChatGPT, Cursor, and Claude Code to run 25+ specialized SEO checks on any URL, generate XML sitemaps and robots.txt files, extract brand design elements, and audit technical SEO, content quality, mobile responsiveness, and security.MIT
- AlicenseNot gradedqualityCmaintenanceEnables auditing websites for AI/LLM readiness and generating fixes such as robots.txt rules, llms.txt, JSON-LD schema, and meta tags, plus checking brand and domain citations through natural language.1MIT