Crawl Readiness MCP Server
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| CRAWL_READINESS_API_KEY | No | Free API key for generator tools (available at crawlreadiness.com/dashboard). Without it, the four audit tools work, but the three generator tools will not be callable. | |
| CRAWL_READINESS_API_BASE | No | Override the API base URL (for testing). Default is https://www.crawlreadiness.com. | https://www.crawlreadiness.com |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": true
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| check_ai_readinessA | Check whether AI crawlers (ChatGPT, Claude, Perplexity, Google AI, and 50+ others) can access a website, and what access policy it states. Returns a 0-100 AI readiness score, per-crawler access status, detected AI-specific files (llms.txt, agents.json), structured data presence, meta signals, and a prioritized fix list. The score counts the crawlers that feed AI search and answers: each entry in |
| validate_schemaA | Validate all JSON-LD structured data on a URL. Extracts every block, runs each through a rules engine covering 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, Recipe, Event, etc.), and reports required-field errors, recommended-field warnings, and type-specific gotchas. |
| validate_robotsA | Audit a robots.txt file line-by-line. Detects syntax errors, empty user-agent groups, orphan Allow/Disallow lines, non-slash paths, non-numeric Crawl-delay values, unofficial Noindex usage, and wildcard traps. Also summarizes AI-bot coverage across 50+ known AI crawlers. Provide EITHER url (to fetch and audit) OR text. |
| check_content_parityA | Compare what human browsers see vs what AI crawlers see. Fetches a page four times in parallel — as Chrome, GPTBot, ClaudeBot, and PerplexityBot — and reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking. |
| generate_llms_txtA | Generate a properly formatted llms.txt file for a website. Crawls the site's sitemap, groups pages by section, pulls page titles and descriptions, and produces both the llms.txt content and a companion robots.txt snippet. The AI client can then write the returned content to disk in the user's project. Requires an API key. |
| generate_robots_txtA | Generate a complete, ready-to-save AI-crawler-aware robots.txt for a website. Fetches the existing robots.txt (if any) and returns the finished file in |
| generate_schemaA | Generate JSON-LD structured data for a website. Detects the site's name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article schemas plus starter templates for BreadcrumbList and FAQPage. Returns each schema as a block ready to paste into the site's . Requires an API key. |
| get_monitor_trendA | Read the user's Answer Analysis: for the questions a brand's customers ask, how often live AI search (Perplexity, Google AI Overviews) named the brand, which competitors were named, and the change run over run. Call with NO argument to list the user's analyses with each brand's latest share of answers naming it and the direction; pass a brand name or project id to get that brand's full trend, per-provider breakdown, competitor comparison, average position, and short example answers. Reads data existing analyses have already collected — it does not trigger new runs, and it measures rather than promises. Read-only. Requires an API key on a plan that includes Answer Analyses (Monitor, Agency or the audit). |
| check_permissionA | Ask whether a page may be used for a purpose, from every machine-readable signal the site publishes: robots.txt per crawler, Cloudflare Content-Signal, the IETF Content-Usage directive and header, an RSL licence, a TDM reservation (header or /.well-known/tdmrep.json), ai.txt, and noai/noindex/nosnippet on the page itself. Purposes: 'search' (index the page and show links and excerpts, no AI summary), 'answer' (input to a generated answer, including retrieval for RAG), 'train' (train or fine-tune a model), 'agent' (fetch or act on the page for a user, live). Returns a |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 9 tools
Most tools occupy clearly separate slots: generate vs. validate pairs for robots.txt and schema are unambiguous, and get_monitor_trend is a distinct read of existing Answer Analysis data. The main friction is the check_* cluster — check_ai_readiness (site-level score), check_permission (page-level verdict from many signals) and check_content_parity (browser vs. bot diff) all read robots.txt/crawler access, so an agent could pick the wrong one at the edges, though descriptions do differentiate them.
Every name follows verb_noun in snake_case, and the verbs map to a small predictable set (check_, generate_, validate_, get_) that mirrors the resource lifecycle. No camelCase or stylistic outliers.
Nine tools is well inside the sweet spot for a focused diagnostic/generation server. Each tool maps to a distinct artifact or check (readiness, robots, llms.txt, schema, permission, parity, monitor) with no filler.
Coverage is strong: diagnose, generate and validate across the two main artifact types (robots.txt, JSON-LD), plus llms.txt generation, per-page permission verdicts, and a crawler-parity view. The notable asymmetry is that llms.txt and agent-ready files (agents.json) can be generated but not validated, and there is no sitemap-inspection tool even though llms.txt generation depends on one.