Skip to main content
Glama
Crawl Readiness MCP Server

Crawl Readiness MCP Server

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault
CRAWL_READINESS_API_KEYNoFree API key for generator tools (available at crawlreadiness.com/dashboard). Without it, the four audit tools work, but the three generator tools will not be callable.
CRAWL_READINESS_API_BASENoOverride the API base URL (for testing). Default is https://www.crawlreadiness.com.https://www.crawlreadiness.com

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

This server publishes no instructions, or was last inspected before Glama recorded them.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": true
}

Tools

Functions exposed to the LLM to take actions

NameDescription
check_ai_readinessA

Check whether AI crawlers (ChatGPT, Claude, Perplexity, Google AI, and 50+ others) can access a website, and what access policy it states. Returns a 0-100 AI readiness score, per-crawler access status, detected AI-specific files (llms.txt, agents.json), structured data presence, meta signals, and a prioritized fix list. The score counts the crawlers that feed AI search and answers: each entry in robots.results carries its purpose and whether it is scored, and training-only crawlers are listed but never counted, so blocking them does not lower the score. Fixes marked extra: true (llms.txt, agents.json, Content-Signal, a markdown copy, agent-ready forms) are informational 'Going further' items and do not affect the score; present them as optional. Use this as the first step whenever a user asks whether AI can read a site.

validate_schemaA

Validate all JSON-LD structured data on a URL. Extracts every block, runs each through a rules engine covering 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, Recipe, Event, etc.), and reports required-field errors, recommended-field warnings, and type-specific gotchas.

validate_robotsA

Audit a robots.txt file line-by-line. Detects syntax errors, empty user-agent groups, orphan Allow/Disallow lines, non-slash paths, non-numeric Crawl-delay values, unofficial Noindex usage, and wildcard traps. Also summarizes AI-bot coverage across 50+ known AI crawlers. Provide EITHER url (to fetch and audit) OR text.

check_content_parityA

Compare what human browsers see vs what AI crawlers see. Fetches a page four times in parallel — as Chrome, GPTBot, ClaudeBot, and PerplexityBot — and reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking.

generate_llms_txtA

Generate a properly formatted llms.txt file for a website. Crawls the site's sitemap, groups pages by section, pulls page titles and descriptions, and produces both the llms.txt content and a companion robots.txt snippet. The AI client can then write the returned content to disk in the user's project. Requires an API key.

generate_robots_txtA

Generate a complete, ready-to-save AI-crawler-aware robots.txt for a website. Fetches the existing robots.txt (if any) and returns the finished file in generated.robotsTxt — the existing rules with an AI-crawler policy section merged in — plus the per-bot allow/block breakdown. Presets: 'allow-all' (every AI crawler allowed), 'recommended' (blocks only the training-only crawlers; everything that feeds AI search and answers stays allowed), 'search-only' (allows only the crawlers that feed AI search and answers, the ones the readiness score counts, and blocks the rest), 'block-all'. A file from 'recommended' or 'search-only' loses no points for crawler access. Write generated.robotsTxt to the site's /robots.txt. Requires an API key.

generate_schemaA

Generate JSON-LD structured data for a website. Detects the site's name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article schemas plus starter templates for BreadcrumbList and FAQPage. Returns each schema as a block ready to paste into the site's . Requires an API key.

get_monitor_trendA

Read the user's Answer Analysis: for the questions a brand's customers ask, how often live AI search (Perplexity, Google AI Overviews) named the brand, which competitors were named, and the change run over run. Call with NO argument to list the user's analyses with each brand's latest share of answers naming it and the direction; pass a brand name or project id to get that brand's full trend, per-provider breakdown, competitor comparison, average position, and short example answers. Reads data existing analyses have already collected — it does not trigger new runs, and it measures rather than promises. Read-only. Requires an API key on a plan that includes Answer Analyses (Monitor, Agency or the audit).

check_permissionA

Ask whether a page may be used for a purpose, from every machine-readable signal the site publishes: robots.txt per crawler, Cloudflare Content-Signal, the IETF Content-Usage directive and header, an RSL licence, a TDM reservation (header or /.well-known/tdmrep.json), ai.txt, and noai/noindex/nosnippet on the page itself. Purposes: 'search' (index the page and show links and excerpts, no AI summary), 'answer' (input to a generated answer, including retrieval for RAG), 'train' (train or fine-tune a model), 'agent' (fetch or act on the page for a user, live). Returns a verdict: 'denied' (a signal reserves this use; any reservation wins, a block on any one crawler that does the job included, and a robots.txt answering a server error counts as a full disallow), 'allowed-under-terms' (permitted on the conditions listed in terms, such as payment or a licence server), 'unknown' (a source that could carry a reservation could not be read: robots.txt answered 401 or 403, the page refused the request, or a licence could not be fetched; unknown outranks allowed) or 'allowed' (no signal reserves the use, which is not a licence). With observe: true it also requests the page once with a representative crawler's user-agent string, from Crawl Readiness's servers, and reports the status in observed: that shows how the server treats the string, not what the real crawler gets. Every source read is listed in signals with what was found and what it decided. The receipt is a dated record of all of this, with the SHA-256 of every file read, signed with Ed25519; /api/permission-key publishes the current and retired public keys, so the receipt can be verified later. Use it before fetching pages for RAG, a dataset or an agent action, and whenever the user asks whether they may use, quote, index, train on or act on a page. It is a reading of the signals the site published on that date, not legal advice. 25 calls a day per network without an API key. Documented at https://machinereadiness.com/permission.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription

No resources

TDQS

A4.4/5.0

Scored across 9 tools

Disambiguation4/5

Most tools occupy clearly separate slots: generate vs. validate pairs for robots.txt and schema are unambiguous, and get_monitor_trend is a distinct read of existing Answer Analysis data. The main friction is the check_* cluster — check_ai_readiness (site-level score), check_permission (page-level verdict from many signals) and check_content_parity (browser vs. bot diff) all read robots.txt/crawler access, so an agent could pick the wrong one at the edges, though descriptions do differentiate them.

Naming Consistency5/5

Every name follows verb_noun in snake_case, and the verbs map to a small predictable set (check_, generate_, validate_, get_) that mirrors the resource lifecycle. No camelCase or stylistic outliers.

Tool Count5/5

Nine tools is well inside the sweet spot for a focused diagnostic/generation server. Each tool maps to a distinct artifact or check (readiness, robots, llms.txt, schema, permission, parity, monitor) with no filler.

Completeness4/5

Coverage is strong: diagnose, generate and validate across the two main artifact types (robots.txt, JSON-LD), plus llms.txt generation, per-page permission verdicts, and a crawler-parity view. The notable asymmetry is that llms.txt and agent-ready files (agents.json) can be generated but not validated, and there is no sitemap-inspection tool even though llms.txt generation depends on one.

Maintenance

ActivityMaintained
ResponsivenessOutdated data