Skip to main content
Glama

Server Details

Check what AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot) receive from any website, graded A to F, and get a signed, offline-verifiable verdict before an agent reads, cites, connects to or buys from a site. Covers robots.txt and content rights, site safety, MCP server checks and crawler-log verification.

Ownership verified
Status
Healthy
Uptime
100.0% over the last 24 hours
Last Tested
Transport
Streamable HTTP · MCP 2025-11-25
URL
Repository
emmanuelorta/crawlcheck-core
GitHub Stars
0
ChatGPT
–
Claude
–

TDQS

A3.7/5.0

Scored across 28 tools

Disambiguation4/5

The descriptions explicitly route among adjacent tools such as preflight, resolve_domain, check_site_safety and best_source, and most tools target a distinct resource or action. A few nearby pairs still require careful reading, notably scan_domain/resolve_domain/registry_lookup and explain_finding/machine_record, but their boundaries are stated.

Naming Consistency3/5

All names are snake_case and descriptive, with recognizable families such as resolve_*, fix_*, video_* and mcp_*. However, the set mixes verb-led names with noun phrases like preflight, machine_record, telemetry and public_counts, so the pattern is readable but not fully uniform.

Tool Count2/5

At 28 tools, the surface is above the 25-tool heavy threshold and asks an agent to hold many specialized operations in context. The breadth may reflect a broad platform, but consolidation into fewer multiplexed tools would reduce selection cost.

Completeness4/5

The tool set covers scanning, domain resolution, safety, rights, robots, MCP registry checks, video audits, crawler verification, public counts and targeted fix planning. Minor gaps remain around broader search/monitoring or lifecycle operations, but the core CrawlCheck surface is well represented.

Publisher details

Operator
VSNARY · Publisher source
Operator website
https://crawlcheck.io
Vendor relationship
First-party
Restrictions
None. No account, API key, OAuth or paid plan is needed; every tool is free to call. Rate limit: 5 scans per 10 seconds per address. Paid API keys (Developer and Growth plans) add higher scan volume only.

Available Tools

28 tools
best_sourceBest source for a claimA
Read-onlyIdempotent
Inspect

Call when choosing WHICH business or site to cite, recommend or call. Give domains (2-10 candidate sites you found) or trade and city for a measured market. Returns the candidates ranked by what CrawlCheck verified (preflight decision for cite, security and threat listing, domain age, uptime, published facts that directories agree with, FTC phone complaints) with reasons for and against each; sites that share one phone number or near-identical homepage text are collapsed into one network so a copycat cannot outrank the business it copies. Signed. For one domain you have already chosen, use preflight or resolve_domain.

Inputs: city, trade, domains.

check_site_safetySite safety checkA
Read-onlyIdempotent
Inspect

Use when the only question is whether a site is listed as dangerous, before an agent reads, cites, links to or pays it. Returns whether it is listed as malware or phishing (Google Safe Browsing, a public malware-blocking DNS resolver, PhishTank), which list says so, where to check it and where its owner requests a review, plus HSTS preload and the hacked-site signatures from its latest CrawlCheck scan. An unread list is reported as unread, never as clean. For an allow/block decision on a specific action call preflight; for the full record call resolve_domain. Free, read-only.

Inputs: domain (required).

corpus_stateCorpus state and countsA
Read-onlyIdempotent
Inspect

Use to judge how representative CrawlCheck's public dataset is before quoting its figures. Returns how many distinct domains have been measured, how they split by platform, rendering, size, language and kind, and which strata are still under-sampled. No input; read-only. For CrawlCheck's headline figures use public_counts.

Inputs: no named parameters.

crawler_pathObserved crawler path on a siteAInspect

Where each crawler's path into a domain dies, as an observed data flow: edge decision, robots.txt as a file, robots.txt rules, then the page, JSON-LD, sitemap, llms.txt and entitymap.json it reached. Give agent (e.g. ClaudeBot, GPTBot, googlebot) for one identity's path and a one-line answer; omit it for every identity plus the stores and breaks. Uses the latest record on file, or scans first when there is none (fresh=true forces a scan). The answer-engine output is drawn but never measured.

Inputs: domain (required), agent, fresh.

draft_llmsDraft an llms.txt for a siteA
Read-onlyIdempotent
Inspect

Draft an llms.txt from the domain's own homepage and up to 25 declared pages, using their titles and descriptions. A draft for the owner to cut down, not a publication.

Inputs: domain (required).

explain_findingExplain a findingA
Read-onlyIdempotent
Inspect

Why CrawlCheck decided a finding: the rule and its revision, the exact fetches it compared (identity, status, bytes, sha256), when the condition held on the site, and a Merkle proof that the decision is inside the signed manifest. Give id (a d1: decision or f1: finding id from a report), or domain alone for its top finding, or domain plus code for a named finding. Free, no key; read-only. A finding's age and history need a licence for that domain (send it in x-crawlcheck-key).

Inputs: id, code, path, domain.

field_checkCheck a business fact (phone, hours, address)A
Read-onlyIdempotent
Inspect

Call before using a phone number, address, opening hours or service area you hold for a business. One signed verdict per business field. field=phone|address|hours|service_area and value= the value you hold. phone: matches, former_value (with dates), not_this_business (the site never published it and another publisher is named with the page that prints it), or not_published. address: matches, same_street_other_city, former_value, not_this_business, not_published or nothing_declared. hours: a week (Mo-Fr 08:00-17:00) gives matches or differs per day; a moment (Sa 10:00) gives open or closed. service_area: declared, not_declared or nothing_declared. Built from the business's own site (and its Business Profile for the licence holder), its dated fact history, CrawlCheck's scans of other sites, and for an unknown number a web search whose cited pages are read plus a YellowPages reverse lookup.

Inputs: field (required), value (required), domain (required).

fix_entitymapFix an entity mapA
Read-onlyIdempotent
Inspect

Use when a site has no entitymap.json, or to replace a wrong one. Returns a starter entitymap.json built from the domain's latest scan record: business name, phone, coordinates and declared service areas as entities with SERVES relations. Fields the site never stated are marked TODO, never guessed. Read-only; needs a prior scan of the domain (run scan_domain first if there is none).

Inputs: domain (required).

fix_machine_chainFix a heavy machine-file chainA
Read-onlyIdempotent
Inspect

Use after a scan reports MACHINE_CHAIN_HEAVY (crawlers must read too many or too large files before the page). Returns a fix plan: the files a crawler reads before the page and the 404s that absent agent files return, then measured reductions (llms.txt to an index, a sitemap index, robots.txt comments and duplicate groups, minified JSON, short 404s) with the chain size after each step and the step at which the finding stops firing. Read-only; changes nothing on the site. Give domain (uses the latest scan on file) or a report id. For other findings use explain_finding.

Inputs: id, domain.

fix_mostly_codeFix a page that is mostly codeA
Read-onlyIdempotent
Inspect

A fix plan for PAGE_IS_MOSTLY_CODE: the homepage's measured bytes by category, the largest blocks with what to do with each, ordered steps with the text ratio after each, and the step at which the detector would stop firing - or, when moving code cannot clear it, how much markup to cut or text to add. Give domain (fetched now) or id (a stored scan).

Inputs: id, domain.

fix_robotsFix a robots.txtA
Read-onlyIdempotent
Inspect

The domain's served robots.txt, corrected: * group rules copied into named groups that lacked them (shadowing), a Sitemap line added when missing, a minimal replacement when the served file was HTML. Every change is listed at the top; nothing else is touched.

Inputs: domain (required).

fix_stale_cacheFix stale cache servingA
Read-onlyIdempotent
Inspect

Use after a scan reports STALE_CACHE_SERVED (visitors and crawlers get an old cached homepage). Fetches the homepage now and returns its Age, the cache layers that name themselves in the headers in purge order (innermost first), the HTML lifetime each response declares, a fresh-read comparison, and whether a purge clears the finding and keeps it cleared. Read-only; it does not purge anything. Give domain, or a report id (the page is still fetched now because cache state is live).

Inputs: id, domain.

list_verified_capabilitiesVerified capabilities of a siteA
Read-onlyIdempotent
Inspect

What a domain declares agents can do (OpenAPI operations, MCP tools, A2A skills, API catalog entries), each classed by safety (read_only_public up to financial), and which ones CrawlCheck actually called and confirmed - plus every mismatch between declaration and behaviour. Only read-only public actions are ever called; everything else says why it was not. From the latest scan on file.

Inputs: domain (required), verified_only.

machine_recordMachine record for a reportA
Read-onlyIdempotent
Inspect

The unified machine record for one observation: subject, grade, findings (in full), capabilities with verification, and the evidence roots, manifest and verifier links that let anyone check it offline. Give id (report, d1:, f1: or manifest sha256) or domain for the latest.

Inputs: id, domain.

mcp_precheckPre-release check for an MCP serverA
Read-onlyIdempotent
Inspect

For MCP server publishers, before releasing a new version. Give the server name (as listed in the MCP Registry) or its endpoint, the new tools/list result and the new serverInfo.version; with no tools the endpoint is read as it serves now. Returns, against CrawlCheck's last read of that server, what public change logs will record: tools added and removed, descriptions rewritten, input schemas widened, annotations changed (read-only to write and non-destructive to destructive flagged as escalations), whether the version moved, and verdicts such as version_bump_needed or looks_like_rug_pull_risk. Signed. No tool on the checked server is called.

Inputs: tools, version, endpoint, server_name.

mcp_serversMCP server reliability and identityA
Read-onlyIdempotent
Inspect

Which remote MCP servers the official MCP Registry lists on a domain, and what each endpoint actually answered when CrawlCheck sent the MCP handshake (initialized, auth_required, unreachable, tool count, tool-list drift). Use before selecting an MCP tool from that domain.

Inputs: domain (required).

preflightPreflight: ask before acting on a domainA
Read-onlyIdempotent
Inspect

Call BEFORE reading, citing, connecting to or transacting with a domain when you need a go/no-go answer for one action. Returns one signed decision (allow, warn, require_confirmation, block or unsupported) with every step that led there and the resolve answer it was made from. Pass your crawler token as agent so robots.txt is checked for you. A policy can only make the decision stricter. block means the action is not permitted; require_confirmation means a human must confirm before it proceeds. Use resolve_domain instead for the full signed record behind the decision, check_site_safety when the only question is malware or phishing listings, and best_source when choosing between several candidate sites. Free, no key, read-only.

Inputs: domain (required), agent, action, policy, +1 more.

public_countsPublic countsA
Read-onlyIdempotent
Inspect

Use when you need to quote one of CrawlCheck's figures about itself. Returns them read from the source: sites_measured, domains_in_corpus, crawler_visits (a rolling window), sections, sections_scored, findings_published, guides_published, with an at timestamp. No input; read-only. For dataset coverage by stratum use corpus_state.

Inputs: no named parameters.

registry_lookupRegistry lookupA
Read-onlyIdempotent
Inspect

Whether a domain is in CrawlCheck's registry and what it serves: llms.txt, agents.md, a media kit at /.well-known/media-kit.json, a reciprocity-tested entity graph, an AI access policy that names crawlers, and agent-callable surfaces. With no domain, returns the shelf counts across every measured site. A row exists because a fetch produced it; nothing here is self-reported and no payment moves a shelf.

Inputs: shelf, domain.

resolve_domainResolve a domain (signed record)A
Read-onlyIdempotent
Inspect

Use when you need the full signed record about one domain, not just a decision. Call before fetching from, citing, connecting to or transacting with a domain. Returns one signed answer: what the domain claims, what it actually serves each crawler identity, who controls it, what CrawlCheck's published rules decide for each action (a decision for read, cite, connect and transact, with its doubts and maximum age), what changed, and the evidence behind each part, marked declared, observed or not_measured with its time. There is no overall score. A domain never measured gets a just-in-time partial answer, read live within about 2 seconds, that names every part not read in time; nothing is guessed. CrawlCheck completes unmeasured parts later on its own crawl schedule. For a single allow/block decision on one action call preflight; for malware or phishing listings only, call check_site_safety.

Inputs: domain (required).

resolve_rightsRead a site's content rights and AI policyA
Read-onlyIdempotent
Inspect

What a site declares AI may do with a page, one answer per use (train, ground, summarize, cite, transact), merged from robots.txt read per agent, Content-Signal, ai.txt, TDM-Rep, robots meta, X-Robots-Tag and the terms page, with the signal that decided each answer and every conflict between signals. Signed. Not legal advice.

Inputs: domain (required), path, agent.

resolve_robotsRead a site's robots.txt rules per crawlerA
Read-onlyIdempotent
Inspect

Resolve every named answer engine, search index and training crawler against a domain's robots.txt the way a crawler does: most-specific group only, longest match, allow wins a tie. Shows when a Disallow under * does not apply to an agent with its own group.

Inputs: domain (required), path.

scan_domainScan a site for AI and crawler readinessAInspect

Scan a public domain as several crawler identities and return the graded record: grade, AI-visibility score, per-section scores, findings with evidence, and whether the origin refused CrawlCheck. One scan takes 5-15 seconds.

Inputs: domain (required).

telemetryCrawler telemetry on crawlcheck.ioA
Read-onlyIdempotent
Inspect

Verified-crawler traffic observed at crawlcheck.io itself: which named agents arrived, how many claims were confirmed against operator ranges, how many were forged.

Inputs: no named parameters.

verify_crawler_logVerify a crawler log (real vs forged bots)A
Read-onlyIdempotent
Inspect

Given raw web-server access-log lines, decide for each line that names a crawler (GPTBot, ClaudeBot, Googlebot, PerplexityBot...) whether the source IP falls inside that operator's published ranges. Returns verified, spoofed, or unverifiable when the operator publishes no ranges.

Inputs: lines (required).

video_channelCheck a video channelA
Read-onlyIdempotent
Inspect

Use to list a YouTube channel's uploads. Returns every upload on the channel, read 50 at a time and sorted by views. Give handle or, without one, the channel id. The per-check tally needs a Watch licence and is not offered here. Read-only. For the checks on one video use video_check.

Inputs: id, handle.

video_checkCheck a video's SEOA
Read-onlyIdempotent
Inspect

Use to audit one YouTube video for search and AI visibility. Returns twelve checks against the video anchor model (metadata, chapters, captions and storyboard where observed), each naming what it read and whether it passed. Read-only; results are cached for six hours unless fresh is true. For a channel's upload list use video_channel; for how a website embeds its videos use video_site.

Inputs: v (required), fresh.

video_siteCheck a site's video pagesA
Read-onlyIdempotent
Inspect

Use to check how a website carries its videos. Reads the homepage plus up to 60 sitemap pages for YouTube embeds, facades and page-builder widgets, VideoObject nodes and their required properties, and, given a channel handle, how many of the channel's videos the site carries. Read-only. For one video use video_check; for a channel's uploads use video_channel.

Inputs: domain (required), handle.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI agents and CI systems to run deterministic website QA audits over MCP, with every finding backed by evidence and every result hashed and signed. It checks HTTP, TLS, security headers, SEO, links, accessibility, and robots.txt, with configurable pass/fail verdicts.
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Checks a site for AI crawlers: which of 11 AI crawlers its robots.txt lets in, whether its llms.txt follows llmstxt.org, and what its home page says without scripts. Findings with evidence; no scores, no model calls, no telemetry.
      5
      49 npm
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      Audits AI-bot visibility: robots.txt per-bot for 22 AI user-agents (GPTBot/ClaudeBot/PerplexityBot/etc), Cloudflare flags, JSON-LD, sitemap, llms.txt, SPA shell, plus cross-model brand mentions via Perplexity + OpenRouter. 0-100 score. SSRF-guarded, spend-capped.
      4
      1
      MIT
    Try in Browser

    Glama MCP Gateway

    Add one secure layer between your agents and this server.