Skip to main content
Glama

Honeyfield Marketing MCP

Website: Crawler check

site_crawler_check
Read-only

Dürfen Suchmaschinen und KI-Bots diese Website lesen? Prüft robots.txt (nach RFC 9309), Cloudflare-Signale und die Robots-Angaben der Startseite für 16 Bots in vier Gruppen: Suche (Googlebot, Bingbot, Applebot), KI-Suche (OAI-SearchBot, PerplexityBot, Claude-SearchBot), KI-Assistenten, die im Auftrag eines Nutzers abrufen (ChatGPT-User, Claude-User, Perplexity-User), und KI-Training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent). Anlass: Cloudflare sperrt seit 15.09.2026 KI-Training- und KI-Agent-Crawler per Default — Seiten können so für ChatGPT/Claude unsichtbar werden, ohne dass jemand etwas umgestellt hat. url: Domain oder URL (z. B. 'example.at'); ohne Angabe die Website des Workspace (Search-Console-Property bzw. WordPress-URL). paths: Pfade, für die je Bot geprüft wird (Standard ['/'], max. 10), z. B. ['/', '/blog/']. check_live: zusätzlich die Startseite einmal je Bot-User-Agent abrufen (Statuscode, Cloudflare-Abwehr). NUR EIN INDIZ: Cloudflare & Co. erkennen echte Bots an ihren IP-Bereichen; ein nachgemachter User-Agent von unserem Server kann abgewiesen werden, obwohl der echte Bot durchkommt (und umgekehrt). Nie als „Bot ist gesperrt“ melden, sondern als „unser Testabruf wurde abgewiesen“. Standard aus. Verlässlich sind robots.txt und Content Signals (erklärte Absicht des Betreibers). Ob Cloudflare zusätzlich auf Netzwerkebene blockt, ist von außen nicht sicher feststellbar — dafür steht ein Hinweis in notes. Rückgabe: origin; robots_txt (state ok|missing|unreachable, status, cloudflare_managed, content_signals der *-Gruppe z. B. {search, ai-train, ai-input} — gilt für Bots ohne eigene Gruppe, sitemaps, warnings); hosting.cloudflare; bots [{bot, operator, category, allowed {pfad: bool}, group, rule, content_signals der für den Bot geltenden Gruppe}]; summary je Gruppe (erlaubt | gesperrt | teilweise gesperrt); page_signals der Startseite (url, status, x_robots_tag, meta_robots); live_probe (Bots ohne Ergebnis: error budget_exceeded); notes = Befunde mit Handlungsempfehlung. missing (4xx) heißt laut RFC: alles erlaubt; unreachable (5xx/Timeout): alles gesperrt. workspace: Slug des Kunden-Workspace (nur nötig ohne url).

Input Schema

NameRequiredDescriptionDefault
urlNo
pathsNo
workspaceNo
check_liveNo

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnlyHint/openWorldHint annotations, the description discloses real behavioral limits: robots.txt and Content Signals are authoritative while network-level Cloudflare blocking is 'von außen nicht sicher feststellbar' (surfaced in notes), and a spoofed User-Agent probe can be rejected even when the real bot passes. It also gives interpretation semantics (missing/4xx = everything allowed, unreachable/5xx = everything blocked per RFC) that an agent could not infer on its own.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Structure is excellent and front-loaded: purpose question, method, motivation, then parameters, then caveats, then return shape. It is long, though – the exhaustive bot list and the multi-line Rückgabe block partly duplicate what the output schema already conveys, which keeps it from a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only, open-world diagnostic with 4 optional params and an existing output schema, the description covers everything an agent needs: what is checked, defaults and fallbacks, how to phrase results honestly (live probe caveat), and how to interpret missing vs unreachable. No material gap in calling it correctly remains.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description carries the full load and does: url accepts a domain or URL with a documented fallback to the workspace's Search Console property / WordPress URL; paths defaults to ['/'] with max 10 and an example; check_live is boolean, default off, with an explanation of what it actually performs; workspace is a slug needed only when url is omitted. All four parameters gain meaning beyond the bare type declarations.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening question plus 'Prüft robots.txt (nach RFC 9309), Cloudflare-Signale und die Robots-Angaben der Startseite für 16 Bots' names a concrete verb (prüft) and resource (robots.txt, Cloudflare signals, page robot directives) and enumerates exactly which bot groups are covered. This is unmistakably a crawler-permission audit, distinguishable from sc_url_inspection or bing_crawl_issues.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The 'Anlass' paragraph gives an explicit trigger context (Cloudflare blocking AI-training/agent crawlers by default since 15.09.2026, pages can silently become invisible to ChatGPT/Claude), which tells an agent when this tool is the right choice. It also warns that check_live is 'NUR EIN INDIZ' and should never be reported as 'Bot ist gesperrt'. It stops short of naming an alternative sibling tool for a narrower/broader check, so 4 rather than 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources