Preferly

Check AI crawler access

check_ai_crawler_access

Read-onlyIdempotent

Use this when the user wants to know whether AI crawlers and AI tools can access a specific website. It reads the site's robots.txt and reports, for 13 AI crawlers (including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended), whether each is allowed or blocked and which rule decides it. It also reports whether the homepage loads for an automated visitor or shows a bot check (such as Cloudflare, SiteGround or AWS WAF), whether the site publishes /llms.txt, and which schema.org types the homepage declares. Input: one public website address. It fetches only robots.txt, the homepage and /llms.txt, identifying itself as PreferlyBot, and changes nothing on the site. A result for the same domain may come from a check run in the last 10 minutes. It does not report what AI tools say about the business, search rankings, or pages other than the homepage.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "website": "<website>"
    },
    "name": "check_ai_crawler_access"
  }
}

Input Schema

Name	Required	Description	Default
`website`	Yes	The public website to check, like example.com or https://www.example.com. One site per call.

Output Schema

Name	Required	Description
`domain`	Yes
`llmsTxt`	Yes
`summary`	Yes
`verdict`	Yes
`crawlers`	Yes
`homepage`	Yes
`learnMore`	Yes
`nextSteps`	Yes
`robotsTxt`	Yes
`notChecked`	Yes
`structuredData`	Yes	schema.org types declared on the homepage

Schema Changelog

Changes observed during successful MCP inspections.

First observedNov 14, 2026

TDQS

A4.1/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnly, idempotent, non-destructive), the description discloses the exact fetch surface (robots.txt, homepage, /llms.txt only), the user agent it presents (PreferlyBot), that nothing on the site is changed, and that results may be served from a check up to 10 minutes old. That last detail in particular is behavior an agent could not derive from annotations or schema and matters for deciding whether to re-run.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is front-loaded with the trigger condition, then moves to outputs, mechanism, and finally exclusions, so the important material comes first. It is dense but not padded; the only cost is that the middle sentence packs five distinct output facts into one long sentence.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only inspection tool with a rich output schema, the description covers what it inspects, what it does not report, side effects, user agent, and caching. Nothing an agent needs in order to decide to call it, or to set expectations about the result, is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single parameter's description already gives format examples and a one-site constraint. The description restates this ("Input: one public website address", "one site per call") without adding format, normalization, or error semantics — correct baseline for a fully documented single parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence gives a specific verb and resource (check whether AI crawlers can access a website) and the following sentences enumerate exactly what is inspected: robots.txt, 13 named AI crawlers, bot-check detection, /llms.txt, and schema.org types. It is very concrete, but it never names the sibling tools it overlaps with (check_llms_txt, check_business_schema, quick_ai_visibility_check), so the agent must infer the boundary itself.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"Use this when the user wants to know whether AI crawlers and AI tools can access a specific website" is an explicit use condition, and the closing sentence adds negative scoping by ruling out AI-tool mentions, rankings, and non-homepage pages. It falls short of a full 5 because it never names an alternative sibling to use for those adjacent questions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Preferly tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.