Skip to main content
Glama
Sign In
fairquery-web

fairquery-web

Official
by fairquery
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

AI crawler access: robots.txt AI-bot rules + llms.txt (free)

web_ai_crawler_access
Read-onlyIdempotent

Check if AI crawlers like GPTBot and ClaudeBot can access a site by reading robots.txt, and detect llms.txt files. Returns per-bot allowed, partial, or blocked status.

Instructions

Report whether AI crawlers may access a site, from its robots.txt: per bot "allowed" (no Disallow), "partial" (some paths disallowed) or "blocked" (Disallow: /). Default bots: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Amazonbot. Also checks /llms.txt and /llms-full.txt (llmstxt.org): found, HTTP status, bytes, H1 title, link count (HTML soft-404s are not counted), and lists Sitemap: lines. WHEN TO USE: "can ChatGPT/Claude/Perplexity crawl this site?", GEO/AI-visibility checks, llms.txt adoption. Uses only the origin of the URL. Fetching: GETs from this server with User-Agent "FairqueryBot ... Fairquery-MCP", 15 s timeout per request, body capped (1.5 MB HTML), max 5 redirects, no JavaScript rendering; URLs resolving to private/loopback/link-local addresses are refused (SSRF guard). 3 requests per site. RETURNS: one "ai_crawler_access" object: origin, robots_txt_url, robots_txt_found, robots_txt_status, ai_bots{bot: status}, allowed_count, partial_count, blocked_count, note, sitemaps_declared[], llms_txt{url,found,status,bytes,title,links}, llms_full_txt{...}, checked_at. Errors (isError=true, JSON {error, message, retryable}): INVALID_URL, BLOCKED_PRIVATE_ADDRESS, DNS_FAILED, FETCH_FAILED, TIMEOUT, TOO_MANY_REDIRECTS. WHEN NOT TO USE: for full page SEO use web_seo_audit; for sitemap contents use web_sitemap_info; for many sites use web_batch_audit (paid). COST: free - runs locally in this MCP server (direct HTTP GETs), no APIFY_TOKEN, no Apify billing. PARAMETERS: url: only its origin (scheme + host + port) is used, so "nytimes.com/section/tech" and "nytimes.com" give the same result; www and non-www hosts are different origins with possibly different robots.txt files. bots replaces (does not extend) the default list - include the defaults you still want. Each token is matched like a crawler would: case-insensitively, against robots.txt User-agent groups whose name is a prefix of the token (so a "Googlebot" group also covers "Googlebot-Image"), falling back to the "*" group when no specific group exists; status is reported under the exact token you passed. If robots.txt is missing (or is an HTML soft-404) every bot is "allowed". bots does not affect the llms.txt / llms-full.txt checks. EXAMPLE: {"url":"nytimes.com","bots":["GPTBot","ClaudeBot"]} -> {ai_bots:{GPTBot:"allowed|partial|blocked", ClaudeBot:...}, blocked_count, llms_txt, ...}

Input Schema

NameRequiredDescriptionDefault
urlYesRequired. One public domain or http(s) URL (non-empty string), e.g. "vercel.com" or "https://www.python.org/about/". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused.
botsNoOptional, default = the 13 AI bots listed in the description. 1-50 robots.txt user-agent tokens (case-insensitive) to evaluate instead of the defaults. Example: ["GPTBot","ClaudeBot","Googlebot"]

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changedv1.2.4
    • changedInput schema / properties / bots / description
      Previous value: -"User-agent tokens to evaluate instead of the default AI-bot list, e.g. [\"GPTBot\",\"Googlebot\"]"New value: +"Optional, default = the 13 AI bots listed in the description. 1-50 robots.txt user-agent tokens (case-insensitive) to evaluate instead of the defaults. Example: [\"GPTBot\",\"ClaudeBot\",\"Googlebot\"]"
    • changedInput schema / properties / url / description
      Previous value: -"Required. One public domain or http(s) URL, e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; private/loopback hosts are refused."New value: +"Required. One public domain or http(s) URL (non-empty string), e.g. \"vercel.com\" or \"https://www.python.org/about/\". Bare domains are fetched over https; other schemes, credentials in the URL, dot-less hosts and private/loopback addresses are refused."
  2. Addedv1.2.1

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only cover the safety profile (readOnly, idempotent, non-destructive), so the description carries the rest and does so richly: User-Agent, 15s timeout, 1.5 MB body cap, max 5 redirects, no JS rendering, SSRF guard refusing private/loopback addresses, and 3 requests per site. Error codes and retryable flags are also disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Long but strictly sectioned (purpose, WHEN TO USE, Fetching, RETURNS, WHEN NOT TO USE, COST, PARAMETERS, EXAMPLE), with the core statement front-loaded. Each block is purposeful — the parameter details are needed given the matching subtleties, and the returns list compensates for the absent output schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers everything an agent needs: what it returns (full object shape), the error taxonomy, cost/model, fetching constraints, and origin-scoping behavior. With no output schema, the explicit RETURNS list and EXAMPLE fill the gap completely.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, yet the description still adds real meaning beyond the schema: only the URL origin is used, www vs non-www are distinct origins, and 'bots' replaces rather than extends the default list. It also explains the prefix-match/fallback matching semantics and that status is reported under the exact token passed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a precise verb and resource — 'Report whether AI crawlers may access a site, from its robots.txt' — then names the secondary capability (llms.txt / llms-full.txt checks). It distinguishes itself from siblings by explicitly routing to web_seo_audit, web_sitemap_info, and web_batch_audit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Has explicit WHEN TO USE ('can ChatGPT/Claude/Perplexity crawl this site?', GEO/AI-visibility checks, llms.txt adoption) and WHEN NOT TO USE with named alternatives for each case. Nothing about tool selection is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools