Skip to main content
Glama

AI crawler access check

check_ai_access
Read-onlyIdempotent

Checks whether AI crawlers can read one public web page and whether the page is set up to be cited. For the exact URL path it evaluates the domain's robots.txt for 8 AI robots (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot, CCBot, Applebot-Extended), each as allowed, blocked or no-rule (no rule for that robot or *, so allowed by default), or unknown when robots.txt could not be read; it checks for a valid /llms.txt (title, link count); and it reads the page's JSON-LD for citable types (Article, BlogPosting, FAQPage, HowTo…) and the fields that are missing (author Person with url, datePublished, dateModified, Organization.sameAs). Each part has a verdict (pass, warn, fail, info). Works without signing in and without a property: it fetches three public URLs of the site (the page, /robots.txt, /llms.txt) and stores nothing. Only public http/https addresses on the standard ports are accepted; private, local and internal addresses return invalid_argument. Use it to answer "can GPTBot / ClaudeBot / PerplexityBot read this page", "does this site have llms.txt" or "does this page have the structured data to be cited". It does not measure traffic or rankings. / Checa, sem login, se os robôs de IA podem ler uma página pública (robots.txt no caminho exato, llms.txt e JSON-LD de citação); não mede tráfego nem posição. Docs: https://organiqgeo.com/docs#check_ai_access

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "check_ai_access"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesPublic page address, e.g. example.com/blog/post or https://example.com/blog/post. The robots.txt rules are evaluated for this exact path.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the supplied readOnly/openWorld/idempotent annotations: it discloses that three public URLs are fetched, that nothing is stored, that private/local/internal addresses return invalid_argument, and that robots.txt being unreadable yields an 'unknown' verdict rather than a failure. It also explains the default-allow semantics of 'no-rule', which an agent would otherwise misread.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the one-sentence purpose, then the mechanics, then the constraint and use cases — a sensible triage order. The second sentence is long but every clause carries distinct information (robot list, verdict vocabulary, llms.txt fields, JSON-LD citable types and missing-field checks); the appended Portuguese restatement and docs link are the only low-value padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the full burden of describing return values, and it does: per-robot allowed/blocked/no-rule/unknown status, llms.txt title and link count, JSON-LD citable types and missing fields, and a pass/warn/fail/info verdict per part. Nothing an agent needs to call or interpret this tool is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single url parameter is already documented, so the baseline would be 3; the description nevertheless adds real constraints the schema lacks — only public http/https on standard ports are accepted, and robots rules apply to the exact path given. That materially changes how an agent should construct the argument.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — checks whether AI crawlers can read one public page and whether the page is set up to be cited — and enumerates exactly which crawlers and which artifacts (robots.txt, /llms.txt, JSON-LD) are examined. An agent can tell this apart from traffic/ranking siblings without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete when-to-use triggers in the form of example questions ("can GPTBot / ClaudeBot / PerplexityBot read this page", "does this site have llms.txt") plus an explicit exclusion ("does not measure traffic or rankings"), and notes it needs no sign-in or property. It stops short of naming a specific sibling to use for traffic/ranking questions, so the alternative is implied rather than named.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.