Skip to main content
Glama

SurfacedBy AI Visibility & GEO Audit

Check AI crawler access

surfacedby_check_ai_crawler_access
Read-onlyIdempotent

Check which AI crawlers a site allows, limits or blocks in robots.txt, what that means for each AI platform and the search index it uses, where to submit pages, whether a security check blocks an ordinary visit, homepage noindex, llms.txt and Content-Signal, with fixes ready to copy. Reads the site's public files and makes one ordinary page request, so firewall rules aimed at crawlers' own IP addresses do not show up here.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "domain": "<domain>"
    },
    "name": "surfacedby_check_ai_crawler_access"
  }
}

Input Schema

NameRequiredDescriptionDefault
domainYes

Output Schema

NameRequiredDescriptionDefault
fixesYes
budgetNo
domainYes
crawlersYes
homepageYes
llms_txtYes
platformsYes
next_stepsNo
robots_txtYes
limitationsNo
observed_atYes
evidence_kindNopublic_crawler_access_check
content_signalYes
evidence_providerNoSurfacedBy
citation_crawlers_blockedNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed3 schema fields changed
    • addedOutput schema / $defs / PlatformAccess / description
      Added value: +"Access to the index the platform mainly answers from, which is the\nplatform's main discovery lever, plus any secondary indexes."
    • addedOutput schema / $defs / PlatformAccess / properties / secondary_indexes
      Added value: +{
      +  "items": {
      +    "$ref": "#/$defs/SecondaryIndexAccess"
      +  },
      +  "title": "Secondary Indexes",
      +  "type": "array"
      +}
    • addedOutput schema / $defs / SecondaryIndexAccess
      Added value: +{
      +  "additionalProperties": false,
      +  "description": "An index the platform's operator says it also draws on. Its crawlers\nand submission paths help, but never decide the platform's own access.",
      +  "properties": {
      +    "access": {
      +      "enum": [
      +        "allowed",
      +        "restricted",
      +        "blocked",
      +        "unknown"
      +      ],
      +      "title": "Access",
      +      "type": "string"
      +    },
      +    "crawlers": {
      +      "items": {
      +        "type": "string"
      +      },
      +      "title": "Crawlers",
      +      "type": "array"
      +    },
      +    "note": {
      +      "anyOf": [
      +        {
      +          "type": "string"
      +        },
      +        {
      +          "type": "null"
      +        }
      +      ],
      +      "default": null,
      +      "title": "Note"
      +    },
      +    "search_index": {
      +      "title": "Search Index",
      +      "type": "string"
      +    },
      +    "submission": {
      +      "items": {
      +        "$ref": "#/$defs/SubmissionPath"
      +      },
      +      "title": "Submission",
      +      "type": "array"
      +    }
      +  },
      +  "required": [
      +    "search_index",
      +    "crawlers",
      +    "access"
      +  ],
      +  "title": "SecondaryIndexAccess",
      +  "type": "object"
      +}
  2. Added

TDQS

B3.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, openWorld and non-destructive, so the safety profile is covered; the description adds the genuinely useful operational fact that it reads public files and makes one ordinary page request. That reveals live network behavior and, importantly, explains that crawler-IP-targeted firewall blocks are invisible to this check — context an agent cannot get from the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The verb 'Check' is correctly front-loaded, but the core sentence is a long comma-chained list of checks that is dense to parse, and the trailing firewall caveat is appended as a fragment. Nothing is strictly wasted, but the content would land better as a short lead plus a compact enumeration.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need no explanation, and the description thoroughly covers what is inspected and a key blind spot. The only material omission is input format guidance for the required domain argument, which is the gap an agent would most likely trip on.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is one parameter ('domain') with 0% schema description coverage, and the description says nothing about it. It is unclear whether to pass a bare hostname, a full URL with scheme, or a path — a real ambiguity given the 2048-character maxLength hints at URL-like input. With no compensation for the coverage gap, the description leaves the only input underspecified.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource — checking which AI crawlers a site allows, limits or blocks in robots.txt — and enumerates the concrete artifacts inspected (robots.txt, homepage noindex, llms.txt, Content-Signal, security check). It is clearly a crawler-access diagnostic rather than a general audit, though it never explicitly distinguishes itself from siblings like plan_ai_visibility_audit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the scope (run this to learn crawler permissions and get copy-ready fixes), but there is no explicit 'use this when' statement and no named alternative for adjacent tasks such as the AI source landscape or page evidence tools. The one negative-scoping note — firewall rules aimed at crawlers' own IPs won't appear — is a caveat about results, not a routing guideline.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.