Skip to main content
Glama
Sign In
Site Audit MCP Server

Site Audit MCP Server

by steadfeed-dev
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Hybrid

AI crawler access audit

audit_ai_crawler_access
Idempotent

Check whether AI crawlers such as ChatGPT, Perplexity, Claude, Apple, and Google can access a website via robots.txt and noindex tags, then get per-crawler allow/block results and fixes.

Instructions

Check whether ChatGPT, Perplexity, Claude, Apple and Google AI crawlers can reach a website: robots.txt (RFC 9309), noindex tags and optional simulated live requests. Returns per-crawler allow/block results and fixes.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "urls": "<urls>"
    },
    "name": "audit_ai_crawler_access"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlsYesPublic http(s) URLs of the websites to check (1-25).
dryRunNoNot supported by this Actor; ignored.
extraPathsNoUp to 4 extra paths per site, e.g. /pricing.
liveChecksNoSend one polite request per crawler user-agent to the homepage and compare with a normal browser (default true).
maxCostUsdNoOptional hard cap in USD for this call. The underlying Actor stops when the cap is reached. Minimum 0.05 (the Apify per-run minimum). Default: a small amount computed from the number of URLs, never below 0.05.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.2.1

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, openWorldHint=true, idempotentHint=true and destructiveHint=false, so the safety bar is lower. The description adds real context beyond that: it discloses the specific checks performed and, importantly, that live requests can be simulated against the site, which explains why the call is not strictly read-only. It does not mention that this is a paid Actor run, though the maxCostUsd parameter covers that.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, both front-loaded: the scope and mechanisms come first, the return shape second. No filler, no restatement of the title, and every clause contributes either scope or output information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a five-parameter audit tool with no output schema, the description covers what is checked and what comes back, which is the key gap an agent would otherwise face. The only omission is that this runs a paid Actor with a cost cap, which the schema itself documents thoroughly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter including dryRun's unsupported status, extraPaths limits, liveChecks behavior and the maxCostUsd cap is already documented in the schema. The description only echoes liveChecks indirectly via 'optional simulated live requests' and adds no format or interaction detail. Baseline 3 is appropriate when the schema carries the semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a concrete verb ('Check whether ... can reach a website') and names the exact resource and mechanisms: robots.txt under RFC 9309, noindex tags, and optional simulated live requests. It also names the return payload (per-crawler allow/block results and fixes). It stops short of explicitly contrasting itself with siblings such as audit_schema_ai_readiness, though the crawler-access framing is naturally distinct.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: an agent can infer this is the tool for verifying AI crawler reachability rather than schema readiness or accessibility. There is no explicit 'use this when...' statement, no exclusions, and no named alternative for related audits. The phrase 'optional simulated live requests' hints at a lighter vs. deeper mode but doesn't say when to choose either.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools