Skip to main content
Glama

ai_crawler_audit

Audit which AI crawlers a site allows or blocks in robots.txt — sorted by CONSEQUENCE: blocking a VISIBILITY crawler (OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User…) removes the site from live AI answers, while blocking a TRAINING crawler (GPTBot, CCBot, Google-Extended…) only opts out of model training. Those need opposite decisions. Checks 25 known AI crawlers + whether /llms.txt exists, plus general robots.txt hygiene (Sitemap directives, whole-site Disallow foot-guns). Free (no LLM calls).

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
domainYesRoot domain to audit, e.g. 'yoursite.com'.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It discloses that the tool checks 25 known AI crawlers, evaluates /llms.txt existence, reviews robots.txt hygiene, sorts findings by consequence, and is 'Free (no LLM calls)'. It doesn't fully describe output format or side effects, but 'audit' plus the detailed scope gives strong transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three dense sentences lead with the tool's purpose, then explain the key consequence distinction, then enumerate checks and cost. Every sentence contributes operational value, and the structure front-loads the most important information without filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter audit tool with no output schema, the description covers purpose, scope, crawler categories, hygiene checks, and cost model. It doesn't specify the return shape or explicitly route to alternatives, but the detail provided is sufficient for an agent to invoke it correctly with minimal ambiguity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already documents the single domain parameter with an example ('yoursite.com'), so schema coverage is 100%. The description adds little parameter-specific meaning beyond the schema; it refers to 'site' generally but doesn't clarify format, normalization, or edge cases. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Audit which AI crawlers a site allows or blocks in robots.txt'. It names concrete sub-checks (25 AI crawlers, /llms.txt, Sitemap directives, whole-site Disallow foot-guns) and a distinguishing consequence-based sort, making it clearly distinct from generic siblings like site_audit or ai_visibility_report.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The intended use is unmistakable: run this when you need to audit robots.txt for AI crawler allow/block decisions and understand the visibility-vs-training consequence. It doesn't explicitly name alternative siblings or give when-not-to-use exclusions, but the context is clear enough for an agent to select it correctly.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4/5.0
Disambiguation4/5

Most tools target distinct actions, but backlinks_for_domain and top_linkers_for_domain overlap significantly since both return ranked linking domains with authority and link counts, differing mainly in scope. The descriptions do clarify one is a full payload and the other a quick skim, so an agent can usually choose correctly.

Naming Consistency3/5

All names are lowercase snake_case and readable, but conventions are mixed: several are noun phrases like site_audit and domain_authority_score, some are verb-first like find_link_prospects, and a couple are object-verb like schema_validate and indexnow_submit. The set is understandable but lacks a single consistent naming pattern.

Tool Count5/5

13 tools is appropriate for a backlink and SEO-focused server, fitting comfortably in the well-scoped range. Each area—backlink analysis, AI visibility, site auditing, and indexing—has dedicated tools without the set feeling bloated.

Completeness4/5

The backlink workflow is well covered: scoring, listing, overlapping linkers, prospect discovery, toxic-link flags, and live verification all exist. Minor gaps remain, such as no per-URL or anchor-level backlink data and no rank-tracking or keyword research, but the core stated purpose is served without major dead ends.