Skip to main content
Glama

ai_crawler_policy_check

Check a site's robots.txt for explicit directives targeting known AI crawlers (GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Bytespider, Amazonbot, and others used for LLM training or AI search/answer products), and check for an llms.txt file. Useful for publishers deciding whether their content policy toward AI crawlers matches their intent, or for auditing a competitor's stance.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesAny URL on the site to check (origin is derived from it)

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden of disclosure. It makes the read-only nature clear via 'Check' and specifies exactly what is inspected: robots.txt directives for a named set of AI crawlers and the presence of llms.txt. It does not discuss rate limits or errors, but nothing about the behavior is hidden or misleading.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no filler: the first front-loads what is checked, and the second explains who benefits. The crawler-name list is long but earned, because it removes ambiguity about which bots are in scope.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter, low-complexity tool, the description covers the target resource, the exact crawlers of interest, and llms.txt; the schema covers the input contract. It could mention the response shape or behavior when robots.txt is absent, but those are minor gaps for a check-style tool with no output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single parameter is already fully described in the schema with coverage at 100%, including the fact that any URL on the site works and the origin is derived. The description reinforces this by talking about 'a site's robots.txt', but it adds no new parameter semantics beyond what the schema provides, so it stays at the baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Check a site's robots.txt for explicit directives targeting known AI crawlers', then adds llms.txt as a second target. It also enumerates the exact crawler names and differentiates itself from generic siblings like check_robots_sitemap by scoping to AI crawler policy.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear, concrete use cases: publishers checking whether their AI-crawler content policy matches intent, and auditing a competitor's stance. It does not explicitly mention alternatives or say when not to use this tool, so it misses the highest bar for routing an agent away from siblings like check_robots_sitemap.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A3.6/5.0
Disambiguation3/5

Many tools audit overlapping site signals (seo_audit vs structured_data_extract vs tech_stack_fingerprint; page_performance_check vs pagespeed_insights; ssl_cert_check vs ssl_labs_grade; broken_link_check vs sitemap_url_validator), so an agent could initially pick the wrong one. Descriptions usually clarify the distinction, but the boundaries are not always obvious.

Naming Consistency3/5

Most names follow a snake_case target+operation pattern (ssl_cert_check, email_deliverability_check), but check_open_ports and check_robots_sitemap reverse the order, and the action suffixes vary widely (check, audit, validate, lookup, extract, grade, report, insights). Still readable, but not a single predictable convention.

Tool Count2/5

At 29 tools, the server is above the 25-tool threshold and feels like an undifferentiated grab bag of single-purpose audits rather than a tightly scoped toolkit. Many checks could be consolidated (e.g. the separate SSL and performance tools, or domain_report versus its component checks).

Completeness4/5

For a web/domain/email/security diagnostics toolbelt, the coverage is unusually broad: DNS, TLS, email, SEO, structured data, vulnerabilities, ports, redirects, and more are all represented. Minor gaps exist (no generic HTTP request/debug tool, no zone-transfer or full WHOIS history), but agents can accomplish most diagnostic workflows without hitting dead ends.

Resources