Skip to main content
Glama

BACKLNK

AI crawler robots.txt rules

ai_crawler_policy
Read-only

Builds robots.txt lines that allow or block AI crawlers from BACKLNK's reviewed crawler catalog, with what each line does and warnings. Use a preset (block-training: Block AI training, stay in AI search; block-ai: Block AI crawlers, keep search engines; allow-all: Allow every crawler) or decide per purpose and per crawler token.

Input Schema

NameRequiredDescriptionDefault
presetNo
crawlersNoPer-crawler overrides by token, e.g. { "GPTBot": "allow" }.
purposesNoDecision per purpose; purposes not named stay allowed when no preset is given.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnlyHint=true, openWorldHint=false), and the description matches that by describing a builder that emits text lines with accompanying explanations and warnings rather than anything that mutates state. It adds real value beyond the annotations by disclosing that the output includes per-line explanations and warnings; the main undisclosed behavior is precedence between an explicit preset and the crawlers/purposes overrides.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences: the first front-loads what the tool produces, the second covers how to drive it via presets or granular decisions. Dense but every clause carries information; only minor friction from nesting three preset definitions inside one sentence.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a no-output-schema tool with nested objects (purposes, crawlers), the description does say the result is robots.txt lines with explanations and warnings, which is the key return-value hint. However, it never clarifies how a preset interacts with explicit per-purpose or per-crawler overrides, nor whether the generated lines are merely returned or applied anywhere — the one gap an agent could actually stumble on.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 67% and the three preset enum values have no descriptions in the schema, so the description's gloss — 'block-training: Block AI training, stay in AI search', 'block-ai: Block AI crawlers, keep search engines', 'allow-all: Allow every crawler' — supplies meaning the schema cannot. It also names the two override axes ('per purpose and per crawler token'), though it does not explain how those combine with a preset.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Builds robots.txt lines that allow or block AI crawlers from BACKLNK's reviewed crawler catalog.' It also discloses what the output contains ('what each line does and warnings'), which no sibling tool (get_report, list_checks, run_checks, usage, verify_fixes) offers, so an agent can route to it unambiguously.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly presents the two invocation modes — pick a named preset for a standard stance, or 'decide per purpose and per crawler token' for fine-grained control — and each preset carries its own intent label (block-training, block-ai, allow-all). It stops short of saying when this tool is *not* appropriate or how preset and manual inputs combine, so it is clear context without exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.