Skip to main content
Glama

Agent Ready Website Audit

Draft robots.txt rules for AI crawlers

draft_robots_txt_for_ai_crawlers
Read-only

Drafts robots.txt groups for the AI crawlers Agent Ready tracks, and a Content-Signal line, from what the site owner wants: crawlers that index for AI search, crawlers that fetch a page because a user asked, and crawlers that collect training data, each allowed or blocked, with per-crawler overrides. It fetches nothing. Use it after check_robots_sitemap_llms_txt or check_ai_crawler_access shows crawlers blocked or allowed against the owner's wishes. result has robotsTxt (the lines to merge), contentSignal, decisions (each crawler with its role and choice) and unknownOverrides (tokens not recognised). Returns JSON: summary (one sentence) and result (the details below); resultId and reportUrl are null, as drafts are not saved. Errors come back as one plain sentence.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "draft_robots_txt_for_ai_crawlers"
  }
}

Input Schema

NameRequiredDescriptionDefault
searchNoWhether AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) may read the site: allow or block. Defaults to allow.
trainingNoWhether training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) may read it: allow or block. Defaults to allow.
overridesNoPer-crawler choices that win over the role choices, by robots.txt token, such as {"GPTBot": "block"}. Defaults to none.
userFetchNoWhether crawlers fetching for a user (ChatGPT-User, Claude-User) may read it: allow or block. Defaults to allow.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, non-destructive and non-open-world, but the description adds behavior not in the annotations: drafts are not persisted (resultId and reportUrl are null), no network fetch occurs, and errors return as a single plain sentence. The only gap is that it does not say whether the draft must later be applied by hand or by another tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and the routing condition are front-loaded, and the return-value sentence is compact. There is mild redundancy between 'result has robotsTxt, contentSignal, decisions and unknownOverrides' and the following 'Returns JSON: summary ... and result,' but the overlap is small and clarifying rather than wasteful.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description takes on return-value disclosure and does it fully: summary, result with its four fields, and the explicit null resultId/reportUrl. It also covers error format and the no-fetch/no-save behavior, so an agent has everything needed to call this with four optional parameters and interpret the response.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and each enum already documents its default, so the baseline is 3. The description earns an extra point by explaining the precedence model — role choices (search, userFetch, training) are overridden per crawler token via overrides — and by naming which crawlers fall into each role, which the schema only partly does.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first sentence names a specific verb (drafts) and specific resource (robots.txt groups plus a Content-Signal line) scoped to the crawler roles the product tracks. It immediately distinguishes itself from the check_* siblings by stating 'It fetches nothing,' and the enumerated role buckets (AI search, user fetch, training) make the output shape predictable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It names two concrete alternative tools and the exact condition that should route the agent here: 'after check_robots_sitemap_llms_txt or check_ai_crawler_access shows crawlers blocked or allowed against the owner's wishes.' It also rules out the wrong expectation by stating it fetches nothing, so an agent will not use it for inspection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources