Skip to main content
Glama
CiteContext MCP Server

CiteContext MCP Server

generate_robots_txt

Create a robots.txt file that permits real-time AI citation search bots and controls offline AI model training scrapers for a specified domain.

Instructions

Generate an optimized robots.txt file that allows real-time AI citation search bots (Claude-SearchBot, OAI-SearchBot, Google-Extended) while controlling offline model training scrapers.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "domain": "<domain>"
    },
    "name": "generate_robots_txt"
  }
}

Input Schema

NameRequiredDescriptionDefault
domainYesDomain name (e.g., 'example.com')
allow_trainingNoAllow general AI model training web scrapers (default: false)
allow_citationsNoAllow real-time AI search bots to cite content (default: true)

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv1.0.0

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations supplied, the description carries the full behavioral burden. It usefully discloses the generated policy (which bot classes are allowed vs. blocked), but says nothing about what the call actually produces — returned text vs. a written/deployed file — or whether repeated calls are idempotent or overwrite existing config.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence with no filler, and the two-sided policy (allow citations / control training) is front-loaded so an agent grasps the tool's effect immediately.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple, has full schema coverage and no nested objects, so the description covers the essential inputs. However, with no annotations and no output schema, a competent agent still cannot tell what the tool hands back or whether invoking it mutates any live site configuration.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents all three parameters including defaults, setting the baseline at 3. The description adds modest meaning by naming the concrete citation bots (Claude-SearchBot, OAI-SearchBot, Google-Extended) and the training-scraper contrast, but it does not explain interactions between allow_training and allow_citations.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a concrete verb and artifact ('Generate an optimized robots.txt file') and states the policy it encodes, which is clearly distinct from file-generating siblings like generate_llms_txt or generate_schema_json. It stops short of explicitly contrasting with those siblings, so it is strong but not maximally differentiated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The policy framing ('allows real-time AI citation search bots while controlling offline model training scrapers') implies the scenario in which to call it, but there is no explicit when-to-use, when-not-to-use, or pointer to alternatives such as check_ai_bot_access or generate_llms_txt. Guidance is inferred rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.