Skip to main content
Glama

OnPage.dev

by dev.onpage
OverviewDiscussionsAdmin

robots.txt

robots_txt
Read-onlyIdempotent

robots.txt for search and AI crawlers. Pick with mode: test = Which crawlers may fetch a URL, and the rule that decides. Needs url. ai_policy = Rules per AI crawler: allow AI search, block training. Needs url, policy.

Input Schema

NameRequiredDescriptionDefault
urlNoFull address of the page, for example https://example.com/pricing.
modeNoDefault test.
detailNobrief = short summary.
policyNoWhich AI crawlers to allow.
memory_keyNosite_memory key, records this call.
robots_txtNoOptional robots.txt content to test instead of the live file.
user_agentsNoCrawler names, for example Googlebot, GPTBot, ClaudeBot, PerplexityBot.

Output Schema

NameRequiredDescriptionDefault

No arguments

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • removedOutput schema / description
      Removed value: -"The result as structured data; fields depend on mode."
  2. Changed2 schema fields changed
    • changedInput schema / properties / detail / description
      Previous value: -"brief returns a short summary, default full."New value: +"brief = short summary."
    • changedInput schema / properties / memory_key / description
      Previous value: -"Optional site_memory key, to record this call in the site memory."New value: +"site_memory key, records this call."
  3. Changed1 schema field changed
    • addedInput schema / properties / detail
      Added value: +{
+  "description": "brief returns a short summary, default full.",
+  "enum": [
+    "full",
+    "brief"
+  ],
+  "type": "string"
+}
  4. Changed1 schema field changed
    • addedInput schema / properties / memory_key
      Added value: +{
+  "description": "Optional site_memory key, to record this call in the site memory.",
+  "maxLength": 80,
+  "type": "string"
+}
  5. Added

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint and openWorldHint, so the safety profile is covered. The description adds the useful behavioral fact that required inputs vary by mode, but says nothing about the robots_txt override, user_agents, or memory_key side effects, leaving that to the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three compact sentences, front-loaded with the resource and then the mode decision; every clause carries information an agent needs to select a mode.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need not be described, and the mode split plus prerequisite params are covered. Coverage of the auxiliary inputs (detail, user_agents, robots_txt, memory_key) is left entirely to the schema, which is acceptable given 100% schema coverage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents url, mode, detail, policy, user_agents and the robots_txt override. The description only reinforces which params are needed per mode, which is marginal added meaning over the schema baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states the resource (robots.txt) and splits it into two concrete capabilities, test and ai_policy, each with a one-line outcome. An agent can tell the two modes apart, though it gives no signal about how this differs from the sibling llms_txt tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly says how to pick: 'Pick with mode: test = ...' and 'ai_policy = ...', and names the prerequisites for each ('Needs url', 'Needs url, policy'). There is no when-not guidance or sibling alternative routing, which keeps it short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All OnPage.dev toolsInspect
Try in Browser
Context Every Agent Can ShareAgents share what they learn, hand off work, and keep it when you switch models. Start with 5 GB free.fulcradynamics.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources