Skip to main content
Glama

Read a site's robots.txt rules per crawler

resolve_robots
Read-onlyIdempotent

Resolve every named answer engine, search index and training crawler against a domain's robots.txt the way a crawler does: most-specific group only, longest match, allow wins a tie. Shows when a Disallow under * does not apply to an agent with its own group.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "domain": "<domain>"
    },
    "name": "resolve_robots"
  }
}

Input Schema

NameRequiredDescriptionDefault
pathNoPath to test, default /
domainYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/openWorld/idempotent/non-destructive, so the bar is lower, and the description still adds real behavior: the matching algorithm applied (most-specific group only, longest match, allow wins a tie) and the 'Disallow under * does not apply to a grouped agent' case. It stops short of covering fetch behavior or return shape, so a 4 rather than 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two dense sentences, front-loaded with the core action and followed by the matching semantics; every clause carries information and nothing is padded or repeated.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description should describe what comes back (per-crawler verdicts? matched groups? the applied rule?), and it only gestures at that with 'Shows when a Disallow under * does not apply'. It is adequate to select the tool but thin on result shape for a tool that returns per-crawler analysis.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 50% — path is documented, domain is not — and the description never mentions either parameter or the default-path behavior. The phrasing 'a domain's robots.txt' implies the domain argument, but no meaning is added beyond the schema, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Resolve ... against a domain's robots.txt" gives a specific verb+resource, and the scope (every named answer engine, search index and training crawler) is concrete. It distinguishes itself conceptually from fix_robots (mutation) but never names a sibling or states the boundary explicitly, which keeps it out of 5 territory.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied — you call it when you need to know how a robots.txt treats each crawler — but there is no explicit when-to-use, when-not-to-use, or pointer to alternatives such as crawler_path or fix_robots. An agent must infer the selection criteria itself.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.