Skip to main content
Glama

Crawlora MCP

domain_robots

Read-only

Fetch the robots.txt policy text for a hostname. Provide a bare domain; the response includes HTTP status, final URL, content up to 256 KiB, and a truncation flag. The HTTPS request is pinned to a validated public IP.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "domain": "<domain>"
    },
    "name": "domain_robots"
  }
}

Input Schema

NameRequiredDescriptionDefault
domainYesBare public DNS hostname such as github.com; do not include a scheme, path, port, or IP address.

Output Schema

NameRequiredDescriptionDefault
dataYesThe tool result payload (shape varies per tool; see each tool's docs resource).

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnly/openWorld annotations, the description discloses concrete behavioral facts: the response carries HTTP status, final URL, content capped at 256 KiB, and a truncation flag, and the HTTPS request is pinned to a validated public IP. The IP pinning plus the 256 KiB cap are real operational constraints an agent needs. It does not say how redirects or missing robots.txt files are handled, which keeps it at 4.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the purpose, followed by the response payload and the security-relevant request behavior. No filler and no repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With annotations covering safety, a fully documented single parameter, and an output schema handling return shape, the description supplies what is missing: payload contents and the public-IP pinning constraint. Only edge-case behavior (404/no robots.txt, redirect handling) is left unstated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single parameter's schema already explains 'bare public DNS hostname, no scheme/path/port/IP'. The description merely restates 'Provide a bare domain', adding no format or validation detail beyond the schema, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource — fetching robots.txt policy text for a hostname — which is unambiguous and does not overlap with any sibling such as domain_dns_resolve or domain_rdap_domain. It stops short of explicitly contrasting itself with domain_site or the other domain_* tools, so full sibling differentiation is left implicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the resource: an agent wanting a site's crawl policy would pick this. However, there is no explicit when-to-use statement, no exclusions, and no pointer to alternatives (e.g., domain_site for fetching a page, domain_dns_resolve for DNS). The agent must infer the boundary from the name.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources