Skip to main content
Glama

Agent Data Tools: Full list (every tool as its own definition)

Robots.txt check: may this crawler fetch this URL?

robots
Read-only

Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap. Example: url=https://www.google.com/search. Price $0.001/call via x402 USDC or a card key (/credits), first 25 free; failed calls free. Reads or computes only and changes nothing; each call is charged, so repeats are not free.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "robots"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesThe page URL to check (its site's /robots.txt is read)
agentNoCrawler user agent or product token (default *)

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnlyHint, destructiveHint false, openWorldHint), but the description adds real context: the deciding rule, group and rule count are returned, a 4xx robots.txt allows everything, and calls are charged so repeats are not free — which explains the non-idempotent hint. It could still say more about failure modes (5xx, timeouts) or rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the purpose, then the matching rules, the return contents, the edge case, the sibling pointer, an example and pricing. Dense rather than padded, though the pricing clause is long enough to slightly dilute the core message.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, but the description enumerates the return fields (deciding rule, group, rule count, sitemaps), and it covers the key non-obvious case (missing robots.txt allows everything). For a two-parameter read tool, nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with both url and agent documented, including agent's default of '*'. The description adds the interpretation rules and an example URL but no new per-parameter semantics beyond what the schema states, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Read a site's robots.txt and say whether a crawler may fetch a URL', plus the exact rule set (RFC 9309 as Google applies it). It separates itself from the sibling sitemap tool by noting that sitemap handles the sitemap URLs themselves.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit when: 'Use before scraping or crawling a site politely', and names the alternative for the adjacent need ('for the sitemap URLs themselves use sitemap'). The agent does not have to infer the routing decision.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.