Skip to main content
Glama

Crypto, web, finance and app data tools

Robots.txt & Sitemap

robots_sitemap

A site's robots.txt rules per user agent, whether AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot) are blocked, and the URLs listed in its sitemap. Use before crawling a site or for SEO audits.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "domain": "<domain>"
    },
    "name": "robots_sitemap"
  }
}

Input Schema

NameRequiredDescriptionDefault
limitNo
domainYes

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it does disclose the returned content (rules, crawler blocks, sitemap URLs). However, it says nothing about how sitemaps are discovered, what happens when robots.txt is absent or unreachable, or any rate limiting, leaving real behavioral gaps for a fetch-based tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence, front-loaded with what is returned and closed with the usage trigger. No filler and nothing buried.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values needn't be explained and the description is adequate on that front. But with zero annotation coverage and an undocumented limit parameter, the definition leaves an agent guessing about input formatting and result capping.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% for both parameters, so the description must compensate. It implies a domain input via "a site's" but never states the expected format (bare domain vs. URL), and the limit parameter (default 100) is entirely unaddressed — an agent cannot tell whether it caps sitemap URLs, robots rules, or something else.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names three concrete resources it returns: robots.txt rules per user agent, AI crawler blocking status for named bots, and sitemap URLs. That is a specific verb-plus-resource statement an agent can match to a task without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"Use before crawling a site or for SEO audits" gives clear usage contexts and implicitly separates it from read_web_page or url_metadata. It stops short of naming an explicit alternative or stating when not to use it, so it falls short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources