Skip to main content
Glama

netintel_robots_txt

Fetch and parse a domain's robots.txt to return crawl rules, sitemap URLs, crawl delays, and check if a path is allowed for a given bot.

Instructions

Fetch and parse a domain's robots.txt. Returns all crawl rules, sitemap URLs, crawl delay settings, and checks if a specific path is allowed for a given bot.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
pathNo
domainYes
user_agentNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv1.1.61

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses return contents (rules, sitemaps, crawl delay, path-allowed check), which is useful, but says nothing about failure modes (missing robots.txt, network errors) or behavior of optional parameters.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tightly packed sentences with no waste, front-loading the core action and enumerating the outputs efficiently.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description's enumeration of return values is helpful, but with zero parameter documentation and no annotations, the definition is adequate but leaves gaps around optional parameters and error behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description should compensate. It mentions domain implicitly ('a domain's robots.txt') and the concept of 'a specific path' and 'a given bot', but does not map these to the path and user_agent parameters explicitly, leaving the agent to infer parameter meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb (fetch and parse) and resource (robots.txt), plus enumerates the returned content. It doesn't differentiate from sibling netintel_sitemap_parser, which also deals with crawl-related resources.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use guidance or alternatives are offered. The description doesn't explain when an agent should reach for robots.txt parsing vs. other crawl-related siblings like sitemap_parser.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools