Skip to main content
Glama

check_robots

Parse robots.txt; answer whether a user-agent may crawl a path; list sitemaps. (web-intel; $0.002/call in USDC via x402, 50 free/day).

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
hostYesTarget URL or hostname.
pathNo
user_agentNo

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, but the description discloses key behaviors: parsing, answering permission, listing sitemaps, and pricing/rate limits. Missing details on return format or error handling, but sufficient for typical use.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single efficient sentence plus parenthetical, with no fluff. All information is front-loaded and earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the main actions but lacks return format, edge cases (missing robots.txt), and parameter details. Given the lack of output schema and annotations, more completeness would be beneficial.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 3 parameters with only 33% coverage (host has a basic description). The description does not add details for path or user_agent, leaving their format and constraints unclear. With low schema coverage, the description should compensate more.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's actions: parse robots.txt, answer crawl permission, and list sitemaps. It uniquely identifies the resource and distinguishes from sibling web-intel tools like check_domain or dns_lookup.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use this tool (web intelligence, with pricing and free tier noted) but does not explicitly mention when not to use or compare to alternatives. Siblings are distinct enough (e.g., whois, dns), so usage is implicitly clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

B3.4/5.0
Disambiguation4/5

Most tools have clearly distinct purposes, but there are overlapping pairs: unfurl_url/get_oembed/find_favicon all provide URL metadata, and render_page/screenshot_page both render a page (render_page can also screenshot). Descriptions help disambiguate, so only minor confusion is likely.

Naming Consistency3/5

Tool names mix verb_noun patterns (check_domain, extract_content) with noun_verb patterns (dns_lookup, ssl_lookup, whois_lookup). Additionally, guard_inspect and guard_tx deviate from the typical verb-first style. The naming is readable but not consistent.

Tool Count3/5

With 19 tools, the server is at the heavier end of the scale. While it covers a broad web-intel toolkit, several tools feel out of place for a 'fetcher' (e.g., generate_qr, guard_tx, guard_inspect), making the set feel bloated and unfocused.

Completeness4/5

The core web-fetching and analysis surface is well covered: DNS, WHOIS, SSL, headers, content extraction, rendering, screenshots, feeds, embeds, redirects, and robots. Missing is a simple raw HTML fetcher, but extract_content and render_page likely cover most needs. The addition of security and utility tools doesn't fill obvious gaps.