Skip to main content
Glama
Agency MCP Server

Agency MCP Server

Crawler access

check_crawler_access
Read-only

Reads a site's robots.txt to confirm whether named crawler user-agents like GPTBot or Googlebot may fetch a URL, and reports which rule decides plus llms.txt detection.

Instructions

Reads the host's robots.txt (RFC 9309) and reports whether each named crawler user-agent may fetch the URL and which rule decides; defaults: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot. Also detects llms.txt.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "client": "<client>",
      "url": "<url>"
    },
    "name": "check_crawler_access"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesAbsolute URL inside the client's allowlist.
agentsNoExtra user-agent tokens to check.
clientYesClient id from the server's config (dental, cafe, plumber).

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover readOnly/openWorld/destructive, and the description adds valuable behavior: it reads live robots.txt per RFC 9309, reports which rule decided the outcome (not just a boolean), and separately detects llms.txt. It does not cover failure behavior when robots.txt is absent or the fetch fails, which matters for an open-world network tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences: the core operation is front-loaded, and defaults and the secondary llms.txt capability follow without filler. Every clause carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description does explain the shape of the result (per-agent verdict plus deciding rule), though the llms.txt detection output is left vague. Enough to call the tool correctly, with a minor gap on return format.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real value by naming the default agent set (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Googlebot) that the schema leaves unstated, clarifying that the agents parameter is purely additive.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States specific verbs (reads robots.txt, reports per-agent permission and deciding rule) plus a secondary capability (llms.txt detection). This is clearly distinct from sibling tools like crawl_site or audit_page, which crawl or audit pages rather than evaluate crawler permissions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the purpose — check whether AI crawlers may fetch a URL — but there is no explicit when-to-use vs when-not guidance and no named alternative. An agent must infer the routing decision itself.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.