Skip to main content
Glama

check_robots

Checks a URL's robots.txt for rules affecting OAI-SearchBot, GPTBot, and ClaudeBot, so you can identify access restrictions for AI crawlers.

Instructions

检查 robots.txt 对 OAI-SearchBot、GPTBot、ClaudeBot 的规则。

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "check_robots"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYes完整的 http:// 或 https:// URL

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are supplied, so the description carries the full behavioral burden, yet it discloses almost nothing beyond the verb. It does not say whether it fetches robots.txt live over the network, what happens when the file is missing or returns 404, whether it follows redirects, or what the per-bot result looks like.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One short sentence with no filler, and the resource and crawler scope are front-loaded. It is efficient but borders on under-specification rather than tight completeness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

This is a network-dependent check with no annotations and no output schema, so the description should explain the result shape and edge cases. It leaves out what happens on missing robots.txt, on HTTP errors, or whether per-bot allow/deny verdicts are returned, which an agent needs to interpret the call.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single parameter is fully documented in the schema (完整的 http:// 或 https:// URL), so the baseline is 3. The description adds no meaning beyond it, such as whether the URL should be the site root or whether the robots.txt path is derived automatically.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb (检查) and resource (robots.txt), and narrows scope to three named crawlers (OAI-SearchBot, GPTBot, ClaudeBot). That distinguishes it from check_meta, check_sitemap, and run_audit by resource, though it never explicitly contrasts itself with them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the resource name: an agent auditing AI-crawler access would pick this. There is no explicit when-to-use statement, no mention of when to prefer run_audit or check_sitemap, and no stated prerequisites such as the site needing a reachable robots.txt.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools