Skip to main content
Glama

crawl permission

crawl_permission
Read-only

May my agent fetch, index, train on or use this page? Reads robots.txt for your user agent (longest-match rules, crawl-delay), Content-Signal (search / ai-input / ai-train), X-Robots-Tag and meta robots (noai, noindex), TDM reservation and ai.txt, and lists which AI crawlers the site allows. Costs $0.002 USDC per call (x402).

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYes
user_agentNoe.g. GPTBot, ClaudeBot, MyAgent/1.0 (default *)

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint and openWorldHint, so safety is partly covered. The description adds genuinely new operational context: the exact upstream sources consulted, the longest-match crawl-delay semantics, and critically the $0.002 USDC per-call x402 cost, which an agent cannot infer from any structured field.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the decisive question, followed by one dense but well-organized sentence grouping sources, semantics and cost. The enumeration is long but each item is informative; only minor tightening is possible.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description must imply the return shape, and it does so partially by describing the sources checked and noting it 'lists which AI crawlers the site allows.' It does not state whether the verdict is returned as booleans per purpose, raw rule text, or a summary, which is the one remaining ambiguity for a permission-adjudication tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 50%: user_agent carries its own description with examples, while url is undocumented. The description reinforces that user_agent drives which robots rules apply ('for your user agent (longest-match rules, crawl-delay)'), adding real meaning, but it never clarifies url's expected form (full URL vs. domain, http/https, path sensitivity), so the coverage gap is only partially compensated.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening question frames a precise resource check — 'May my agent fetch, index, train on or use this page?' — and the enumeration of robots.txt, Content-Signal, X-Robots-Tag, meta robots, TDM and ai.txt pins down exactly what is inspected. This is clearly distinguishable from siblings like crawl, sitemap or domain_audit, which fetch rather than adjudicate permission.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The question format implies the tool is used before fetching or training on a page, and referencing 'your user agent' hints that UA selection matters. However, it never states when to call this versus crawl, x402_check or url_metadata, nor any exclusion or prerequisite. Usage is implied, not specified.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources