Skip to main content
Glama

List the AI crawler registry

list_crawlers
Read-onlyIdempotent

Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
categoryNoOptional: only crawlers in this category.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
crawlersYes
generatedNo
categoriesYes
unverifiedNo
schema_versionNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the full safety profile (readOnly, idempotent, non-destructive, closed-world), so the description is only lightly additive. It does add genuinely useful context beyond the annotations: the data is static with no network access, and the registry includes unverified entries and notes, which warns the agent that not all rows are authoritative.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the purpose and then the returned fields; the trailing "Read-only, no network access. Same data as GET /api/crawlers." is compact and earns its place. The field enumeration is a little dense but stays within a few lines with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given full annotation coverage, a 100%-documented single parameter, and an output schema, the description covers everything an agent needs: what the tool returns, its non-networked read-only nature, and the presence of unverified entries. Nothing material is missing for a safe list operation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is a single optional parameter and schema description coverage is 100%, with the enum values spelled out in the schema and echoed in the description's category list. The description adds no syntax or filtering semantics beyond that, so the baseline 3 for schema-documented params applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Starts with a specific verb+resource ("Return RobotsGate's AI crawler registry") and enumerates what each entry contains, so the agent knows exactly what comes back. It is unmistakably different from sibling tools like check_site, generate_robots, and validate_robots, which operate on a specific site's robots.txt rather than listing registry data.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage (look up AI crawler tokens, operators, categories) and maps the tool to its REST equivalent (GET /api/crawlers), but it never states when to prefer it over the sibling tools or any conditions/exclusions. Usage is inferable rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources