Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Scored across 4 tools
check_site and validate_robots both report which AI crawlers are allowed/blocked for a path, creating some surface overlap, but the descriptions make the key distinction clear (check_site fetches a live remote file; validate_robots checks supplied text locally with no network). generate_robots and list_crawlers are clearly distinct.
All four tools follow a strict verb_noun snake_case pattern: check_site, generate_robots, list_crawlers, validate_robots. The convention is consistent and predictable.
Four tools is reasonable and each covers a distinct capability (fetch/check, generate, registry lookup, validate). It is slightly lean, but the domain is narrow enough that this scope is appropriate.
The set covers the core robots.txt lifecycle: inspect a live site, build a file, validate text, and consult the crawler registry. No write/deploy operation exists, but that appears intentional given the read-only design, leaving only minor gaps.
Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Site URL or bare hostname, e.g. example.com. Only the scheme and host are used. | |
| path | No | URL path to test, starting with "/" (default "/"). |
| Name | Required | Description |
|---|---|---|
| url | Yes | |
| path | No | |
| cache | No | |
| final_url | Yes | |
| redirects | No | |
| fetched_at | No | |
| ai_crawlers | No | Per-crawler results, or null when robots.txt was unreachable |
| http_status | Yes | |
| robots_found | Yes | |
| interpretation | Yes | |
| personal_agents | No | Object, or null when robots.txt was unreachable |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only/idempotent/open-world, and the description adds substantial extra context: a 512,000-byte fetch limit, a 5-minute result cache, refusal of private/loopback/internal addresses, default-port-only restriction, a single GET with a specific user-agent, and the fact that output includes validation errors and warnings. These are exactly the operational traits annotations cannot express.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the verb and the resource, followed by a constraints parenthetical and two short supporting sentences. Dense but every clause carries information; the long parenthetical is the only mild readability cost.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained, and the description still covers scope, fetch limits, caching, security refusals and the engine identity. Nothing an agent needs to invoke this correctly appears to be missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3; the description goes slightly further by framing the result as being 'for one path', clarifying that the path parameter drives which allow/block determination is returned. It adds a little meaning over the schema but no format or syntax detail beyond it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Fetch a site's /robots.txt ... report ... which registry AI crawlers are allowed or blocked'), which is concrete and agent-legible. It implicitly separates itself from validate_robots by emphasizing a live fetch, but never names a sibling or explicitly contrasts the check/validate/generate split among the four tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: the agent can infer this is the tool for inspecting a live site rather than generating or validating a local file. There is no explicit when-to-use/when-not guidance and no sibling alternative named, so selection among check_site, validate_robots and list_crawlers is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
| Name | Required | Description | Default |
|---|---|---|---|
| date | No | Date for the header comment (YYYY-MM-DD). Default: today (UTC). | |
| agents | No | Per-token overrides, e.g. {"GPTBot": "block"}. Tokens: letters, digits, "_", "." and "-", up to 64 characters (not "*"). At most 100. | |
| default | No | Policy for "User-agent: *" (default allow). | |
| sitemap | No | A single sitemap URL (alternative to sitemaps). | |
| sitemaps | No | Absolute http(s) sitemap URLs. At most 20. | |
| categories | No | Action per crawler category. Omitted categories get no group of their own. | |
| allow_paths | No | Paths to allow explicitly. Each path starts with "/" or "*" and has no whitespace or "#". At most 100. | |
| custom_rules | No | Extra robots.txt lines appended after validation (max 20000 bytes). | |
| disallow_paths | No | Paths to disallow for "*" and for allowed AI groups. Each path starts with "/" or "*" and has no whitespace or "#". At most 100. |
| Name | Required | Description |
|---|---|---|
| notes | Yes | |
| summary | Yes | |
| robots_txt | Yes | |
| personal_agents_note | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint and destructiveHint=false, so the safety profile is covered. The description still adds value by describing the produced output ('the file text, a per-crawler summary and notes') and asserting no side effects, though 'Read-only: it only returns text and changes nothing' largely restates the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action and packed with the parameter surface in one dense sentence, then the output and no-side-effect notes. The read-only sentence is partly redundant with annotations, but overall there is little waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With a rich 9-parameter schema, full description coverage, an output schema, and comprehensive annotations, the description supplies enough for correct invocation. It is slightly short on tool-selection guidance against the validate/check siblings, but nothing needed to call the tool correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter is already documented in the schema with constraints (path syntax, token pattern, max counts). The description names the parameter groups at a high level but adds no syntax, format or default detail beyond what the schema provides, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and artifact ('Build a robots.txt') and enumerates the exact input surface (per-category choices, per-token overrides, default policy, paths, sitemaps, custom rules). This clearly distinguishes it from siblings like validate_robots and check_site, which assess rather than generate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the generation framing, and the note 'Same engine as POST /api/generate' gives an equivalence hint. However, there is no explicit when-to-use vs when-not guidance, and no routing against validate_robots (validate an existing file) or check_site, which an agent choosing among four tools would want.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.
| Name | Required | Description | Default |
|---|---|---|---|
| category | No | Optional: only crawlers in this category. |
| Name | Required | Description |
|---|---|---|
| crawlers | Yes | |
| generated | No | |
| categories | Yes | |
| unverified | No | |
| schema_version | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare the full safety profile (readOnly, idempotent, non-destructive, closed-world), so the description is only lightly additive. It does add genuinely useful context beyond the annotations: the data is static with no network access, and the registry includes unverified entries and notes, which warns the agent that not all rows are authoritative.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the purpose and then the returned fields; the trailing "Read-only, no network access. Same data as GET /api/crawlers." is compact and earns its place. The field enumeration is a little dense but stays within a few lines with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given full annotation coverage, a 100%-documented single parameter, and an output schema, the description covers everything an agent needs: what the tool returns, its non-networked read-only nature, and the presence of unverified entries. Nothing material is missing for a safe list operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is a single optional parameter and schema description coverage is 100%, with the enum values spelled out in the schema and echoed in the description's category list. The description adds no syntax or filtering semantics beyond that, so the baseline 3 for schema-documented params applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Starts with a specific verb+resource ("Return RobotsGate's AI crawler registry") and enumerates what each entry contains, so the agent knows exactly what comes back. It is unmistakably different from sibling tools like check_site, generate_robots, and validate_robots, which operate on a specific site's robots.txt rather than listing registry data.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage (look up AI crawler tokens, operators, categories) and maps the tool to its REST equivalent (GET /api/crawlers), but it never states when to prefer it over the sibling tools or any conditions/exclusions. Usage is inferable rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | URL path to test, starting with "/" (default "/"). | |
| robots_txt | Yes | The robots.txt file content. |
| Name | Required | Description |
|---|---|---|
| info | No | |
| path | No | |
| stats | No | |
| valid | Yes | |
| errors | Yes | |
| warnings | Yes | |
| ai_crawlers | Yes | |
| diagnostics | No | |
| personal_agents | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare read-only, closed-world, idempotent behavior, so the bar is lower, yet the description still adds non-obvious context: no network access, same engine as POST /api/validate, and the 50,000-character / 64 KB JSON-RPC payload ceiling. These operational constraints are not derivable from the annotations alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three tightly packed sentences, front-loaded with the core purpose, followed by safety posture and the size/alternative constraint. No sentence is redundant and nothing important is buried.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the description needn't explain return values, and it still covers limits, alternatives, and engine parity. Nothing an agent needs to call this correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents both parameters and the 50,000 maxLength. The description nonetheless explains the rationale behind the limit (the whole JSON-RPC request must fit in 64 KB) and clarifies that 'path' tests exactly one URL path, adding meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Validate robots.txt text against RFC 9309') plus the exact outputs (errors, warnings, per-path crawler allow/block verdicts). This is clearly distinguishable from the sibling tools generate_robots and list_crawlers, which produce rather than validate content.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit conditional alternative: 'for larger files use POST /api/validate or check_site,' naming a sibling tool by name. It does not spell out when-not to use validation at all, but the escape hatch for oversized inputs is clear and actionable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
check_sitegenerate_robotslist_crawlersvalidate_robotsCheck which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
Free check whether a website is technically open to AI assistants (AI crawlers, llms.txt, JSON-LD).
Scan and fix your site's AI discoverability: crawler access, llms.txt, JSON-LD. Free.
Free AI-readiness audit of any URL: AI crawler rules, JS-free text, JSON-LD, llms.txt. Tool catalog.
Add one secure layer between your agents and this server.
