Check which AI bots a site's robots.txt allows; read-only facts about Parallel World
Scored across 2 tools
The two tools do different things (robots.txt bot policy vs. the operator's own published facts), but both are read-only website inspections with no shared framing, and get_site_info's implicit scope (this site only, in Japanese) versus check_ai_bot_access's arbitrary-domain scope is easy to mix up. An agent could plausibly reach for one when it needs the other.
Both names are snake_case verb_noun forms (check_ai_bot_access, get_site_info), which is consistent and readable. The only minor deviation is that 'check' vs 'get' imply different return shapes without that being signaled by naming.
Two tools is thin for a server, though the narrow robots.txt/AI-bot-checking purpose makes a small surface defensible. There is no tool for listing the supported bot registry or checking a specific path, which makes the minimal set feel constrained rather than intentionally minimal.
The robots.txt side is partially complete (per-bot verdicts and deciding rule are covered) but only for path '/', with no way to query other paths, specific user-agents, or enumerate the bot registry. get_site_info is a single fixed-document fetch, so the surface has notable gaps an agent cannot work around.
Reads the public robots.txt of a website (one request to https:///robots.txt, nothing else on the site) and says, for each AI crawler and AI agent of the migurei bot registry (training crawlers such as GPTBot and ClaudeBot, search bots, fetchers that act for a user, and others), whether that bot is allowed or disallowed for the path "/", which robots.txt rule decided it, and whether the file exists. Input is a domain or URL; only public domain names are accepted (no IP addresses, no ports, no private hosts). Read-only; the domain is not stored. robots.txt is a request to bots, not access control, and a User-Agent is only a claim.
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | A public domain name or a URL, for example example.com or https://example.com/page. Only its host is used. |
| Name | Required | Description |
|---|---|---|
| bots | Yes | |
| domain | Yes | |
| method | Yes | |
| summary | Yes | |
| learn_more | Yes | |
| robots_txt | Yes | |
| robots_url | Yes | |
| registry_version | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare read-only, open-world, idempotent, non-destructive behavior, yet the description adds substantial context beyond them: exactly one HTTP request is made, the domain is not stored, robots.txt is advisory rather than access control, and a User-Agent is only a claim. These are real behavioral traits an agent could not infer from annotations alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the action and the request scope before detailing the per-bot output and input constraints. The final caveat about robots.txt not being access control is relevant but slightly lengthens an already dense definition; every clause is useful, though the description is near the upper bound of appropriate length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-value explanation is unnecessary; the description covers scope, input restrictions, per-bot output semantics, and the advisory nature of robots.txt. For a single-parameter read-only tool, an agent has everything needed to call it correctly and interpret the result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and already explains that the domain may be a URL and that only its host is used. The description adds meaning beyond the schema by excluding IP addresses, ports, and private hosts, which clarifies what values are valid. It stops short of describing length limits or URL parsing edge cases, so not a 5.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource (reads robots.txt) and precisely scopes the operation: one request to https://<host>/robots.txt, nothing else on the site. It also enumerates what the answer contains (per-bot allowed/disallowed, deciding rule, file existence), which lets an agent distinguish it from get_site_info without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear conditions for use: input is a domain or URL, only public domain names accepted, no IPs, ports, or private hosts. It does not explicitly route the agent to or away from the sibling get_site_info, so it falls short of full alternative guidance, but the scope constraints are strong enough to select the tool correctly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Returns the facts published on this website, in Japanese: service overview, plans, prices and payment terms, how the scanner behaves, how to contact us, and the operating company. Each fact comes with the URL of the page that shows it. Read-only; it returns nothing that is not on the pages.
| Name | Required | Description | Default |
|---|---|---|---|
| topic | Yes | Which facts to return |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, so safety is covered; the description adds genuinely new context — content is in Japanese, every fact carries its source page URL, and nothing beyond the published pages is returned. It stops short of describing result shape or volume, but the added context is real.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, each earning its place: what it returns, what accompanies each fact, and the read-only/source-bound guarantee. Front-loaded with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With a single required enum parameter, full schema coverage, and no output schema, the description covers enough: content domain, language, return format (facts plus source URL), and fidelity limits. Minor gap is what 'all' yields and roughly how much content comes back.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the enum values are self-describing, so the schema does the work. The description's topic list paraphrases the enum without adding format, scoping, or conflict-resolution detail (e.g., what 'all' returns).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Names a specific verb and resource ('Returns the facts published on this website') and enumerates the fact categories, which line up with the topic enum. It is clearly distinct from check_ai_bot_access by subject matter, though it never explicitly contrasts with that sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: it is the tool for published site facts. There is no statement of when to pick it over check_ai_bot_access, no note on whether topic='all' is the default approach, and no exclusions (e.g., that it will not answer questions about dynamic data).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
check_ai_bot_accessget_site_infoCheck which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Check whether a website is readable by AI agents: robots, llms.txt, sitemap, HTML. Bounded.
Free check whether a website is technically open to AI assistants (AI crawlers, llms.txt, JSON-LD).
Add one secure layer between your agents and this server.