crawlerpeek
The crawlerpeek MCP server provides six read-only tools for auditing AI crawler access, robots.txt, sitemaps, and crawler IPs.
check_ai_access: check robots.txt decisions per bot and optionally fetch the page with each bot's user-agent to report live HTTP status.see_as_bot: fetch one URL as a specified bot and report status, redirects, content type, size, title, and block/challenge signals.verify_bot_ip: verify an IP against an operator's published IP ranges or forward-confirmed reverse DNS, optionally for a specified bot; never connects to the IP.generate_robots_txt: generate a robots.txt that welcomes AI crawlers, with optional disallowed paths, blocked bots, and sitemap URL; no network requests.lint_robots_txt: lint robots.txt from a site URL or supplied text, returning stable finding codes, line numbers, and severities.scan_site: read a site's sitemap (following an index one level) and report which sitemap URLs robots.txt blocks per bot, without fetching the pages.
Checks access for Apple's crawlers (Applebot, Applebot-Extended) and verifies whether an IP belongs to Apple using its published IP ranges or forward-confirmed reverse DNS.
Checks DuckDuckGo's crawler (DuckAssistBot) against robots.txt and live HTTP responses, and verifies claimed DuckDuckGo IPs using the operator's published IP ranges.
Checks access for Googlebot and Google-Extended (robots.txt rules and live fetches) and verifies IPs claiming to be Googlebot via Google's published IP ranges or reverse DNS with forward confirmation.
Checks whether Meta's crawlers (meta-externalagent, meta-externalfetcher) may access a URL, combining robots.txt decisions with live HTTP fetches using their user agents.
Checks and verifies OpenAI's crawlers (GPTBot, OAI-SearchBot, ChatGPT-User): robots.txt access decisions, live HTTP status when fetched as the bot, and IP verification against OpenAI's published IP-range lists.
Checks access for PerplexityBot and Perplexity-User via robots.txt and live HTTP fetches, and verifies Perplexity IPs using the operator's published IP ranges.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@crawlerpeekCheck whether GPTBot and ClaudeBot can crawl example.com/pricing"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
crawlerpeek
Check whether AI crawlers can access your site. For each bot (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot and more) it tells you:
whether
robots.txtallows the URL's path for that bot (RFC 9309: user-agent groups, wildcards,$, longest match wins), andwhat the page actually returns when fetched with that bot's user-agent string: status, final URL after redirects, content type, size, title, and whether the response looks like a block or challenge page.
It also lints a robots.txt for mistakes, checks every URL in a sitemap against robots.txt, and generates a robots.txt that welcomes AI crawlers. A library, a CLI for CI, and an MCP server. Node 20 or newer, no runtime dependencies, ESM and CommonJS, TypeScript types included.
The same checks as a web tool: crawlerpeek.com. Documentation with examples for CI and MCP clients: crawlerpeek.com/cli.
Install
npm install crawlerpeek # library
npx crawlerpeek --help # CLI, no install neededRelated MCP server: geo-tool-check
CLI
npx crawlerpeek check https://example.comhttps://example.com
robots.txt: ok (HTTP 200) at https://example.com/robots.txt
Bot robots HTTP
GPTBot blocked 200 FAIL
ClaudeBot allowed 403 (block page) FAIL
Google-Extended allowed -npx crawlerpeek check https://example.com/pricing --bots GPTBot,ClaudeBot --fail-on blocked
npx crawlerpeek check https://example.com --bots all --json
npx crawlerpeek check https://example.com --no-http # robots.txt only
npx crawlerpeek see https://example.com --bot ClaudeBot # one bot, full detail
npx crawlerpeek verify 66.249.66.1 --bot Googlebot # is this IP really Googlebot?
npx crawlerpeek lint https://example.com # find mistakes in its robots.txt (or: lint ./robots.txt, lint -)
npx crawlerpeek scan https://example.com --limit 50 # robots.txt per bot for the URLs in its sitemap
npx crawlerpeek check https://example.com --markdown # GitHub-flavoured Markdown table
npx crawlerpeek robots --disallow /admin/,/cart/ --sitemap https://example.com/sitemap.xmlExit codes: 0 ok, 1 the --fail-on condition was met, 2 usage error (including an unknown option) or request error. see exits 0 when the page was fetched, whatever its HTTP status, and 2 when the request failed or was refused.
verify <ip> [--bot GPTBot] [--json] checks whether an address from your server logs really belongs to a crawler. It uses only what the operator documents: the published IP-range JSON (Google, OpenAI, Anthropic, Perplexity, Bing, Apple, Common Crawl, DuckDuckGo, Ahrefs) or reverse DNS with forward confirmation (Google, Bing, Apple, Common Crawl, Ahrefs). Without --bot every bot with such a method is checked. Each result has method (ip-list, reverse-dns or none), verified (true, false, or null when it cannot be decided, for example the list is unreachable or the operator documents no method) and a detail. Lists are cached for one hour, and nothing is ever sent to the IP. Exit codes: 0 verified for at least one bot, 1 not verified or undecidable, 2 usage error.
check --markdown prints the result as a GitHub-flavoured Markdown table (a heading with the URL, a ✅/❌ summary line and one row per bot) instead of the text table; it cannot be combined with --json. --summary-file <path> appends the same Markdown to a file in addition to the normal output, which is how the GitHub Action fills the job summary (--summary-file "$GITHUB_STEP_SUMMARY").
lint <url|file|-> [--fail-on error|warning] [--json] checks a robots.txt: the /robots.txt of a URL, a local file, or - for stdin. It prints one line per finding (L12 warning unknown-field ...) and a summary. Each finding has a stable code, a severity (error, warning, info) and the line number, and is based on RFC 9309 or Google's robots.txt documentation:
Code | Severity | Finding |
| error | the server answered |
| error | the file is binary or compressed data, not text |
| warning | Allow or Disallow before any User-agent line (ignored) |
| warning | a misspelled field such as |
| warning | an Allow or Disallow path that does not start with |
| warning | a Sitemap value that is not an absolute http(s) URL |
| warning | every AI training, search and user crawler in the default list is blocked from |
| warning | more than 500 KiB; Google reads only the first 500 KiB |
| info |
|
| info |
|
| info | a misspelled bot token (for example |
| info | a User-agent group without rules, or the same token in more than one group |
| info | the file has more than 20 000 lines, or more than 200 findings |
Exit codes: 0 nothing at or above the --fail-on threshold (default warning), 1 findings, 2 usage error or a robots.txt that could not be fetched. A missing robots.txt (404) has no findings.
scan <url> [--limit 25] [--bots a,b|all] [--fail-on none|blocked] [--json] evaluates robots.txt for the URLs in the site's sitemap: the Sitemap: lines of its robots.txt, else /sitemap.xml; a sitemap index is followed one level (at most 5 child sitemaps). Only http(s) URLs on the same host are kept (www. and the bare domain count as different hosts, unless robots.txt redirects from one to the other), without duplicates, and --limit (1 to 200) takes the first ones. .gz sitemaps are reported as skipped. No page is fetched: the only requests are robots.txt and the sitemap files. The output has a table of how many URLs each bot is blocked from, then the blocked URLs per bot (the first 10, then "... and N more"; --json has all). Exit codes: 0 ok, 1 with --fail-on blocked when any checked bot is blocked from any URL, 2 usage or request error, or when no URL could be read from the sitemap.
| Fails when |
| robots.txt closes the path for a bot, or the page answers that bot with 401/403/406/451 or a challenge page |
| as above, or any other status (404, 5xx, redirects that end elsewhere) or a failed request |
Without --fail-on the exit code is 0 whatever the result. --bots takes tokens or all; the default is 15 bots: the main AI crawlers plus Googlebot and Bingbot. A robots.txt that cannot be fetched (timeout, 5xx, a redirect that is refused or has no target) is reported as unknown, with the reason in robotsTxt.error, and never counts as blocked. Note: this deliberately differs from RFC 9309 §2.3.1.4, where crawlers treat an unreachable robots.txt (5xx) as "disallow everything" (Google does too), so a site whose robots.txt returns 5xx is effectively closed to many bots even though --fail-on blocked passes. Treat unknown as a problem to fix.
Library
import { checkAiAccess, seeAsBot, verifyBotIp, lintRobotsTxt, lintRobotsUrl, scanSite, formatMarkdown, generateRobotsTxt, failingBots } from "crawlerpeek";
const report = await checkAiAccess("https://example.com/pricing", { bots: ["GPTBot", "ClaudeBot"] });
report.bots[0].robots; // { access: "allowed", pathAllowed: true, matchedGroup: "*" }
report.bots[0].http; // { status: 200, finalUrl, redirects, contentType, bytes, title, looksBlocked, outcome: "ok", ... }
failingBots(report, "blocked"); // ["ClaudeBot"] when it is closed or challenged
await seeAsBot("https://example.com", "PerplexityBot");
await verifyBotIp("66.249.66.1", { bot: "Googlebot" }); // [{ bot: "Googlebot", method: "ip-list", verified: true, detail, source }]
lintRobotsTxt("User-agent: *\nDissallow: /a\n"); // [{ line: 1, severity: "info", code: "empty-group", ... }, { line: 2, severity: "warning", code: "unknown-field", message: '... Did you mean Disallow?' }]
await lintRobotsUrl("https://example.com"); // { url, status, state, findings, ... }
const scan = await scanSite("https://example.com", { limit: 50, bots: ["GPTBot"] });
scan.summary.perBot; // [{ bot: "GPTBot", blocked: 3 }]
scan.urls[0]; // { url: "https://example.com/private/a", blockedFor: ["GPTBot"] }
formatMarkdown(report, failingBots(report, "blocked")); // GitHub-flavoured Markdown
generateRobotsTxt({ disallow: ["/admin/"], blockBots: ["CCBot"], sitemap: "https://example.com/sitemap.xml" });generateRobotsTxt names each AI bot explicitly with Allow: /. Disallowed paths are repeated in that group on purpose: a bot with its own group ignores the * group.
MCP server
crawlerpeek mcp speaks the Model Context Protocol over stdio and exposes six read-only tools: check_ai_access, see_as_bot, verify_bot_ip, generate_robots_txt, lint_robots_txt (give a url or the file text) and scan_site (url, limit, bots). Results are cached for 10 minutes per call. No LLM is called; the tools only fetch the URL you give them.
Client | Install |
Claude Code |
|
Claude Code (plugin: MCP server and skill) |
|
Claude Desktop | Download |
Codex |
|
VS Code |
|
Any other client | the JSON below |
Generic configuration (Claude Desktop, Cursor, Windsurf, Cline and others use this mcpServers shape):
{ "mcpServers": { "crawlerpeek": { "command": "npx", "args": ["-y", "crawlerpeek", "mcp"] } } }VS Code reads .vscode/mcp.json with servers as the root key and "type": "stdio" on the entry. Codex reads ~/.codex/config.toml; raise the startup timeout so the first npx download does not run into the default 10 seconds:
[mcp_servers.crawlerpeek]
command = "npx"
args = ["-y", "crawlerpeek", "mcp"]
startup_timeout_sec = 30Agent skill
skills/crawlerpeek/ is an Agent Skill: a SKILL.md that tells an AI agent when to check AI crawler access, which commands to run, and how to read the result. It uses only the common frontmatter fields, so the same folder works in several agents.
Agent | Install |
Claude Code | installed with the plugin above, or copy the folder to |
claude.ai | zip the |
Codex | copy the folder to |
OpenClaw | copy the folder to |
Hermes |
|
git clone --depth 1 https://github.com/crawlerpeek/crawlerpeek.git
mkdir -p ~/.agents/skills && cp -r crawlerpeek/skills/crawlerpeek ~/.agents/skills/Restart the agent afterwards so it picks up the new skill. The skill and the MCP server are independent: with the MCP server connected the skill uses its tools, otherwise it runs npx -y crawlerpeek.
GitHub Action
A composite action lives in action/; see action/README.md.
- uses: crawlerpeek/crawlerpeek/action@v0
with: { url: "https://example.com", fail-on: blocked }The action also writes a Markdown table of the result to the job summary (input summary, on by default).
What you should know
see-as-bot sends the bot's own user-agent string from your machine. It is not the real bot and not from the operator's IP ranges. Sites that verify bot IPs (Googlebot, GPTBot and others publish theirs) may treat a request that only claims to be the bot differently, often by blocking it. A 403 for
Googlebothere can therefore be normal; a 403 for an unusual bot with a 200 for a browser usually points at a WAF or CDN rule.robots.txtitself is fetched with an honestcrawlerpeek/<version>user agent.Google-Extended and Applebot-Extended are robots.txt control tokens, not crawlers: they get a robots.txt decision but are never fetched.
Pages are fetched as raw HTML. No JavaScript runs.
robots.txt is a request, not access control. Some user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow it; the operators' documentation says which.
The bot list and user-agent strings come from the operators' documentation and can drift; the list was last compared with the operators' documentation on 2026-10-18.
Safety limits
check: one request per bot plus one for robots.txt, sequential with a 300 ms pause (--delay); 10 s timeout for the whole request chain; at most 5 redirects, each re-checked (a redirect to a refused target is reported for that bot or for robots.txt, it does not stop the run); response bodies capped at 2 MB; no cookies. lint makes one request (robots.txt, read up to 500 KiB); scan makes at most 11: robots.txt, up to 5 sitemaps and up to 5 child sitemaps of an index (5 MB each, 50 000 entries), and never fetches the pages themselves. Private, loopback, link-local and metadata addresses are refused after DNS resolution (and again at connect time), as are ports other than 80 and 443 and URLs with credentials. --allow-private (CLI and crawlerpeek mcp) or allowPrivate: true lifts these limits for local testing.
Contributing
Issues and pull requests are welcome, see CONTRIBUTING.md. Report security problems privately, see SECURITY.md.
License
Available Tools
6 toolscheck_ai_accessARead-onlyIdempotent
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) may access a URL: robots.txt decision per bot plus the live HTTP status when the page is fetched with each bot's user-agent string. Read-only; one request per bot.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL of the page to check. | |
| bots | No | Bot tokens to check, or omit for the default 15. Known: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Googlebot, Bingbot, Applebot, meta-externalfetcher, DuckAssistBot, Amazonbot, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit, Slackbot. | |
| http | No | Also fetch the page with each bot's user agent (default true). false checks robots.txt only. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld and non-destructive, so the safety profile is covered. The description adds real behavioral context beyond that: it makes one live HTTP request per bot, and the cost grows with the bot set (default 15), which is exactly the kind of operational detail an agent needs before calling an open-world tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no filler. The core purpose and the bot list come first, followed by the read-only and one-request-per-bot caveats — everything is front-loaded and every clause carries information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 3-parameter, no-output-schema tool the description covers what an agent needs: the per-bot robots.txt verdict and the echoed HTTP status are described, so return values are not a mystery. It stops short of describing result shape per bot or error behavior for unreachable/blocked URLs, which keeps it from a 5.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents url, bots and http, including the full known-bot list and the default. The description only echoes the http semantics ('live HTTP status when the page is fetched with each bot's user-agent string') without adding format or constraint detail, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a concrete verb and resource ('Check whether AI crawlers ... may access a URL') and enumerates the specific bots plus the two outputs (robots.txt decision, live HTTP status). It is clear what the tool does, though it never names the closest sibling see_as_bot, so an agent must infer the boundary itself.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the description: use it when you need to know whether AI crawlers are allowed on a page. There is no explicit when-not guidance, no mention of generate_robots_txt or see_as_bot as alternatives, and no prerequisites stated (e.g. URL must be public, which only appears in the schema).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
generate_robots_txtARead-onlyIdempotent
Generate a robots.txt that welcomes AI crawlers, with optional disallowed paths and bots to close. Pure text generation; makes no network requests.
| Name | Required | Description | Default |
|---|---|---|---|
| sitemap | No | Absolute sitemap URL. | |
| disallow | No | Paths no bot should fetch, e.g. ["/admin/"]. | |
| block_bots | No | Bot tokens to close completely, e.g. ["CCBot"]. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive/openWorld=false, so the safety bar is low. The description still adds real value: it discloses the default posture (welcoming AI crawlers) and that nothing is applied to a live site or fetched, which is the key mental model for an agent deciding whether to persist the output.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, purpose front-loaded, with the scope-of-effect caveat immediately after. No filler and nothing that could be cut without losing meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Zero required parameters and a simple text-generation contract make this a low-complexity tool that the description covers well, including the crucial fact that no network mutation happens. Only gap is that the returned artifact's shape (raw robots.txt text vs. structured) is left implicit, though that is nearly self-evident.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with examples for both arrays, so the schema carries the parameter burden. The description only restates 'optional disallowed paths and bots to close' and omits the sitemap parameter entirely, adding no syntax or default behavior beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Generate a robots.txt') plus the distinguishing policy stance ('welcomes AI crawlers') and the optional knobs. Echoes the title's verb but adds scope that separates it from the sibling tools see_as_bot and check_ai_access, which inspect rather than generate.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies the use case (produce a permissive robots.txt with carve-outs), and 'pure text generation; makes no network requests' hints this is not the checking path. But it never names see_as_bot or check_ai_access or states when an agent should pick those instead.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
lint_robots_txtARead-onlyIdempotent
Lint a robots.txt for mistakes: HTML served instead of robots.txt, misspelled fields or bot tokens, rules before any User-agent, relative paths or sitemap URLs, duplicate or empty groups, directives Google ignores, every AI crawler blocked. Give the site's URL (fetches its /robots.txt) or the file's text. Findings carry stable codes, line numbers and a severity.
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | Public http(s) URL of the site; its /robots.txt is fetched. | |
| text | No | The robots.txt content to lint, instead of url. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld and non-destructive, so the safety bar is lowered. The description adds genuinely new context: that the URL mode performs an outbound fetch of /robots.txt, and that results are structured findings with stable codes, line numbers and severity — useful for planning downstream handling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the purpose, then a tight list of finding categories, then input modes, then output shape. Three sentences, no filler, every element earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description carries the return-value burden and does so ('Findings carry stable codes, line numbers and a severity'). Combined with the fetch semantics and dual input modes, an agent has everything needed for a read-only lint call.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and both parameters already document the URL-fetch and inline-text behavior, so the description adds only a paraphrase of the same distinction. Baseline 3 applies since the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Lint a robots.txt') and enumerates the concrete defect classes it detects, which also separates it from generate_robots_txt (authoring) and check_ai_access (AI-specific probing). An agent can pick this tool without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The two entry modes are made explicit ('Give the site's URL (fetches its /robots.txt) or the file's text'), which frames the usage context well. However, no sibling is named and there are no stated exclusions, so routing versus generate_robots_txt or check_ai_access is left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scan_siteARead-onlyIdempotent
Check robots.txt for many pages of one site at once: takes the URLs from its sitemap (listed in robots.txt, else /sitemap.xml; a sitemap index is followed one level) and reports per bot which URLs robots.txt blocks. Same host only, up to 200 URLs, .gz sitemaps are skipped. Read-only; fetches only robots.txt and the sitemap files, never the pages.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL on the site to scan. | |
| bots | No | Bot tokens to check, or omit for the default 15. Known: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Googlebot, Bingbot, Applebot, meta-externalfetcher, DuckAssistBot, Amazonbot, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit, Slackbot. | |
| limit | No | How many sitemap URLs to check (default 25). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only/idempotent/open-world, yet the description adds real behavior: it fetches only robots.txt and sitemap files and never the pages, follows a sitemap index one level, falls back from robots.txt-declared sitemap to /sitemap.xml, and skips .gz sitemaps. Missing only what happens when robots.txt or the sitemap is absent/blocked (error vs empty result).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One dense paragraph, front-loaded with the purpose and sitemap discovery order before the constraints. Every clause carries information (discovery fallback, index depth, host/limit bounds, .gz skip, fetch scope); nothing is padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, and the description gives the return shape at a high level ('per bot which URLs robots.txt blocks'), plus discovery and limit semantics. Enough to call correctly; a sentence on behavior when no sitemap is found would close the remaining gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the schema already documents bots (including the known-token list), limit (max 200, default 25), and the http(s) url constraint. The description re-states 'up to 200 URLs' and adds the same-host restriction, but contributes almost nothing the schema does not already say, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource+scope: 'Check robots.txt for many pages of one site at once', with the URL source and the output shape ('reports per bot which URLs robots.txt blocks'). This clearly separates it from lint_robots_txt (rule validation) and check_ai_access (single-page check) without needing the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'for many pages of one site at once' and 'Same host only, up to 200 URLs' give clear conditions for use and the boundary where the tool stops being appropriate. It never explicitly names the sibling to use instead for a single-URL check, so no full alternative routing, but the bulk-vs-single distinction is inferable.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
see_as_botARead-onlyIdempotent
Fetch one URL with a single bot's user-agent string from this machine and report status, final URL after redirects (max 5), content type, size, page title and whether the response looks like a block or challenge page. Does not prove what the real bot gets (operators verify IP ranges).
| Name | Required | Description | Default |
|---|---|---|---|
| bot | Yes | Bot token, for example GPTBot. Known: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Googlebot, Bingbot, Applebot, meta-externalfetcher, DuckAssistBot, Amazonbot, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit, Slackbot. | |
| url | Yes | Public http(s) URL. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, openWorld and non-destructive, so the description does not need to restate safety. Instead it adds behavior the annotations cannot convey: a five-redirect cap, the exact spoofed user-agent mechanism, block/challenge detection, and the important limitation that results are not proof of what the real crawler receives.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, zero filler, front-loaded with the mechanism and immediately followed by the report list and the caveat. Every clause carries information an agent needs to predict the output.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description steps in to name every returned field (status, final URL, content type, size, title, block-likelihood) plus the redirect cap and the interpretive limitation. For a two-parameter read-only fetch, nothing an agent needs before calling is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% — both url and bot are documented in the schema, including the enumerated list of known bot tokens. The description adds no syntax or format detail beyond that, so the schema does all the parameter work and the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource — fetch one URL with a given bot user-agent — and enumerates what is reported (status, final URL, content type, size, title, block heuristics). That is highly distinctive, but it never names the sibling tools it could be confused with, so sibling differentiation is left to inference.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The closing caveat 'Does not prove what the real bot gets (operators verify IP ranges)' is a real usage boundary that a caller should know. However, it offers no positive when-to-use statement and no explicit comparison to check_ai_access or generate_robots_txt, so routing between siblings remains implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
verify_bot_ipARead-onlyIdempotent
Check whether an IP address really belongs to a crawler (for example one seen in server logs claiming to be Googlebot), using only what the operator documents: its published IP-range list or forward-confirmed reverse DNS. Without bot, checks every bot that has such a method. Reads the operators' public lists; never connects to the IP. verified null means it could not be decided.
| Name | Required | Description | Default |
|---|---|---|---|
| ip | Yes | IPv4 or IPv6 address. | |
| bot | No | Bot token to verify against, for example Googlebot. Omit to check all. Known: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Googlebot, Bingbot, Applebot, meta-externalfetcher, DuckAssistBot, Amazonbot, AhrefsBot, SemrushBot, MJ12bot, facebookexternalhit, Slackbot. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only, idempotent, open-world and non-destructive, but the description adds real behavioral facts on top: it reads operators' public lists, never connects to the target IP, and 'verified null means it could not be decided'. That last point is essential tri-state semantics that no structured field conveys.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, purpose front-loaded, with no filler. The second sentence ('Without bot, checks every bot that has such a method') is slightly awkward in phrasing, keeping it from a perfect score.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the return-value burden and does explain the null case, but it never states the shape of a successful result (e.g. the fields returned alongside 'verified'). Otherwise it covers method, scope and safety adequately for a 2-parameter tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3; the description earns an extra point by clarifying that omitting 'bot' checks every bot 'that has such a method' – a nuance beyond the schema's plain 'Omit to check all'. It also implicitly frames the result as a 'verified' tri-state value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource (verify whether an IP really belongs to a crawler) and names the two documented verification methods (published IP-range list, forward-confirmed reverse DNS). It does not explicitly contrast itself with the siblings (see_as_bot, check_ai_access, generate_robots_txt), though the operation is distinct enough to be inferred.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The server-log spoofing example ('claiming to be Googlebot') gives a concrete when-to-use context, and 'Without bot, checks every bot' clarifies the default scope. There are no explicit when-not-to-use conditions or named alternatives, so it stops short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
v0.3.1- Added
lint_robots_txt - Added
scan_site
1 tool update
v0.2.0- Added
verify_bot_ip
3 tool updates
v0.1.0- First observed
check_ai_access - First observed
generate_robots_txt - First observed
see_as_bot
TDQS
Scored across 6 tools
Each tool has a distinct core action: generate, lint, per-URL bot check, site-wide scan, single-bot fetch, and IP verification. The main overlap is between check_ai_access, see_as_bot, and scan_site, which all involve fetching with crawler user-agents, but the descriptions clarify their different scopes (robots decision vs. live response detail vs. bulk sitemap checking).
All names are snake_case and verb-first, which is predictable overall. see_as_bot is the only notable deviation from the verb_noun pattern, but it remains readable and clearly action-oriented.
Six tools is well-scoped for a focused robots.txt and AI-crawler access server. Each tool covers a distinct workflow and none feels redundant or padding.
The surface covers generation, linting, per-page checking, bulk scanning, live bot-user-agent fetching, and IP verification, which is strong end-to-end coverage. Minor gaps exist, such as no explicit tool to list supported bots or explain their individual rules, but agents can work around this through the existing checks.
Maintenance
Related MCP Connectors
Check which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
Free MCP tools for AI-search visibility: crawler checks, page audits, and llms.txt generation.
AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Related MCP Servers
- AlicenseAqualityCmaintenanceAnalyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.51MIT
- AlicenseNot gradedqualityAmaintenanceProvides agents with tools to score a page's AI readiness, check which AI crawlers robots.txt blocks, and assess passage citability—all performed locally.56 npm2MIT
- AlicenseAqualityAmaintenanceLets an MCP client check whether an ecommerce store can be read by AI assistants, auditing robots.txt to see which AI search versus training crawlers are blocked and sampling product pages to confirm structured data and facts survive in raw HTML without JavaScript. It can also generate a training-only opt-out robots.txt that leaves answer-engine crawlers allowed, all inside the conversation the user is already in.2MIT
- AlicenseAqualityCmaintenanceEnables AI agents to audit and improve a site's visibility in AI search by checking robots.txt against 17 major AI crawlers, detecting llms.txt, and returning a 0-100 visibility score with per-crawler verdicts; it also lists known AI crawlers and their purposes.2MIT