AI Bot Access Checker

Can ChatGPT, Claude, Perplexity and Google actually reach your page? This tool checks 13 AI and search crawlers against one URL and tells you, for each, whether robots.txt blocks it or whether your server or firewall refuses it while a normal browser gets the page.

• Web version: seopaslaugos.io/en/tools/ai-bot-access-checker

• REST API: https://tools.seopaslaugos.io/v1/check?url=example.com (OpenAPI)

• Remote MCP server: https://tools.seopaslaugos.io/mcp (Streamable HTTP, tool check_ai_bot_access)

• Agent skill: skills/ai-bot-access-check/SKILL.md

Free, no key, MIT licence.

Why

robots.txt is only half of it. Shared-hosting security rules (ModSecurity), rate limiters that answer 429 to bots on uncached pages, and CDN bot settings (Cloudflare's AI crawler blocking) refuse AI crawlers before robots.txt is ever read, often by default. Site owners then wonder why ChatGPT says it "cannot access" their site. Most checkers only read robots.txt; this one also requests the page as each bot and compares the answer with a browser's.

Related MCP server: ai-visibility-mcp

Bots checked

Use it

REST

curl "https://tools.seopaslaugos.io/v1/check?url=example.com&format=text"
curl "https://tools.seopaslaugos.io/v1/check?url=https://example.com/pricing&bots=GPTBot,ClaudeBot"

AI bot access for https://www.example.lt/
Browser baseline: HTTP 200
robots.txt: found (HTTP 200)

GPTBot             BLOCKED (server/firewall) - Gets a Cloudflare block page (HTTP 403) instead of the page.
OAI-SearchBot      BLOCKED (server/firewall) - Gets a Cloudflare block page (HTTP 403) instead of the page.
Claude-User        ALLOWED
Googlebot          ALLOWED
...

MCP (Claude Code, Cursor, ChatGPT, any MCP client)

{
  "mcpServers": {
    "ai-bot-access-checker": { "type": "http", "url": "https://tools.seopaslaugos.io/mcp" }
  }
}

Hermes Agent

hermes plugins install ai-bot-access-checker

The package is an Agent Plugins v1 bundle: plugin.json, mcp.json (remote Streamable HTTP server, no keys, no local process) and the skill in skills/. The one tool is read-only; it sends HTTP requests to the URL you ask it to check, from tools.seopaslaugos.io.

Agent skill

Copy skills/ai-bot-access-check into your agent's skills folder, or install it from ClawHub. The skill calls the REST API with curl, so it needs no MCP support.

Results

How it works and its limits

• robots.txt is parsed per RFC 9309: most specific user-agent group, merged duplicate groups, longest match wins, Allow wins ties, * and $ wildcards; a 4xx robots.txt means no restrictions, a 5xx means full disallow.

• The page is requested once with a desktop Chrome User-Agent and once per bot with the bot's published User-Agent, from Cloudflare's network.

• Requests do not come from the vendors' IP ranges. Blocks by User-Agent (most hosting firewalls) are caught; firewalls that verify crawler IPs are reported as inconclusive instead of guessed.

• Rate limits: 20 checks a minute per caller IP, and 6 fresh checks a minute per checked website from all callers together, so the checker cannot be used to flood a site. Results are cached 10 minutes per URL.

Security

• Only public http/https URLs on ports 80/443 with a domain name are checked. IP literals (dotted, decimal, octal, hex, IPv6), localhost and reserved TLDs, wildcard-DNS hosts that point at private addresses (nip.io, sslip.io, ...), URLs with credentials and the checker's own hosts are refused.

• Redirects are followed by hand (max 5) and every hop is validated the same way, so a public page cannot bounce the checker to an internal address, another port or its own API.

• At most 64 KiB of each response is read; each request times out after 10 s.

• robots.txt matching uses a linear wildcard matcher with a work budget and caps on lines, rules and pattern length, so a hostile robots.txt cannot stall the Worker; if the budget runs out the result says inconclusive.

• Text taken from the checked site (robots.txt rules, URLs, headers) is stripped of control and bidi characters and length-capped before it is returned to people or AI agents.

• MCP batches are capped at 10 messages and one tools/call; request bodies are capped at 16 KiB.

• No keys, cookies or user data are stored. Results are cached by URL only.

Found a problem? Open an issue or email seo@seopaslaugos.io.

Run your own

npm install
npm test          # robots.txt parser tests
npx wrangler dev  # local Worker on :8787

Deploy with npx wrangler deploy after changing the route in wrangler.toml.

Author

Built by Mantas Tamošaitis, SEO specialist in Lithuania, who works on Google and AI search visibility for service businesses and e-shops at seopaslaugos.io. Issues and pull requests welcome.