Provides tools to analyze and manage robots.txt rules for Amazonbot, enabling control over how Amazon's AI crawlers access website content.
Provides tools to analyze and manage robots.txt rules for Baiduspider, enabling control over how Baidu's search crawlers access website content.
Provides tools to analyze and manage robots.txt rules for Googlebot and Google-Extended, enabling control over how Google's search and AI agents crawl website content.
Provides tools to analyze and manage robots.txt rules for Meta-ExternalAgent, enabling control over how Meta's AI agents crawl website content.
Provides tools to analyze and manage robots.txt rules for PerplexityBot, enabling control over how Perplexity's AI search agents crawl website content.
Manage which AI bots can crawl your website — visually.
Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.
Visual Toggle UI — Block or allow AI bots with simple on/off switches
20+ AI Bots Database — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent, and more
Analyze Existing robots.txt — Paste or fetch any robots.txt to see which AI bots are blocked
Generate robots.txt — Create a complete robots.txt with your chosen rules
MCP Server — Use with Claude Desktop, Cursor, or any MCP-compatible AI assistant
Check Bot Status — Verify if a specific bot is blocked on any website
Related MCP server: ai-visibility-mcp
Tool | Description |
| Fetch and analyze a robots.txt from any URL |
| Analyze pasted robots.txt content for AI bot blocking status |
| Generate a robots.txt with specified blocked bots and custom rules |
| List all known AI bots with user-agents, companies, and descriptions |
| Check if a specific bot is blocked on a given website |
git clone https://github.com/sharozdawa/robotstxt-ai.git
cd robotstxt-ai
npm install
npm run devOpen http://localhost:3000 in your browser.
Add to your claude_desktop_config.json:
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": ["-y", "robotstxt-ai-mcp"]
}
}
}Add to .cursor/mcp.json:
{
"mcpServers": {
"robotstxt-ai": {
"command": "npx",
"args": ["-y", "robotstxt-ai-mcp"]
}
}
}The server knows about 25+ bots including:
AI Crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Diffbot, cohere-ai, Amazonbot, Meta-ExternalAgent
AI Search: ChatGPT-User, OAI-SearchBot, PerplexityBot, YouBot
Search Engines: Googlebot, Bingbot, YandexBot, Baiduspider, DuckDuckBot
Feature | robotstxt.ai | Manual Editing |
Visual toggle UI | Yes | No |
20+ AI bots database | Yes | Research yourself |
Analyze existing robots.txt | Yes | No |
MCP Server | Yes | No |
Price | Free | Free but tedious |
Tool | Description |
Curated list of SEO MCP servers and agent skills | |
Instant URL indexing via IndexNow | |
Schema.org JSON-LD markup generator | |
AI brand visibility tracker |
MIT
Built by Sharoz Dawa — SEO Professional & Digital Marketing Expert
Analyze pasted robots.txt content. Returns which AI bots are blocked or allowed based on the rules.
Inputs: content (required).
Check if a specific bot is blocked or allowed on a given website by fetching and analyzing its robots.txt.
Inputs: url (required), bot_name (required).
Fetch and analyze a robots.txt file from a URL. Returns which AI bots are blocked or allowed.
Inputs: url (required).
Generate a robots.txt file with specified blocked bots, sitemap URLs, and custom rules.
Inputs: block_all_ai, blocked_bots, custom_rules, sitemap_urls.
List all known AI bots with their user-agents, companies, and descriptions. Useful for deciding which bots to block.
Inputs: no named parameters.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 5 tools
Each tool targets a distinct operation: analyzing pasted content, checking a specific bot, fetching from a URL, generating, and listing bots. No overlap in functionality.
All tool names follow a consistent verb_noun pattern with snake_case (e.g., analyze_robots, check_bot_status). No mixing of conventions.
Five tools is a well-scoped set for this domain, covering the essential tasks without being too few or excessive.
The tools cover the full lifecycle: fetching, analyzing, checking specific bots, generating, and listing known AI bots. No obvious gaps for the intended purpose.
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
See what GPTBot, ClaudeBot and other crawlers get from a site: robots.txt checks and AI scans.
Check which AI bots a website or a pasted robots.txt lets in, grouped by purpose.