Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@RobotsGate check example.com for AI crawler directives"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Web app and API: https://robotsgate.mike-tusa.workers.dev/
Pricing and plans: https://robotsgate.mike-tusa.workers.dev/
Support: digitalpromohub.support@gmail.com
Related MCP server: mcp-geo
RobotsGate is a remote MCP server (Streamable HTTP) at https://robotsgate.mike-tusa.workers.dev/mcp. Tools: list_crawlers, generate_robots, validate_robots, check_site.
Most MCP clients (Claude Desktop, Cursor, Cline, Windsurf) accept this config:
{
"mcpServers": {
"robotsgate": {
"type": "http",
"url": "https://robotsgate.mike-tusa.workers.dev/mcp"
}
}
}Clients that only support stdio can bridge with npx mcp-remote https://robotsgate.mike-tusa.workers.dev/mcp. VS Code uses a servers key instead of mcpServers.
The server.json file is the MCP Registry manifest for this server.
Requires Node.js 20+ and a Cloudflare account.
npm install
npx wrangler dev # local dev serverwrangler.toml is a template: create your own KV namespaces / D1 databases / rate-limit namespaces and replace the <PLACEHOLDER> values before npx wrangler deploy. Any secrets the Worker reads go in wrangler secret put (or .dev.vars locally), never in the repo.
MIT (c) Digital Promo Hub. See LICENSE.
This server cannot be deployed
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Check which AI crawlers a site allows and see pages the way AI crawlers do. Free, read-only.
AI-search-visibility tools: check crawler access, generate llms.txt, lint robots.txt, census data.
Agent-readiness scanner (0-5 score), robots.txt + llms.txt generators, managed agent enablement.
