"author:crawlerpeek" matching MCP servers:

Exposes three read-only MCP tools that let an assistant check whether specified AI crawlers are allowed by a site's robots.txt and how a URL responds when fetched with each bot's user-agent, fetch a page as a single named bot, and generate a robots.txt that welcomes AI crawlers.