"Roots" matching MCP connectors:

Checks if a URL path is allowed by robots.txt. x402 payment required (testnet USDC).

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.

robots.txt Allow count, body discarded

robots.txt group count

URL to clean markdown for LLMs: a polite, robots.txt-respecting web reader. Free, no API key