robots_check
Check if major AI crawlers are allowed or blocked by fetching a site's robots.txt and llms.txt, and also retrieve sitemaps.
Instructions
Fetch a site's robots.txt + llms.txt and report whether the major AI crawlers are allowed/blocked, plus sitemaps. POST { url }; unpaid returns 402. (PAID x402 service — USDC on Base; the MCP server needs a funded wallet to settle.)
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Any URL on the site |