robots_check
Fetch and analyze a site's robots.txt and llms.txt to report which major AI crawlers are allowed or blocked, along with sitemaps.
Instructions
Fetch a site's robots.txt + llms.txt and report whether the major AI crawlers are allowed/blocked, plus sitemaps. POST { url }; unpaid returns 402. (PAID x402 service — USDC on Base; the MCP server needs a funded wallet to settle.)
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Any URL on the site |