robots.txt crawl permission check
robots_txt_checkCrawl preflight: whether each user agent may fetch each URL under the site's robots.txt (RFC 9309, longest match, '*' fallback), with crawl-delay, the sitemaps robots.txt lists and whether the site publishes /llms.txt. Default agents: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot, Applebot-Extended, Bytespider and *. Paid only: up to 5 URLs per call for $0.001 USDC. Paid calls made without a payment return isError with the x402 v2 PaymentRequired object (amount in USDC atomic units, network, payTo, HTTP endpoint); retry with the signed PaymentPayload in params._meta["x402/payment"]. Invalid input and calls where every item fails are never charged. Fetches follow robots.txt, use the User-Agent AdamHaleTools/1.0 and refuse private or internal addresses.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | 1-5 URLs to check. | |
| agents | No | Optional user-agent product tokens, e.g. ["GPTBot", "*"]. | |
| x402_payment | No | Optional, paid tier only: a signed x402 v2 PaymentPayload, base64 (the value an HTTP client sends as the PAYMENT-SIGNATURE header). Prefer params._meta["x402/payment"]. Never a private key or seed phrase. |