Robots.txt Check
robots_checkAnswer 'may this user-agent crawl this URL?' by fetching and parsing the site's robots.txt: allowed flag, crawl-delay, request-rate and declared sitemaps. Pay per call with USDC or USDT on Base via x402. 3 free calls per wallet.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Full URL to test. | |
| wallet | No | Your EVM wallet address (0x...). Unlocks the free tier (3 free calls per wallet). | |
| api_key | No | Optional AMR enterprise API key. | |
| x_payment | No | Optional signed x402 payment payload (base64 JSON) for USDC/USDT on Base. | |
| user_agent | No | UA string to evaluate (default *). |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| token | No | Token actually used for payment (USDC, USDT or FREE). | USDC |
| source | Yes | ||
| allowed | Yes | ||
| routing | Yes | How the request was routed (provider, fallback, cache, latency). | |
| sitemaps | No | ||
| cost_usdc | Yes | Price charged in stablecoin units (USDC/USDT 1:1); 0 on the free tier. | |
| fetched_at | No | UTC timestamp when the data was fetched. | |
| user_agent | Yes | ||
| path_checked | Yes | ||
| request_rate | No | ||
| robots_txt_url | Yes | ||
| robots_txt_found | Yes | ||
| freshness_seconds | Yes | Age of the underlying data in seconds (0 = fetched live). | |
| robots_status_code | No | ||
| x_payment_response | No | Base64-encoded x402 settlement receipt (mirrors the X-PAYMENT-RESPONSE header). Only present when the call was paid with `x_payment`. | |
| crawl_delay_seconds | No |