robots-check
Robots.txt compliance check for crawlers and AI agents. Fetch a public site robots.txt and determine whether a user-agent may crawl a path; return the matched allow/disallow rule, crawl policy, HTTP status and declared sitemap URLs. Costs $0.001 USDC on Base via x402.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| path | No | ||
| user_agent | No |