crawl permission
crawl_permissionMay my agent fetch, index, train on or use this page? Reads robots.txt for your user agent (longest-match rules, crawl-delay), Content-Signal (search / ai-input / ai-train), X-Robots-Tag and meta robots (noai, noindex), TDM reservation and ai.txt, and lists which AI crawlers the site allows. Costs $0.002 USDC per call (x402).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| user_agent | No | e.g. GPTBot, ClaudeBot, MyAgent/1.0 (default *) |