AI crawler access matrix from robots.txt
ai_crawler_matrixReads a domain's robots.txt (RFC 9309 matching) and reports, for each known AI crawler and AI agent (OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon, Common Crawl and others), whether it may fetch the site root and up to five extra paths, with the deciding rule and any Content-Signal preferences. Use it to see whether a site is open to AI search but closed to AI training. Price: $0.01 per call, paid with x402.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| paths | No | Optional comma-separated extra paths to test, e.g. /blog/,/docs/ | |
| domain | Yes | Domain or URL, e.g. example.com |