agent-access-check
Determines if a website is agent-friendly by analyzing robots.txt, .well-known/ai.txt, sitemap.xml, and HTTP headers, providing a verdict for safe scraping.
Instructions
Checks whether a website is accessible and agent-friendly. Fetches robots.txt, .well-known/ai.txt, and sitemap.xml; inspects HTTP headers (CORS, CSP, rate-limit); and returns a readiness verdict. Useful for agents that need to decide whether to scrape, crawl, or interact with a domain before committing to a workflow. Returns allowed/blocked status, disallowed paths, crawl delay, AI-specific rules, and sitemap URL if present.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | Domain or URL to check. Can be a bare domain (example.com) or full URL (https://example.com/path). |