robots_check
Read a domain's robots.txt as crawlers do: raw file, parsed user-agent groups, Sitemap lines, and an allowed/blocked matrix for search and AI bots (Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-SearchBot, Google-Extended, CCBot, Applebot, Bytespider…) over the homepage plus any urls given. Also reports whether /llms.txt exists and its shape. Any public domain.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| bots | No | Restrict to these user agents (JSON array or comma-separated). Default: the full search + AI list | |
| urls | No | Paths or URLs to test per bot (JSON array or comma-separated, max 25). The homepage is always included | |
| detail | No | Output verbosity: "concise" (default) trims long lists to top-25 and caps size; "detailed" returns the full result. | |
| domain | Yes | Domain or URL, e.g. navyaai.com |