datasets_llms_facets
Facet the AI-readiness (llms.txt) dataset. Returns distribution counts over the AI-readiness census (dataset id enum value llms), honoring the same filters as search — adoption by producer, generator, TLD or category, and which AI…
Instructions
Facet the AI-readiness (llms.txt) dataset. Returns distribution counts over the AI-readiness census (dataset id enum value llms), honoring the same filters as search — adoption by producer, generator, TLD or category, and which AI crawlers robots.txt blocks. Facet enum: state, source_kind, generator, bot, policy, tld, category, robots_state, robots_default, blocked_crawler, partial_crawler, content_signal_train, full_state, home_status. The filter enums are the same as search. Counts include blocked and error probes, which are unknown rather than absent; filter state=present or state=absent to count only sites that were established either way.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| q | No | Substring match on the site domain, max 256 characters | |
| bot | No | AI crawler named inside a valid /llms.txt, e.g. GPTBot (see facet=bot for the values) | |
| tld | No | Exact top-level-domain filter, e.g. com, org, io | |
| text | No | Full-text match on the /llms.txt title and summary, max 256 characters | |
| facet | Yes | Facet enum: state, source_kind, generator, bot, policy, tld, category, robots_state, robots_default, blocked_crawler, partial_crawler, content_signal_train, full_state, home_status | |
| state | No | /llms.txt state. Enum: present, absent, blocked, error, skipped. blocked and error mean unknown, not absent | |
| valid | No | true keeps only sites serving a valid /llms.txt, false keeps the rest | |
| run_id | No | Scan run id; defaults to the latest run | |
| category | No | Exact category label carried by the source list; the top-1M run has none, so this is empty for it | |
| generator | No | Generator detected in a valid /llms.txt. Enum: yoast, aioseo, rankmath, wix, shopify, parked | |
| injection | No | true keeps only valid /llms.txt files containing instruction-override text aimed at AI readers | |
| min_links | No | Minimum number of links in a valid /llms.txt, 0 or greater | |
| reachable | No | true keeps only sites whose homepage control request answered, false only sites that did not | |
| blocks_all | No | Repeatable AI crawler that robots.txt must block (AND). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot | |
| blocks_any | No | Repeatable AI crawler; robots.txt must block at least one (OR). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot | |
| full_valid | No | true keeps only sites serving a valid /llms-full.txt | |
| not_blocked | No | Repeatable AI crawler that robots.txt must NOT block. Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot | |
| placeholder | No | true keeps only valid /llms.txt files that still hold template placeholder text | |
| source_kind | No | Producer of a valid /llms.txt. Enum: shopify_agent_template, seo_plugin, platform_default, parked, custom | |
| home_blocked | No | true keeps only sites whose homepage was walled off from the scan (a block or challenge) | |
| mentions_mcp | No | true keeps only valid /llms.txt files that mention the Model Context Protocol | |
| robots_state | No | /robots.txt state. Enum: present, absent, blocked, error, skipped | |
| all_ai_blocked | No | true keeps only sites whose robots.txt blocks all 17 tracked AI crawlers | |
| robots_default | No | Default stance of robots.txt toward AI crawlers. Enum: allowed, partial, blocked | |
| mentions_openapi | No | true keeps only valid /llms.txt files that mention OpenAPI or Swagger | |
| is_infrastructure | No | false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but serve no public homepage; true keeps only those | |
| min_train_blocked | No | Minimum number of AI training crawlers (of 11 tracked) that robots.txt blocks, 0 or greater | |
| content_signal_train | No | Content-Signal ai-train value declared in robots.txt. Enum: yes, no |