datasets_llms_search
Search the AI-readiness (llms.txt) dataset. Searches the AI-readiness census of the top 1M sites (dataset id enum value llms) — one record per site describing whether it serves a real /llms.txt or /llms-full.txt (who produced it…
Instructions
Search the AI-readiness (llms.txt) dataset. Searches the AI-readiness census of the top 1M sites (dataset id enum value llms) — one record per site describing whether it serves a real /llms.txt or /llms-full.txt (who produced it, what it links to and mentions) and what its /robots.txt says to 17 AI crawlers, next to a homepage control request. Search results omit the file bodies; read one site with the item endpoint. blocked and error states mean unknown, never absent. Sort enum: relevance, rank_asc, links_desc, bytes_desc, domain_asc, crawled_desc. state and robots_state enum: present, absent, blocked, error, skipped. source_kind enum: shopify_agent_template, seo_plugin, platform_default, parked, custom. generator enum: yoast, aioseo, rankmath, wix, shopify, parked. robots_default enum: allowed, partial, blocked. content_signal_train enum: yes, no. Crawler enum for blocks_all, blocks_any and not_blocked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot. Site owners can ask for removal at privacy@crawlora.net. Billing: this search costs 1 credit per record returned, with a minimum of 2 (an empty result costs 2, a full page of 100 costs 100).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| q | No | Substring match on the site domain, max 256 characters | |
| bot | No | AI crawler named inside a valid /llms.txt, e.g. GPTBot (see facet=bot for the values) | |
| tld | No | Exact top-level-domain filter, e.g. com, org, io | |
| page | No | Page number, defaults to 1 | |
| sort | No | Sort enum: relevance, rank_asc, links_desc, bytes_desc, domain_asc, crawled_desc | |
| text | No | Full-text match on the /llms.txt title and summary, max 256 characters | |
| state | No | /llms.txt state. Enum: present, absent, blocked, error, skipped. blocked and error mean unknown, not absent | |
| valid | No | true keeps only sites serving a valid /llms.txt, false keeps the rest | |
| run_id | No | Scan run id; defaults to the latest run | |
| category | No | Exact category label carried by the source list; the top-1M run has none, so this is empty for it | |
| generator | No | Generator detected in a valid /llms.txt. Enum: yoast, aioseo, rankmath, wix, shopify, parked | |
| injection | No | true keeps only valid /llms.txt files containing instruction-override text aimed at AI readers | |
| min_links | No | Minimum number of links in a valid /llms.txt, 0 or greater | |
| page_size | No | Page size, defaults to 20 and maxes at 100; page * page_size must be <= 10000 | |
| reachable | No | true keeps only sites whose homepage control request answered, false only sites that did not | |
| blocks_all | No | Repeatable AI crawler that robots.txt must block (AND). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot | |
| blocks_any | No | Repeatable AI crawler; robots.txt must block at least one (OR). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot | |
| full_valid | No | true keeps only sites serving a valid /llms-full.txt | |
| not_blocked | No | Repeatable AI crawler that robots.txt must NOT block. Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot | |
| placeholder | No | true keeps only valid /llms.txt files that still hold template placeholder text | |
| source_kind | No | Producer of a valid /llms.txt. Enum: shopify_agent_template, seo_plugin, platform_default, parked, custom | |
| home_blocked | No | true keeps only sites whose homepage was walled off from the scan (a block or challenge) | |
| mentions_mcp | No | true keeps only valid /llms.txt files that mention the Model Context Protocol | |
| robots_state | No | /robots.txt state. Enum: present, absent, blocked, error, skipped | |
| all_ai_blocked | No | true keeps only sites whose robots.txt blocks all 17 tracked AI crawlers | |
| robots_default | No | Default stance of robots.txt toward AI crawlers. Enum: allowed, partial, blocked | |
| mentions_openapi | No | true keeps only valid /llms.txt files that mention OpenAPI or Swagger | |
| is_infrastructure | No | false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but serve no public homepage; true keeps only those | |
| min_train_blocked | No | Minimum number of AI training crawlers (of 11 tracked) that robots.txt blocks, 0 or greater | |
| content_signal_train | No | Content-Signal ai-train value declared in robots.txt. Enum: yes, no |