Crawlora MCP

datasets_llms_search

Read-only

Search the AI-readiness dataset — which top-1M sites serve a real llms.txt or llms-full.txt (by producer, links, MCP/OpenAPI mentions) and what their robots.txt says to 17 AI crawlers. Costs 1 credit per record returned (minimum 2), so keep page_size to what you need.

Input Schema

Name	Required	Description
`q`	No	Optional substring match on the site domain, max 256 characters.
`bot`	No	Optional AI crawler named inside a valid /llms.txt, e.g. GPTBot, ClaudeBot. See facet=bot for the values.
`tld`	No	Optional exact top-level-domain filter, e.g. com, org, io.
`page`	No	Result page number, 1-based, default 1; page times page_size must not exceed 10000.
`sort`	No	Optional sort order. Allowed values: relevance, rank_asc, links_desc, bytes_desc, domain_asc, crawled_desc. Defaults to relevance with q or text, otherwise rank_asc.
`text`	No	Optional full-text match on the llms.txt title and summary, max 256 characters.
`state`	No	Optional /llms.txt state. Allowed values: present, absent, blocked, error, skipped. blocked and error mean unknown, not absent.
`valid`	No	Optional filter; true keeps only sites serving a valid /llms.txt, false keeps the rest.
`run_id`	No	Optional scan run id; defaults to the latest run.
`category`	No	Optional exact category label carried by the source list. The top-1M run has none, so this is empty for it.
`generator`	No	Optional generator detected in a valid /llms.txt. Allowed values: yoast, aioseo, rankmath, wix, shopify, parked.
`injection`	No	Optional filter for a valid /llms.txt containing instruction-override text aimed at AI readers.
`min_links`	No	Optional minimum number of links in a valid /llms.txt, 0 or greater.
`page_size`	No	Page size, default 20, max 100; page times page_size must not exceed 10000.
`reachable`	No	Optional filter; true keeps only sites whose homepage control request answered, false only sites that did not.
`blocks_all`	No	Repeatable AI crawler that robots.txt must block (AND). Allowed values: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot.
`blocks_any`	No	Repeatable AI crawler; robots.txt must block at least one (OR). Same allowed values as blocks_all.
`full_valid`	No	Optional filter; true keeps only sites serving a valid /llms-full.txt.
`not_blocked`	No	Repeatable AI crawler that robots.txt must NOT block. Same allowed values as blocks_all.
`placeholder`	No	Optional filter for a valid /llms.txt that still holds template placeholder text.
`source_kind`	No	Optional producer of a valid /llms.txt. Allowed values: shopify_agent_template, seo_plugin, platform_default, parked, custom.
`home_blocked`	No	Optional filter for sites whose homepage was walled off from the scan (a block or challenge).
`mentions_mcp`	No	Optional filter for a valid /llms.txt that mentions the Model Context Protocol.
`robots_state`	No	Optional /robots.txt state. Allowed values: present, absent, blocked, error, skipped.
`all_ai_blocked`	No	Optional filter; true keeps only sites whose robots.txt blocks all 17 tracked AI crawlers.
`robots_default`	No	Optional default stance of robots.txt toward AI crawlers. Allowed values: allowed, partial, blocked.
`mentions_openapi`	No	Optional filter for a valid /llms.txt that mentions OpenAPI or Swagger.
`is_infrastructure`	No	Optional filter; false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but serve no public homepage; true keeps only those.
`min_train_blocked`	No	Optional minimum number of AI training crawlers (of 11 tracked) that robots.txt blocks, 0 or greater.
`content_signal_train`	No	Optional Content-Signal ai-train value declared in robots.txt. Allowed values: yes, no.

Output Schema

Name	Required	Description	Default
`data`	Yes	The tool result payload (shape varies per tool; see each tool's docs resource).

Schema Changelog

Changes observed during successful MCP inspections.

Changed1 schema field changedOct 11, 2026

changedInput schema / properties / category / description

Previous value: -"Optional exact category filter from the source ranking list."New value: +"Optional exact category label carried by the source list. The top-1M run has none, so this is empty for it."

AddedOct 11, 2026

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare readOnlyHint=true and openWorldHint=true, but the description adds a genuinely valuable behavioral disclosure beyond them: a cost model of 1 credit per returned record with a 2-credit minimum. This is exactly the kind of rate/pricing context annotations cannot carry. It stops short of describing pagination limits (delegated to the schema) so it is not a full 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, both earning their place: the first defines the dataset and its content dimensions; the second carries the cost/paging guidance. The purpose is front-loaded with zero filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 30-parameter, all-optional search tool with a full output schema, the description covers what the dataset is and the cost implications of calling it. The parameter semantics live in the exhaustive schema, and return shape is covered by the output schema, so little is missing; only the routing versus sibling tools is absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents all 30 parameters, giving a baseline of 3. The description adds meaning to page_size by tying it to the credit cost and advising minimal sizing, which is a semantic layer the schema's terse 'Page size, default 20, max 100' does not provide.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Search) and resource (the AI-readiness dataset), then defines the dataset's contents precisely: top-1M sites serving llms.txt/llms-full.txt and what their robots.txt says to 17 AI crawlers. This clearly distinguishes it from siblings like datasets_llms_facets (aggregation) and datasets_llms_item (single record).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives operational guidance ('keep page_size to what you need' because of credit cost), which is real usage advice. However, it never states when to use this search tool versus alternatives such as datasets_llms_facets for counts or datasets_llms_item for a single record, nor any when-not conditions. Usage is implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Crawlora MCP tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.