Skip to main content
Glama
spider-rs

spider-cloud-mcp

by spider-rs

spider_ai_crawl

Crawl websites guided by natural language prompts. Specify the content to find, and the AI directs the crawl to extract relevant data.

Instructions

AI-guided crawling using natural language prompts. Describe what content to find and Spider's AI will guide the crawl. REQUIRES an active AI subscription plan (https://spider.cloud/ai/pricing).

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to crawl
limitNoMaximum pages to crawl
promptYesNatural language prompt to guide the crawl (e.g. 'Find all product pages and extract pricing')
cookiesNoHTTP cookies
requestNoRequest type
proxy_enabledNoEnable premium proxies
return_formatNoOutput format
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of disclosing behaviors. It discloses a key operational constraint—the mandatory AI subscription plan—which is not visible in the schema or annotations. It also states that the AI 'will guide the crawl,' signaling autonomous behavior. Yet it could elaborate on what happens without the plan or how AI decisions affect outcomes.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is exceptionally concise: two sentences and a URL. The core action and differentiator appear in the first sentence, with the subscription requirement in the second. Every word earns its place without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 7 parameters, no output schema, and AI-driven behavior, the description is adequate but sparse. It covers the primary capability and the subscription prerequisite, but does not explain what a successful response looks like, how the prompt guides the crawl in practice, or any error/edge-case behavior. More context would improve operational confidence.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all parameters clearly. The description adds a modest reinforcement of the 'prompt' role ('Describe what content to find and Spider's AI will guide the crawl'), but does not provide significant new parameter-level meaning beyond what the schema offers.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'crawling' with a clear resource ('AI-guided') and method ('using natural language prompts'). It immediately distinguishes itself from the sibling tool 'spider_crawl' by emphasizing AI guidance and prompt-driven content discovery.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It clearly establishes the usage context: when you need AI-guided crawling based on natural language descriptions. The explicit 'REQUIRES an active AI subscription plan' provides a critical prerequisite, implying that without such a plan this tool should not be used. However, it does not explicitly compare against alternatives like spider_crawl or state when to choose one over the other.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server

Other Tools

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/spider-rs/spider-cloud-mcp-server'

If you have feedback or need assistance with the MCP directory API, please join our Discord server