Spider Cloud MCP Server
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Spider Cloud MCP ServerScrape the latest blog posts from https://example.com/blog"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Spider Cloud MCP Server
A high-performance Model Context Protocol (MCP) server that provides comprehensive web scraping, crawling, and data extraction capabilities through the Spider Cloud API. This server enables AI assistants like Claude to interact with web content using Spider Cloud's advanced scraping infrastructure.
π Features
Core Tools
spider_scrape- Advanced single-page scraping with JavaScript rendering and anti-bot bypassspider_crawl- Intelligent website crawling with depth control and filteringspider_search- Google-like web search with content fetching capabilitiesspider_links- Comprehensive link extraction and analysisspider_screenshot- High-quality webpage screenshots with customizationspider_transform- HTML to markdown/text conversion with readability processing
Advanced Capabilities
π‘οΈ Anti-bot Detection Bypass - Stealth mode and advanced evasion techniques
π Premium Proxy Support - Geographic targeting with country-specific proxies
π JavaScript Rendering - Full browser emulation for dynamic content
π Metadata Extraction - Comprehensive page metadata and analytics
π CSS Selectors - Precise content targeting and extraction
πΎ Cloud Storage - Optional data persistence in Spider Cloud
β‘ High Performance - Optimized for speed with configurable timeouts
π Secure Authentication - Bearer token authentication with API key
π Cost Tracking - Real-time API usage cost monitoring
π Debug Mode - Comprehensive logging for troubleshooting
Related MCP server: FineData MCP Server
π Prerequisites
Node.js 18 or higher
Spider Cloud API key (Get one free at spider.cloud)
MCP-compatible client (Claude Desktop, Claude Code, Cursor, etc.)
π Quick Start
Option 1: Install from npm (Recommended)
# Global installation
npm install -g @willbohn/spider-mcp
# Or use with npx (no installation needed)
npx @willbohn/spider-mcpOption 2: Clone from GitHub
Windows:
# Clone and install
git clone https://github.com/willbohn/spider-mcp.git
cd spider-mcp
# Run the Windows installer (PowerShell)
.\install-windows.ps1
# Or use the batch file (Command Prompt)
install-windows.bat
# Test the installation
$env:SPIDER_API_KEY="your_key"
node test.jsmacOS/Linux:
# Clone and install
git clone https://github.com/willbohn/spider-mcp.git
cd spider-mcp
./install-local.sh
# Or manually:
npm install
npm link
# Test the installation
SPIDER_API_KEY=your_key node test.jsOption 3: Direct Path Configuration
Skip installation and point directly to the built files in your MCP client configuration.
βοΈ Configuration
Platform-Specific Setup Instructions
Claude Desktop (Windows)
Find your configuration file:
Press
Win + R, type%APPDATA%\Claudeand press EnterOpen
claude_desktop_config.json(create it if it doesn't exist)
Add the Spider MCP configuration:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["@willbohn/spider-mcp"],
"env": {
"SPIDER_API_KEY": "your_spider_api_key_here"
}
}
}
}Alternative: Using direct path (if npm doesn't work):
{
"mcpServers": {
"spider": {
"command": "node",
"args": ["C:\\Users\\YourName\\spider-mcp\\dist\\index.js"],
"env": {
"SPIDER_API_KEY": "your_spider_api_key_here"
}
}
}
}Note: On Windows, use double backslashes (\\) in paths or forward slashes (/).
Testing on Windows
# PowerShell
$env:SPIDER_API_KEY="your_key"
node test.js
# Command Prompt
set SPIDER_API_KEY=your_key
node test.jsClaude Desktop (macOS)
Find your configuration file:
open ~/Library/Application\ Support/Claude/Open
claude_desktop_config.json(create it if it doesn't exist)Add the Spider MCP configuration:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["@willbohn/spider-mcp"],
"env": {
"SPIDER_API_KEY": "your_spider_api_key_here"
}
}
}
}Testing on macOS
export SPIDER_API_KEY="your_key"
node test.jsClaude Desktop (Linux)
Find your configuration file:
# Location varies by distribution, commonly: ~/.config/Claude/claude_desktop_config.json # or ~/.claude/claude_desktop_config.jsonAdd the Spider MCP configuration:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["@willbohn/spider-mcp"],
"env": {
"SPIDER_API_KEY": "your_spider_api_key_here"
}
}
}
}Testing on Linux
export SPIDER_API_KEY="your_key"
node test.jsOther MCP Clients
Claude Code automatically detects MCP servers. Simply:
Install the package globally:
npm install -g @willbohn/spider-mcpSet your API key:
Windows (PowerShell):
$env:SPIDER_API_KEY="your_key"Windows (CMD):
set SPIDER_API_KEY=your_keymacOS/Linux:
export SPIDER_API_KEY="your_key"
The server will be available in Claude Code
Add to your Cursor settings:
{
"mcp.servers": {
"spider": {
"command": "npx",
"args": ["@willbohn/spider-mcp"],
"env": {
"SPIDER_API_KEY": "your_spider_api_key_here"
}
}
}
}Add to your Continue configuration:
{
"mcpServers": [
{
"name": "spider",
"command": "npx",
"args": ["@willbohn/spider-mcp"],
"env": {
"SPIDER_API_KEY": "your_spider_api_key_here"
}
}
]
}Environment Variables
Variable | Required | Description | Default |
| Yes | Your Spider Cloud API key | - |
| No | API endpoint URL |
|
| No | Request timeout in milliseconds |
|
| No | Enable debug logging |
|
π οΈ Tool Documentation
spider_scrape
Scrape content from a single URL with advanced options.
Parameters:
url(required): Target URL to scrapereturn_format: Output format (markdown,raw,text,html,screenshot,links)js: Enable JavaScript renderingwait_for: Wait time for page load (0-60000ms)css_selector: CSS selector for specific contentproxy_enabled: Use premium proxyproxy_country: Two-letter country codestealth: Enable stealth modeanti_bot: Advanced anti-bot bypassheaders: Custom HTTP headerscookies: Cookie stringmetadata: Include metadataclean_html: Clean and sanitize HTMLmedia: Include media elements
Example:
{
"url": "https://example.com",
"return_format": "markdown",
"js": true,
"stealth": true,
"css_selector": ".main-content"
}spider_crawl
Crawl an entire website with intelligent navigation.
Parameters:
url(required): Starting URLlimit: Max pages to crawl (1-10000)depth: Max crawl depth (0-10)return_format: Output formatwhitelist: URL patterns to includeblacklist: URL patterns to excludebudget: Crawl budget configurationsubdomains: Include subdomainssitemap: Use sitemap.xmlrespect_robots: Respect robots.txtPlus all proxy and rendering options from scrape
Example:
{
"url": "https://docs.example.com",
"limit": 50,
"depth": 3,
"whitelist": ["*/api/*"],
"return_format": "markdown"
}spider_search
Search the web with Google-like results.
Parameters:
query(required): Search querysearch_limit: Max results (1-100)fetch_page_content: Fetch full contenttbs: Time-based search (qdr:d,qdr:w,qdr:m,qdr:y)gl: Country code (e.g.,us,uk)hl: Language code (e.g.,en,es)safe: SafeSearch level (off,medium,high)Plus content fetching options
Example:
{
"query": "artificial intelligence news",
"search_limit": 10,
"tbs": "qdr:w",
"gl": "us",
"fetch_page_content": true
}spider_links
Extract and analyze links from a webpage.
Parameters:
url(required): Target URLlimit: Max links (1-5000)depth: Extraction depth (0-5)unique: Return only unique linkssubdomains: Include subdomain linksexternal: Include external linksPlus standard options
spider_screenshot
Capture webpage screenshots.
Parameters:
url(required): Target URLfullpage: Full page screenshotviewport_width: Width in pixels (320-3840)viewport_height: Height in pixels (240-2160)format: Image format (png,jpeg,webp)quality: JPEG/WebP quality (0-100)omit_background: Transparent background (PNG only)clip: Region to capture
spider_transform
Transform HTML to clean, readable formats.
Parameters:
data(required): HTML/text to transformreturn_format(required): Target format (markdown,text,raw,clean_html)readability: Apply readability processingclean: Remove unnecessary elementsinclude_links: Include hyperlinksinclude_images: Include images
π§ͺ Testing
Run the comprehensive test suite:
Windows
# PowerShell
$env:SPIDER_API_KEY="your_api_key_here"
node test.js
# With debug output
$env:DEBUG="true"
$env:SPIDER_API_KEY="your_api_key_here"
node test.js
# Command Prompt
set SPIDER_API_KEY=your_api_key_here
node test.jsmacOS/Linux
# Set your API key
export SPIDER_API_KEY=your_api_key_here
# Run tests
node test.js
# With debug output
DEBUG=true SPIDER_API_KEY=your_api_key_here node test.jsTest Suites
# Quick smoke tests
npm run test:quick
# Full comprehensive suite (100+ tests)
npm run test:full
# LinkedIn-specific tests
npm run test:linkedin
# Run specific category
npm run test:category -- --category scrapingπ API Response Format
All tools return responses in a consistent format:
{
"success": true,
"results": [...],
"count": 10,
"costs": {
"total_cost": 0.00012,
"compute_cost": 0.00008,
"bandwidth_cost": 0.00004
},
"metadata": {
"duration": 1234,
"status": 200
}
}π§ Development
Building from Source
npm install
npm run buildRunning in Development Mode
npm run devProject Structure
spider-mcp/
βββ src/
β βββ index.ts # Main server implementation
βββ dist/ # Compiled JavaScript
βββ examples/ # Configuration examples
βββ package.json # Dependencies and scripts
βββ tsconfig.json # TypeScript configuration
βββ README.md # This fileπ Troubleshooting
Common Issues
"SPIDER_API_KEY environment variable is required"
Ensure your API key is set in the environment or configuration
Check the key is valid at spider.cloud
"Payment required" error
Your API key needs credits
Add credits at spider.cloud
"Rate limit exceeded"
You've hit the API rate limit
Wait a few minutes or upgrade your plan
Search tool timeout
Search operations can take 15-30 seconds
This is normal behavior for comprehensive searches
Debug Mode
Enable detailed logging:
Windows (PowerShell):
$env:DEBUG="true"
$env:SPIDER_API_KEY="your_key"
node dist/index.jsWindows (Command Prompt):
set DEBUG=true
set SPIDER_API_KEY=your_key
node dist/index.jsmacOS/Linux:
DEBUG=true SPIDER_API_KEY=your_key node dist/index.jsπ Error Handling
The server provides detailed error messages:
401: Invalid API key
402: Payment required (add credits)
429: Rate limit exceeded
500+: Server errors (contact support)
π Security
API keys are never logged or stored
All requests use HTTPS
Bearer token authentication
Input validation on all parameters
Sanitized error messages
π Performance
Configurable timeouts (default: 60s)
Automatic retry logic for transient failures
Connection pooling for efficiency
Response caching at API level
Optimized for concurrent requests
π€ Contributing
Contributions are welcome! Please:
Fork the repository
Create a feature branch
Commit your changes
Push to the branch
Open a Pull Request
π License
MIT License - see LICENSE file for details
π Resources
π¬ Support
MCP Server Issues: GitHub Issues
Spider API Support: spider.cloud/support
API Status: status.spider.cloud
Built with β€οΈ for the MCP ecosystem
Available Tools
6 toolsspider_crawlC
Crawl an entire website with configurable depth, limits, and filtering options
| Name | Required | Description | Default |
|---|---|---|---|
| js | No | ||
| url | Yes | ||
| depth | No | ||
| limit | No | ||
| budget | No | ||
| headers | No | ||
| sitemap | No | ||
| stealth | No | ||
| anti_bot | No | ||
| metadata | No | ||
| blacklist | No | ||
| whitelist | No | ||
| store_data | No | ||
| subdomains | No | ||
| proxy_country | No | ||
| proxy_enabled | No | ||
| return_format | No | ||
| respect_robots | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It doesn't disclose that this is a long-running, high-cost operation, rate limiting, permission requirements, or what 'store_data' actually mutates in Spider Cloud β significant gaps for an 18-parameter mutation-capable crawl.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no waste, though it is arguably too terse for an 18-parameter tool. Efficient phrasing that earns its place but under-delivers given the complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 18-parameter crawl with no annotations and no output schema, the description covers almost none of what an agent needs: cost/time expectations, parameter interactions, or the effect of store_data. Inadequate for the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is reported 0% (raw schema has empty descriptions), so the description must compensate. It mentions 'depth, limits, and filtering options' generically, leaving 15+ parameters β budget, blacklist/whitelist wildcards, stealth/anti_bot, proxy, store_data β completely unexplained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Crawl) and resource (entire website) and lists the dimensions of configurability. It distinguishes itself from spider_scrape (single page) and spider_search (query-based) by scope, though it doesn't name those siblings explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implied usage: crawl an 'entire website' when you need multi-page coverage versus a single scrape. No explicit when-to-use, when-not-to-use, or named alternatives, so the agent must infer routing among the five siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spider_linksC
Extract and analyze all links from a webpage with filtering options
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| depth | No | ||
| limit | No | ||
| unique | No | ||
| external | No | ||
| metadata | No | ||
| store_data | No | ||
| subdomains | No | ||
| proxy_country | No | ||
| proxy_enabled | No | ||
| return_format | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden and mostly fails it. It does not disclose that store_data writes to Spider Cloud, whether proxy_enabled incurs cost or latency, or any rate/limit behavior β all of which matter for an 11-parameter tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single clean sentence is front-loaded and free of waste, but for a tool with 11 parameters it is under-specified rather than genuinely concise; the sentence has no follow-up detail to justify its brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An 11-parameter, no-annotation, no-output-schema tool needs substantially more context than one sentence. Key behaviors β what 'analyze' means, how depth interacts with limit, what store_data persists, what return_format changes β are entirely absent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 11 parameters and effectively no substantive description coverage, the description is the only place to add meaning, yet 'with filtering options' names none of the filters (depth, limit, unique, external, subdomains, proxy settings, return_format). The individual schema descriptions exist but are terse one-liners that do not convey interaction or defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb+resource ('Extract and analyze all links from a webpage'), which is clearly distinguishable from spider_scrape, spider_crawl, and spider_search. The word 'analyze' is vaguer than 'extract,' but the core purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus spider_scrape or spider_crawl, both of which also operate on a URL. 'With filtering options' hints at capability but gives no conditions, prerequisites, or alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spider_scrapeC
Scrape content from a single URL with advanced options for JavaScript rendering, anti-bot bypass, and content extraction
| Name | Required | Description | Default |
|---|---|---|---|
| js | No | ||
| url | Yes | ||
| media | No | ||
| cookies | No | ||
| headers | No | ||
| stealth | No | ||
| timeout | No | ||
| anti_bot | No | ||
| metadata | No | ||
| wait_for | No | ||
| clean_html | No | ||
| store_data | No | ||
| css_selector | No | ||
| proxy_country | No | ||
| proxy_enabled | No | ||
| return_format | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden, and it discloses almost nothing: no note that premium proxy/stealth modes may carry cost or latency, no authentication or Spider Cloud credential requirement, no rate limits, and no statement of what happens to the page content when store_data persists it remotely. The three 'advanced options' named are capabilities rather than behaviors.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One front-loaded sentence with no filler; the core action leads and the qualifiers follow. It is efficient, though it spends its back half listing three features that the schema already documents, which is space that could have bought usage guidance.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 16-parameter, no-annotation, no-output-schema tool, this is thin: nothing about the default markdown return, nothing about cost or credential prerequisites, nothing about how it differs from the five sibling tools. An agent can call it, but cannot confidently choose it over spider_crawl or predict what comes back.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema itself carries a per-parameter description for all 16 fields (js, anti_bot, css_selector, timeout bounds, return_format default markdown, etc.), so the schema does the heavy lifting even though the reported coverage signal reads 0%. The description adds essentially no parameter meaning beyond echoing the js, anti_bot, and extraction options already named in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb+resource ('Scrape content from a single URL') and enumerates the capability areas: JavaScript rendering, anti-bot bypass, content extraction. This separates it from the plural-scope siblings (crawl/search), but it never names those siblings, so an agent must infer the boundary itself.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no when-to-use guidance, no prerequisites, and no mention of alternatives such as spider_crawl for multi-page work or spider_screenshot when an image is wanted. The word 'single URL' is the only implicit routing signal, and it is embedded in the purpose statement rather than framed as a decision rule.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spider_screenshotC
Capture high-quality screenshots of webpages with customizable viewport and options
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| clip | No | ||
| format | No | ||
| quality | No | ||
| timeout | No | ||
| fullpage | No | ||
| wait_for | No | ||
| proxy_country | No | ||
| proxy_enabled | No | ||
| viewport_width | No | ||
| omit_background | No | ||
| viewport_height | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full disclosure burden, yet it only says 'high-quality' and 'customizable'. It omits whether proxies are required for some sites, how rendering/waiting behaves, rate limits, or that output is an image rather than content.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler or repetition. It is efficient, though its brevity reflects under-specification rather than disciplined selection of detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 12-parameter tool with no annotations and no output schema, the description is far too thin. It says nothing about proxy usage, timing behavior, or the nature of the returned image, so an agent must infer everything from the schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds only the vague phrase 'customizable viewport and options' and explains no parameter. Although the input schema itself carries per-parameter descriptions, the description contributes no semantics beyond it, leaving the description-level meaning at a minimum.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Capture ... screenshots of webpages') with a hint at configurable scope. It is clearly distinct from siblings like spider_scrape and spider_links, though it never names or contrasts them explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this versus spider_scrape/spider_crawl, no prerequisites, and no mention of when a screenshot is inappropriate. The only implied usage is taking a visual capture of a page.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spider_searchC
Search the web with Google-like results and optional page content fetching
| Name | Required | Description | Default |
|---|---|---|---|
| gl | No | ||
| hl | No | ||
| num | No | ||
| tbs | No | ||
| safe | No | ||
| query | Yes | ||
| start | No | ||
| metadata | No | ||
| store_data | No | ||
| search_limit | No | ||
| proxy_country | No | ||
| proxy_enabled | No | ||
| return_format | No | ||
| fetch_page_content | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are supplied, so the description carries the entire burden. It discloses only that page fetching is optional; it says nothing about the fact that store_data writes results into Spider Cloud (a mutating side effect), nor about proxy usage, rate limits, or credential requirements. That is a meaningful gap for a 14-parameter tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler, which is structurally clean. It is arguably under-sized for the surface area it must cover, but there is no verbosity to trim.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 14 parameters, no annotations, no output schema, and no parameter documentation, one sentence is far short of complete. The agent lacks the information needed to choose between the search, scrape, links, and crawl siblings or to know what the response contains.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 14 parameters and reported schema description coverage of 0%, the description must compensate, and it does not. The only parameter behavior it touches is fetch_page_content ("optional page content fetching"); geo/language, pagination (start, num, search_limit), tbs, safe, return_format, and the proxy/store flags are all left unexplained.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ("Search the web") and adds the shape of the result ("Google-like results") plus the optional content-fetch behavior. However it never distinguishes itself from spider_scrape, spider_links, or spider_crawl, so an agent must infer the boundary on its own.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus the five sibling tools, and no mention of prerequisites or exclusions. The only usage signal is the phrase "optional page content fetching," which hints at a capability but not a selection criterion.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
spider_transformC
Transform HTML content to clean markdown, text, or formatted output
| Name | Required | Description | Default |
|---|---|---|---|
| data | Yes | ||
| clean | No | ||
| readability | No | ||
| include_links | No | ||
| return_format | Yes | ||
| include_images | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full behavioral burden, yet it discloses only the transformation targets. It does not state that this is a pure, side-effect-free operation, nor does it explain what 'clean', 'readability', or 'formatted output' actually do to the input.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no wasted words. It is efficient, though its brevity is partly the cause of the coverage gaps rather than a deliberate trade-off.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 6-parameter tool with no annotations and no output schema, the description should explain transformation behavior and the optional flags. Instead it only restates the high-level purpose, leaving the agent with no guidance on how the options interact or what the result looks like.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is reported at 0%, and the description adds no parameter-level meaning. It mentions markdown and text (partially mapping to return_format) but says nothing about the other options (raw, clean_html) or the clean/readability/include_links/include_images toggles.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Transform'), a specific resource ('HTML content'), and names the output target formats (markdown, text, formatted output). An agent can tell it is a conversion utility, though it never explicitly contrasts itself with siblings like spider_scrape or spider_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus alternatives, no prerequisites, and no mention of when it should not be used. The agent must infer that this is for post-scrape content conversion rather than retrieval.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
6 tool updates
v1.1.0- First observed
spider_crawl - First observed
spider_links - First observed
spider_scrape - First observed
spider_screenshot - First observed
spider_search - First observed
spider_transform
TDQS
Scored across 6 tools
Each tool targets a distinct web operation: scrape single URL, crawl site, search web, extract links, screenshot, and transform HTML. Overlap is minimal and descriptions clarify boundaries.
All tools use the same 'spider_' prefix followed by a clear verb/noun, creating a fully predictable naming pattern.
Six tools are well-scoped for a web scraping and content processing server, covering common workflows without bloat.
Core lifecycle is covered: discovery/search, extraction/scraping, crawling, link analysis, screenshots, and transformation. Minor gaps exist, such as batch operations or structured data extraction, but agents can work around them.
Maintenance
Related MCP Connectors
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabiliβ¦
Crawl, scrape, search the web, and automate browsers at scale with anti-bot bypass.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables AI assistants to access real-time web data through search, markdown scraping, and browser automation while bypassing anti-bot protections. It provides tools for web research, e-commerce monitoring, and data extraction from across the globe.47,023 npm5MIT
- AlicenseAqualityBmaintenanceEnables AI agents to scrape any website by providing tools for JavaScript rendering, antibot bypass, and automatic captcha solving. It supports synchronous, asynchronous, and batch scraping operations with built-in proxy rotation.5207 npmMIT
- AlicenseNot gradedqualityDmaintenanceProvides AI agents and assistants with advanced web crawling and RAG capabilities, enabling them to scrape websites and perform semantic search over crawled content.1MIT
- AlicenseAqualityDmaintenanceEnables AI agents to crawl, scrape, search, and automate browsers with anti-bot bypass, providing fast web access via 22 tools.2233 npm3MIT