Spider MCP Server lets AI agents crawl, scrape, search, extract, and automate the web via Spider's cloud API, with anti-bot bypass and optional AI tools.
Crawl websites: multi-page crawling with depth/limit, JS rendering, proxies, formats, filters, and structured extraction.
Scrape pages: fetch/process a single URL with output formats and screenshots.
Search the web: query with country/language/time filters and optional full-page content.
Extract links/screenshots: discover URLs without fetching; capture base64 PNGs.
Transform HTML: convert existing HTML to markdown/text without web requests.
Unblock protected sites: anti-bot bypass with fingerprinting/proxy rotation (extra credits).
Use AI tools: AI-guided crawl/scrape/search/browser/link filtering (requires AI subscription).
Automate browsers: open session, navigate, click, fill forms, screenshot, get HTML/text, run JS, wait, close session.
Manage credits: check API credit balance; core tools are pay-per-use, no subscription.
Integrates with Windsurf (Codeium's IDE) to enable web scraping and browser automation capabilities within the editor.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Spider MCP Server crawl the React hooks documentation"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
The fastest web crawling, scraping, and browser automation server for AI agents. Gives Claude direct access to the web through 21 tools — crawl sites at 100K+ pages/sec, extract structured data with AI, and control remote browsers with built-in anti-bot bypass.
Speed — Crawl 100K+ pages per second. Smart request routing picks HTTP or headless Chrome automatically. Streaming responses start delivering data immediately.
Cost — Pay-per-use credits with no subscription required for core tools. Check your balance anytime with spider_get_credits. AI tools available with an AI subscription.
Reliability — Anti-bot bypass with fingerprinting and proxy rotation. Browser fleet with automatic fallback across Chrome, Firefox, and more. Built-in retry and stealth escalation.
Related MCP server: The Web MCP
claude mcp add spider -- npx -y spider-cloud-mcpSet your API key:
export SPIDER_API_KEY="your-api-key"Add to claude_desktop_config.json:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Add to .cursor/mcp.json:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Add to ~/.codeium/windsurf/mcp_config.json:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Variable | Required | Description |
| Yes | Your Spider API key. Get one at spider.cloud/api-keys |
These work on pay-per-use credits with no subscription required.
Tool | Description |
| Crawl a website and extract content from multiple pages. Follows links up to a depth/limit. |
| Scrape a single page. Faster and cheaper than crawling when you need one URL. |
| Search the web. Optionally fetch full page content from results. |
| Extract all links from a page without fetching content. |
| Capture a page screenshot as base64 PNG. |
| Convert HTML to markdown or text without making web requests. |
| Check your API credit balance. |
Natural language web interaction. Describe what you want in plain English.
Requires an AI subscription.
Tool | Description |
| AI-guided crawling — describe what content to find. |
| Extract structured data with plain English — no CSS selectors. |
| AI-enhanced search with intent understanding. |
| Automate browser actions with natural language. |
| Find and filter links by description. |
Direct browser control via spider-browser. Browsers run in Spider's cloud with smart retry, browser switching, anti-bot protection, proxy rotation, and automatic stealth escalation. Supports CDP and BiDi protocols.
Tool | Description |
| Open a remote browser session. Returns a session_id. |
| Navigate to a URL and wait for load. |
| Click an element by CSS selector. |
| Fill a form field with text. |
| Take a screenshot of the current page. Returns base64 PNG. |
| Get page HTML or visible text. |
| Execute JavaScript in the page context. |
| Wait for an element, navigation, or network idle. |
| Close the session and stop billing. |
Browser sessions auto-close after 5 minutes of inactivity. Always call spider_browser_close when done.
"Crawl the React documentation and summarize the hooks API"
Uses spider_crawl to fetch 50+ pages in seconds and return clean markdown ready for context.
spider_crawl: {
url: "https://react.dev/reference/react",
limit: 50,
return_format: "markdown",
filter_output_main_only: true
}"Get all product names and prices from this e-commerce page"
Uses spider_ai_scrape to extract structured JSON with zero CSS selectors.
spider_ai_scrape: {
url: "https://example-store.com/products",
prompt: "Extract every product name, price, and availability status as JSON"
}"Log into the dashboard, go to reports, and screenshot the monthly summary"
Uses spider_browser_* tools to drive a remote browser with full anti-bot protection.
1. spider_browser_open: { browser: "auto" }
2. spider_browser_navigate: { url: "https://app.example.com/login" }
3. spider_browser_fill: { selector: "input[name='email']", value: "user@example.com" }
4. spider_browser_fill: { selector: "input[name='password']", value: "..." }
5. spider_browser_click: { selector: "button[type='submit']" }
6. spider_browser_wait_for: { navigation: true }
7. spider_browser_navigate: { url: "https://app.example.com/reports/monthly" }
8. spider_browser_screenshot: {}
9. spider_browser_close: {}"Search for recent AI startup funding rounds and get the details"
Uses spider_search with time filtering, then spider_scrape for details.
spider_search: {
search: "AI startup Series A funding 2025",
num: 10,
fetch_page_content: true,
return_format: "markdown",
tbs: "qdr:m"
}All tools map directly to the Spider API. Core tools accept the same parameters as their API counterparts:
Crawl/Scrape: url, return_format, request, readability, root_selector, proxy_enabled, cache, and many more
Search: search, num, fetch_page_content, country, language, tbs
AI tools: Add a prompt parameter describing what you want in natural language
Browser tools: Use session_id from spider_browser_open for all operations
Full parameter reference: spider.cloud/docs/api
MIT
AI-powered browser automation using natural language. Describe what to do and Spider automates the browser — click buttons, fill forms, navigate pages. Requires an active AI subscription (https://spider.cloud/ai/pricing).
Inputs: url (required), prompt (required), cookies, proxy_enabled, +1 more.
AI-guided website crawling. Describe what you want in plain English and Spider's AI optimizes the crawl automatically. Requires an active AI subscription (https://spider.cloud/ai/pricing).
Inputs: url (required), prompt (required), limit, cookies, +3 more.
AI-powered link extraction and filtering. Describe which links you want and the AI finds and categorizes them. Requires an active AI subscription (https://spider.cloud/ai/pricing).
Inputs: url (required), prompt (required), limit, request, +1 more.
Extract structured data from a page using plain English. Describe the data you need and get clean JSON back — no CSS selectors required. Requires an active AI subscription (https://spider.cloud/ai/pricing).
Inputs: url (required), prompt (required), cookies, request, +2 more.
AI-enhanced web search with intent understanding and relevance ranking. Requires an active AI subscription (https://spider.cloud/ai/pricing).
Inputs: search (required), num, tbs, prompt, +4 more.
Click an element on the page. Waits for the element to appear before clicking.
Inputs: selector (required), session_id (required), timeout.
Close a browser session and release its resources. Always call this when done with a browser session to stop billing.
Inputs: session_id (required).
Get the current page content. Returns the full HTML or extracted text of the page.
Inputs: session_id (required), format.
Execute JavaScript in the browser page and return the result. The expression is evaluated in the page context with access to the DOM. Use for advanced interactions, data extraction, or anything not covered by other browser tools.
Inputs: expression (required), session_id (required).
Fill a form field with text. Clears existing content first, then types the new value. Use for text inputs, textareas, and contenteditable elements.
Inputs: value (required), selector (required), session_id (required), timeout.
Open a new remote browser session in Spider's cloud. Returns a session_id for use with other browser tools. The browser comes with anti-bot protection and proxy rotation. Sessions auto-close after 5 minutes of inactivity. Always close sessions with spider_browser_close when done to avoid unnecessary charges.
Inputs: browser, stealth.
Take a screenshot of the current page. Returns a base64-encoded PNG image. Use for visual verification, debugging, or capturing page state.
Inputs: session_id (required).
Wait for a condition on the page. Use after navigation or actions that trigger dynamic content loading.
Inputs: session_id (required), timeout, selector, navigation.
Crawl a website and extract content from multiple pages. Follows links up to the specified depth/limit. Returns content in markdown, HTML, text, or other formats. Powered by Spider — crawls 100K+ pages/sec with smart JS rendering.
Inputs: url (required), tld, cron, cache, +57 more.
Check your available Spider API credit balance. Returns the number of credits remaining on your account.
Inputs: no named parameters.
Extract all links from a page without fetching their content. Fast way to discover URLs on a site for further processing.
Inputs: url (required), limit, request, return_format.
Scrape a single page and extract its content. No link following — fetches and processes one URL. Faster and cheaper than crawling. Supports all output formats and optional screenshot capture.
Inputs: url (required), tld, cron, cache, +60 more.
Capture a screenshot of a web page. Returns base64-encoded PNG by default. Supports full-page capture and custom viewports.
Inputs: url (required), binary, locale, cookies, +15 more.
Search the web and optionally fetch full page content from results. Supports location, language, and time range filters. Set fetch_page_content=true to get full page data, not just URLs.
Inputs: search (required), num, tbs, url, +13 more.
Transform HTML content to markdown, text, or other formats without making any web requests. Use when you already have HTML and need to convert it.
Inputs: data (required), clean, clean_full, readability, +1 more.
Access content from bot-protected websites. Uses advanced anti-bot bypass with fingerprinting and proxy rotation. Costs 10-40 extra credits per successful unblock on top of base scrape cost.
Inputs: url (required), tld, cron, cache, +60 more.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 22 tools
Each tool has a clear, distinct purpose, with AI-prefixed, browser-prefixed, and general tools easily differentiated. Descriptions clearly separate AI-guided from manual operations, and browser tools cover specific actions like click, fill, navigate, etc., with no overlap.
All tools follow the 'spider_<category>_<action>' pattern, with consistent verb_noun order for actions. AI tools use 'spider_ai_', browser tools use 'spider_browser_', and general tools use a direct verb after 'spider_', maintaining a predictable and clear naming convention.
22 tools is appropriate for a comprehensive web automation and scraping server. The range covers AI-assisted operations, full browser automation, direct scraping, crawling, search, and utilities, without being excessive or missing key functionality.
The tool set covers the full lifecycle of web interaction: opening sessions, navigating, interacting (click, fill, wait, evaluate), extracting content (scrape, crawl, screenshot, links), and closing. Additionally, AI versions, search, transform, and unblocker utilities address advanced needs, leaving no obvious gaps.
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
Web scraping, search, crawling, structured extraction and research for AI agents.
Live web access for agents: scrape, SERP search, crawl/map, 100+ collectors, datasets, proxies.
Direct access to 60+ scraping and search tools. Extract structured data from Google (Search, Maps, Trends), Amazon, Airbnb, Social Media, and any web page directly into your AI agent.
