Integrates with Windsurf (Codeium's IDE) to enable web scraping and browser automation capabilities within the editor.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Spider MCP Server crawl the React hooks documentation"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
The fastest web crawling, scraping, and browser automation server for AI agents. Gives Claude direct access to the web through 22 tools — crawl sites at 100K+ pages/sec, extract structured data with AI, and control remote browsers with built-in anti-bot bypass.
Speed — Crawl 100K+ pages per second. Smart request routing picks HTTP or headless Chrome automatically. Streaming responses start delivering data immediately.
Cost — Pay-per-use credits with no subscription required for core tools. Check your balance anytime with spider_get_credits. AI tools available with an AI subscription.
Reliability — Anti-bot bypass with fingerprinting and proxy rotation. Browser fleet with automatic fallback across Chrome, Firefox, and more. Built-in retry and stealth escalation.
Related MCP server: The Web MCP
claude mcp add spider -- npx -y spider-cloud-mcpSet your API key:
export SPIDER_API_KEY="your-api-key"Add to claude_desktop_config.json:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Add to .cursor/mcp.json:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Add to ~/.codeium/windsurf/mcp_config.json:
{
"mcpServers": {
"spider": {
"command": "npx",
"args": ["-y", "spider-cloud-mcp"],
"env": {
"SPIDER_API_KEY": "your-api-key"
}
}
}
}Variable | Required | Description |
| Yes | Your Spider API key. Get one at spider.cloud/api-keys |
These work on pay-per-use credits with no subscription required.
Tool | Description |
| Crawl a website and extract content from multiple pages. Follows links up to a depth/limit. |
| Scrape a single page. Faster and cheaper than crawling when you need one URL. |
| Search the web. Optionally fetch full page content from results. |
| Extract all links from a page without fetching content. |
| Capture a page screenshot as base64 PNG. |
| Access bot-protected content with advanced anti-bot bypass. |
| Convert HTML to markdown or text without making web requests. |
| Check your API credit balance. |
Natural language web interaction. Describe what you want in plain English.
Requires an AI subscription.
Tool | Description |
| AI-guided crawling — describe what content to find. |
| Extract structured data with plain English — no CSS selectors. |
| AI-enhanced search with intent understanding. |
| Automate browser actions with natural language. |
| Find and filter links by description. |
Direct browser control via spider-browser. Browsers run in Spider's cloud with smart retry, browser switching, anti-bot protection, proxy rotation, and automatic stealth escalation. Supports CDP and BiDi protocols.
Tool | Description |
| Open a remote browser session. Returns a session_id. |
| Navigate to a URL and wait for load. |
| Click an element by CSS selector. |
| Fill a form field with text. |
| Take a screenshot of the current page. Returns base64 PNG. |
| Get page HTML or visible text. |
| Execute JavaScript in the page context. |
| Wait for an element, navigation, or network idle. |
| Close the session and stop billing. |
Browser sessions auto-close after 5 minutes of inactivity. Always call spider_browser_close when done.
"Crawl the React documentation and summarize the hooks API"
Uses spider_crawl to fetch 50+ pages in seconds and return clean markdown ready for context.
spider_crawl: {
url: "https://react.dev/reference/react",
limit: 50,
return_format: "markdown",
filter_output_main_only: true
}"Get all product names and prices from this e-commerce page"
Uses spider_ai_scrape to extract structured JSON with zero CSS selectors.
spider_ai_scrape: {
url: "https://example-store.com/products",
prompt: "Extract every product name, price, and availability status as JSON"
}"Log into the dashboard, go to reports, and screenshot the monthly summary"
Uses spider_browser_* tools to drive a remote browser with full anti-bot protection.
1. spider_browser_open: { browser: "auto" }
2. spider_browser_navigate: { url: "https://app.example.com/login" }
3. spider_browser_fill: { selector: "input[name='email']", value: "user@example.com" }
4. spider_browser_fill: { selector: "input[name='password']", value: "..." }
5. spider_browser_click: { selector: "button[type='submit']" }
6. spider_browser_wait_for: { navigation: true }
7. spider_browser_navigate: { url: "https://app.example.com/reports/monthly" }
8. spider_browser_screenshot: {}
9. spider_browser_close: {}"Search for recent AI startup funding rounds and get the details"
Uses spider_search with time filtering, then spider_scrape for details.
spider_search: {
search: "AI startup Series A funding 2025",
num: 10,
fetch_page_content: true,
return_format: "markdown",
tbs: "qdr:m"
}All tools map directly to the Spider API. Core tools accept the same parameters as their API counterparts:
Crawl/Scrape: url, return_format, request, readability, root_selector, proxy_enabled, cache, and many more
Search: search, num, fetch_page_content, country, language, tbs
AI tools: Add a prompt parameter describing what you want in natural language
Browser tools: Use session_id from spider_browser_open for all operations
Full parameter reference: spider.cloud/docs/api
MIT
AI-powered browser automation using natural language. Describe what to do and Spider automates the browser — click buttons, fill forms, navigate pages. Requires an active AI subscription (https://spider.cloud/ai/pricing).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL | |
| prompt | Yes | Automation instructions (e.g. 'Click the Sign In button, enter email, submit the form') | |
| cookies | No | HTTP cookies | |
| proxy_enabled | No | Enable premium proxies | |
| return_format | No | Output format. Default: raw |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It reveals the requirement for an AI subscription, but does not cover other aspects such as session management, error handling, rate limits, or whether automation is headless. The description is incomplete for a complex tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, both front-loaded with key information. The first explains the tool's purpose, the second adds the subscription requirement. No filler or redundant content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no output schema and moderate complexity (5 params, natural language interpretation), the description is insufficient. It lacks explanation of how the AI interprets prompts, what the return format represents, or behavioral constraints like session lifetime. The sibling tools suggest lower-level alternatives exist, but the description does not clarify when to use this aggregate tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% parameter description coverage, so baseline is 3. The description adds no extra meaning beyond what the schema provides, such as clarifying the role of cookies or proxy_enabled in the automation context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it performs AI-powered browser automation using natural language, mentioning specific actions like clicking buttons, filling forms, and navigating pages. This distinguishes it from sibling tools such as spider_ai_crawl (crawling) and lower-level browser tools like spider_browser_click, which are step-by-step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when wanting to automate browser actions via natural language, and mentions a required AI subscription. However, it does not explicitly state when to use this tool versus alternatives (e.g., lower-level browser tools) or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
AI-guided website crawling. Describe what you want in plain English and Spider's AI optimizes the crawl automatically. Requires an active AI subscription (https://spider.cloud/ai/pricing).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to crawl | |
| limit | No | Max pages to crawl | |
| prompt | Yes | Natural language instructions (e.g. 'Find all product pages and extract pricing info') | |
| cookies | No | HTTP cookies | |
| request | No | Request type: http (fast), chrome (JS rendering), smart (auto-detect). Default: smart | |
| proxy_enabled | No | Enable premium proxies | |
| return_format | No | Output format. Default: raw |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses the AI subscription requirement but lacks details on behavioral aspects like whether the crawl respects robots.txt, rate limits, or how the AI optimization works.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with two sentences that effectively introduce the tool's key feature and requirement. It is front-loaded but could benefit from a brief structure like a bullet list.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema is provided, and the description does not explain what the tool returns (e.g., scraped content, status, errors). Given the tool's complexity (AI crawl), this is a significant gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all parameters. The description adds minimal extra meaning beyond restating that AI optimizes based on prompt, not justifying a higher score.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it performs AI-guided website crawling with natural language instructions. However, it does not differentiate from sibling tools like spider_ai_scrape or spider_crawl, which also involve crawling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description mentions the requirement for an active AI subscription but does not specify when to use this tool versus alternatives such as spider_ai_browser or spider_crawl, nor does it provide exclusion criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
AI-powered link extraction and filtering. Describe which links you want and the AI finds and categorizes them. Requires an active AI subscription (https://spider.cloud/ai/pricing).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to extract links from | |
| limit | No | Max links | |
| prompt | Yes | Link filter instructions (e.g. 'Find all documentation links and API reference pages') | |
| request | No | Request type: http (fast), chrome (JS rendering), smart (auto-detect). Default: smart | |
| return_format | No | Output format. Default: raw |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It mentions the AI subscription requirement but lacks details on rate limits, error handling, or behavioral traits. The brief description offers limited transparency beyond the core capability.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences deliver the purpose and a key prerequisite (AI subscription). No wasted words; front-loaded with the core action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 5 parameters (2 required) and no output schema, the description minimally explains the tool's behavior. The limit and return_format parameters are well-described in the schema, but the description does not elaborate on results or pagination. It is adequate but not rich.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The tool description adds little beyond what the schema already provides (e.g., 'AI-powered' framing). However, the schema descriptions are clear, making this adequate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts and filters links using AI, with a directive to 'describe which links you want.' It distinguishes from sibling tools like 'spider_links' by emphasizing AI-powered functionality.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use cases by saying 'describe which links you want' and notes the requirement of an active AI subscription. However, it does not explicitly state when not to use or compare to alternatives like 'spider_links' for non-AI extraction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract structured data from a page using plain English. Describe the data you need and get clean JSON back — no CSS selectors required. Requires an active AI subscription (https://spider.cloud/ai/pricing).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to scrape | |
| prompt | Yes | Extraction instructions (e.g. 'Extract the article title, author, publish date, and main text') | |
| cookies | No | HTTP cookies | |
| request | No | Request type: http (fast), chrome (JS rendering), smart (auto-detect). Default: smart | |
| proxy_enabled | No | Enable premium proxies | |
| return_format | No | Output format. Default: raw |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that the tool uses AI and requires a subscription, and that output is 'clean JSON'. However, it omits behavioral details such as rate limits, error handling, authentication method beyond the subscription link, or what constitutes a failed extraction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, front-loading the core purpose and key differentiator (plain English, no CSS selectors). Every sentence provides necessary information without extraneous detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has no output schema, the description effectively states the return format ('clean JSON back'). It also notes the subscription requirement. However, it could briefly mention that the tool returns structured JSON based on the prompt, and it lacks guidance on expected output structure or error scenarios. Overall, it is mostly complete for a straightforward scraping tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds no additional semantic information beyond the schema; it reiterates the 'plain English' concept for the prompt parameter but doesn't elaborate on other parameters like cookies, request type, or return format. The schema already adequately describes each parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Extract structured data from a page using plain English' and promises 'clean JSON back', specifying the verb, resource, and output format. The name distinguishes it as AI-powered, but it does not explicitly contrast with the sibling tool 'spider_scrape', though the AI mention implies the difference.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description indicates when to use: for structured data extraction with plain English. It also mentions the requirement of an active AI subscription. However, it does not specify when not to use this tool (e.g., if you need raw HTML or non-AI extraction) or explicitly name alternatives like 'spider_scrape' for non-AI cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
AI-enhanced web search with intent understanding and relevance ranking. Requires an active AI subscription (https://spider.cloud/ai/pricing).
| Name | Required | Description | Default |
|---|---|---|---|
| num | No | Max results | |
| tbs | No | Time range filter | |
| prompt | No | Additional AI guidance for filtering/ranking results | |
| search | Yes | Search query | |
| country | No | Two-letter country code | |
| language | No | Two-letter language code | |
| return_format | No | Output format. Default: raw | |
| fetch_page_content | No | Fetch full page content from results |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; description only mentions AI enhancement and subscription. No disclosure of behavioral traits such as rate limits, error handling, or what the AI does beyond ranking. Minimal transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Very concise, one sentence plus a subscription note, no fluff. However, it could be slightly more informative while remaining concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Description is very brief given the tool has 8 parameters with no output schema. Does not explain how AI enhancement works, interaction between parameters, or return values. Lacks context for agent to fully understand usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, baseline is 3. Description does not add additional meaning to the parameters; it is a generic description without parameter-specific details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states it is an AI-enhanced web search with intent understanding and relevance ranking. Distinguishes from sibling tools like spider_search (basic search) and spider_ai_browser (browser). The mention of subscription requirement adds context.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly notes that an active AI subscription is required, providing a link. Does not explicitly list when to vs. when not to use, but the tool's name and description imply use for AI-enhanced search vs. basic search.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Click an element on the page. Waits for the element to appear before clicking.
| Name | Required | Description | Default |
|---|---|---|---|
| timeout | No | Max time to wait for element in ms. Default: 10000 | |
| selector | Yes | CSS selector of the element to click (e.g. "button.submit", "#login-btn", "a[href='/pricing']") | |
| session_id | Yes | Session ID from spider_browser_open |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses waiting behavior before clicking, which is useful. No annotations provided, so description carries full burden. Lacks details on side effects (e.g., navigation, destructive actions) and error handling when element not found.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences that are front-loaded with the core action. No redundant or unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple click tool, the description covers primary behavior (wait and click). No output schema, but the action is straightforward. Could mention return value, but not essential given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and each parameter has clear descriptions. The tool description adds no additional information beyond the schema, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the action (click) and the resource (element on page). Distinguishes from siblings like fill or navigate because it's specifically for clicking. Also adds that it waits for the element, which clarifies behavior.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives like spider_browser_fill or spider_browser_wait_for. No mention of prerequisites or when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Close a browser session and release its resources. Always call this when done with a browser session to stop billing.
| Name | Required | Description | Default |
|---|---|---|---|
| session_id | Yes | Session ID from spider_browser_open |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, description carries burden. It discloses resource release and billing implications. Lacks details like idempotency or error handling, but covers key behavioral traits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no fluff. Front-loaded with action and resource, then billing reminder. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Simple tool with one param, no output schema. Description covers purpose, usage, and billing side effect. Complete for the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and schema already describes 'session_id' as 'Session ID from spider_browser_open'. Description adds no extra param meaning, earning baseline 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the action (close) and resource (browser session). It distinguishes from sibling tools like spider_browser_open and browser actions by focusing on closing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says 'Always call this when done with a browser session to stop billing', providing clear context. No explicit alternatives or exclusion, but adequate for the tool's simplicity.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Get the current page content. Returns the full HTML or extracted text of the page.
| Name | Required | Description | Default |
|---|---|---|---|
| format | No | Content format: html (full DOM) or text (visible text only). Default: html | |
| session_id | Yes | Session ID from spider_browser_open |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose behavioral traits such as whether JavaScript is executed, if the page state is affected, or any prerequisites. It only mentions output format.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with purpose. Every word is meaningful with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers return types (HTML or text) but lacks details on error handling or session validity. For a tool with no output schema and no annotations, it is adequate but not comprehensive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already describes both parameters. The description adds context like 'full DOM' for HTML format, but doesn't provide significant new meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'get' and the resource 'current page content', and distinguishes it from sibling tools like spider_browser_screenshot by specifying that it returns HTML or text.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for retrieving page content but offers no explicit guidance on when to use this tool versus alternatives like spider_browser_screenshot or spider_browser_evaluate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Execute JavaScript in the browser page and return the result. The expression is evaluated in the page context with access to the DOM. Use for advanced interactions, data extraction, or anything not covered by other browser tools.
| Name | Required | Description | Default |
|---|---|---|---|
| expression | Yes | JavaScript expression to evaluate in the page context. Use a function wrapper for multi-line code: (function() { ... })() | |
| session_id | Yes | Session ID from spider_browser_open |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It correctly notes the expression is evaluated in page context with DOM access, but does not disclose potential side effects like page state modification or security implications. This is adequate but not fully transparent for a JavaScript execution tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core action in the first sentence and usage context in the second. No unnecessary words, every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With two parameters, no output schema, and no annotations, the description is adequate but leaves gaps: it does not specify the format of the returned result or error handling. For a JavaScript execution tool, more details on return value would be helpful.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters. The description adds no extra meaning beyond the schema, except a tip about using a function wrapper for multi-line code. With high coverage, baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it executes JavaScript in the browser page and returns the result, with access to the DOM. It distinguishes from siblings by noting it covers 'anything not covered by other browser tools'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says to use for 'advanced interactions, data extraction, or anything not covered by other browser tools', implying when other browser tools suffice, you should use them instead. It provides clear context but no explicit exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fill a form field with text. Clears existing content first, then types the new value. Use for text inputs, textareas, and contenteditable elements.
| Name | Required | Description | Default |
|---|---|---|---|
| value | Yes | Text to type into the field | |
| timeout | No | Max time to wait for element in ms. Default: 10000 | |
| selector | Yes | CSS selector of the input field (e.g. "input[name='email']", "#search-box") | |
| session_id | Yes | Session ID from spider_browser_open |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses the key behavioral trait of clearing existing content before typing, which is valuable beyond the schema. With no annotations provided, the description carries the full burden of transparency. Could add more details on handling of read-only fields or validation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Extremely concise with two short sentences: first covers purpose and behavior, second specifies supported element types. No wasted words, and the structure is front-loaded with the key action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and full schema coverage, the description provides sufficient context for an agent to understand its usage. Lacks explicit mention of session requirement (though implied by session_id parameter) and return value, but no output schema exists to describe.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description does not add additional semantics for parameters beyond what is already in the schema; it only describes the overall function.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the verb 'fill' and the resource 'form field', and specifies the behavior of clearing existing content before typing. Distinguishes from siblings like spider_browser_click and spider_browser_navigate by focusing on text input.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states the use case for text inputs, textareas, and contenteditable elements, providing clear context. However, it does not explicitly exclude other use cases or compare with siblings, limiting guidance on when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Open a new remote browser session in Spider's cloud. Returns a session_id for use with other browser tools. The browser comes with anti-bot protection and proxy rotation. Sessions auto-close after 5 minutes of inactivity. Always close sessions with spider_browser_close when done to avoid unnecessary charges.
| Name | Required | Description | Default |
|---|---|---|---|
| browser | No | Browser engine. auto (recommended), chrome, chrome-new (dedicated), firefox. Default: auto | |
| stealth | No | Stealth/proxy level 0-3. 0=auto, 1=standard, 2=residential, 3=premium. Default: 0 |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description fully discloses key behaviors: anti-bot protection, proxy rotation, auto-close after 5 minutes of inactivity, and the recommendation to explicitly close sessions to avoid charges. No contradictions or missing critical traits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise (4 sentences), front-loads the main purpose, and each sentence adds value without redundancy. No unnecessary words or details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple; the description covers purpose, return value, key features, and usage advice. It lacks mention of error conditions or prerequisites, but given the low complexity and no output schema, it is nearly complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already describes both parameters (browser and stealth). The description adds no additional parameter-level information beyond what is in the schema. Baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: opening a new remote browser session in Spider's cloud and returning a session_id. It distinguishes this from other browser tools (e.g., navigate, click, close) by specifying it initiates the session.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context: it should be used first to obtain a session_id before other browser tools. It advises closing sessions with spider_browser_close when done. However, it does not explicitly state when not to use it or contrast with alternative tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Take a screenshot of the current page. Returns a base64-encoded PNG image. Use for visual verification, debugging, or capturing page state.
| Name | Required | Description | Default |
|---|---|---|---|
| session_id | Yes | Session ID from spider_browser_open |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, but description discloses that it returns a base64-encoded PNG, which is sufficient for a read-only screenshot tool. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with purpose and output format. No redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Complete for a simple tool with one parameter and no output schema. Covers purpose, output, and use cases effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with one parameter (session_id) well-described. Description adds no extra parameter detail beyond the schema, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Take a screenshot of the current page' with a specific verb and resource, differentiating from siblings like spider_browser_content or spider_screenshot.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly mentions use cases: visual verification, debugging, or capturing page state. Could be improved by noting when to avoid (e.g., dynamic content) but still clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Wait for a condition on the page. Use after navigation or actions that trigger dynamic content loading.
| Name | Required | Description | Default |
|---|---|---|---|
| timeout | No | Max wait time in ms. Default: 30000 | |
| selector | No | CSS selector to wait for (element must appear in DOM) | |
| navigation | No | Wait for the next navigation to complete | |
| session_id | Yes | Session ID from spider_browser_open |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full burden of disclosing behavior. It states the tool waits for a condition but omits critical details like timeout expiration behavior (error or retry), whether it blocks, and what it returns. This is insufficient for an agent to predict side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences that are clear and front-loaded. The first sentence states the purpose, the second provides usage context. It is concise but not overly abbreviated, earning a score of 4.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (waiting for dynamic conditions) and the absence of an output schema, the description should explain return behavior, timeout handling, and required session context. It does not, leaving the agent uninformed about key aspects.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All four parameters have descriptions in the input schema (100% coverage). The description adds no extra parameter meaning beyond the schema, so a baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool waits for a condition on the page, which is a specific action distinct from sibling tools like clicking, navigating, or scraping. It also provides context on when to use it (after navigation or dynamic content actions).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description mentions using the tool after navigation or actions that trigger dynamic content, but does not explicitly specify when not to use it or suggest alternatives. This leaves some ambiguity for an AI agent deciding between wait_for and other tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Crawl a website and extract content from multiple pages. Follows links up to the specified depth/limit. Returns content in markdown, HTML, text, or other formats. Powered by Spider — crawls 100K+ pages/sec with smart JS rendering.
| Name | Required | Description | Default |
|---|---|---|---|
| tld | No | Follow top-level domain variations | |
| url | Yes | The URL to process. Comma-separate for multiple URLs. | |
| cron | No | Schedule recurring crawls | |
| cache | No | HTTP caching. true/false or {maxAge, allowStale, period} | |
| delay | No | Delay between requests in ms (max 60000). Disables concurrency | |
| depth | No | Maximum crawl depth from start URL. Default: 25 | |
| limit | No | Maximum pages to crawl. 0 for unlimited. Default: 0 | |
| proxy | No | Proxy pool type. residential (x1.2), mobile (x2), isp/datacenter (x1.2) | |
| budget | No | Page budget per URL path (e.g. {'*': 100, '/blog': 20}) | |
| locale | No | Browser locale (e.g. 'en-US') | |
| cookies | No | HTTP cookies for authenticated scraping | |
| request | No | Request type: http (fast), chrome (JS rendering), smart (auto-detect). Default: smart | |
| session | No | Persist cookies across requests. Default: true | |
| sitemap | No | Discover pages via sitemap.xml | |
| timeout | No | Overall request timeout in milliseconds | |
| metadata | No | Collect page metadata (title, description, keywords) | |
| sitemaps | No | Specific sitemap URLs to use | |
| timezone | No | Browser timezone | |
| viewport | No | Device viewport settings | |
| wait_for | No | Chrome-only readiness gate (request: 'chrome' or 'smart'). Object with any of: selector ({selector, timeout}), idle_network ({timeout}), idle_network0, almost_idle_network0, dom ({selector, timeout}), delay ({timeout}), page_navigations (bool). Each timeout is a Rust Duration: { secs, nanos }. | |
| webhooks | No | Webhook URLs for async events (on_find, on_credits_depleted) | |
| blacklist | No | URL path patterns to exclude (supports regex) | |
| block_ads | No | Block advertisements. Default: true | |
| whitelist | No | URL path patterns to include (supports regex) | |
| automation | No | Browser automation actions to run before extraction (Click, Fill, Wait, Scroll) | |
| clean_html | No | Strip unwanted HTML attributes (class, style, etc.) | |
| filter_svg | No | Remove SVG elements from markup before processing | |
| subdomains | No | Follow subdomains | |
| user_agent | No | Custom HTTP user agent string | |
| fingerprint | No | Advanced browser fingerprint detection. Default: true | |
| readability | No | Use readability algorithm for cleaner content extraction | |
| storageless | No | Prevent data storage. Default: true | |
| chunking_alg | No | Segment content: bysentence, bylines, bycharacterlength, bywords | |
| country_code | No | ISO country code for geo-located proxy (e.g. 'gb', 'us') | |
| remote_proxy | No | External proxy URL. Saves 50% on data transfer credits | |
| event_tracker | No | Track detailed request/response events | |
| filter_images | No | Remove image elements from markup before processing | |
| preserve_host | No | Preserve the HOST header on redirects | |
| proxy_enabled | No | Enable premium proxies. Multiplies credit cost by 1.5x | |
| return_format | No | Output format. Default: raw | |
| root_selector | No | Root CSS selector to scope extraction (e.g. "#main-content") | |
| full_resources | No | Download all resources including images, CSS, JS | |
| respect_robots | No | Obey robots.txt rules. Default: true | |
| return_cookies | No | Include HTTP response cookies | |
| return_headers | No | Include HTTP response headers | |
| block_analytics | No | Block analytics scripts. Default: true | |
| redirect_policy | No | How to handle redirects. Default: Loose | |
| request_timeout | No | Per-request timeout in milliseconds | |
| exclude_selector | No | CSS selector for elements to exclude from output | |
| external_domains | No | External domains to follow. Use ['*'] to allow all | |
| filter_main_only | No | Keep only main content. Default: enabled | |
| return_json_data | No | Extract JSON-LD and structured data from pages | |
| block_stylesheets | No | Block CSS stylesheets. Default: true | |
| concurrency_limit | No | Max concurrent requests to the target site | |
| disable_intercept | No | Disable request interception | |
| filter_output_svg | No | Remove SVGs from output | |
| return_page_links | No | Include links found on each page in the response | |
| run_in_background | No | Run asynchronously. Requires webhooks or storageless=false | |
| css_extraction_map | No | CSS/XPath selectors for structured extraction. Shape: { '<url-path>': [{ name, selectors: [...] }] }. '/' matches all paths. Results returned under css_extracted. | |
| filter_output_images | No | Remove images from output | |
| filter_output_main_only | No | Remove nav, aside, footer from output |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden. It mentions following links up to depth/limit and JS rendering ('smart JS rendering'), but omits many behavioral details such as caching, robots.txt respect, or concurrency. The description adds some value but is not comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with purpose. The third sentence is slightly promotional but not excessive. Still efficient for a brief summary.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 61 parameters, no output schema, the description is too brief. It does not explain output structure, scheduling, proxies, or automation. An agent would lack context for many configuration choices.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds minimal value beyond the schema—only summarizing depth/limit and return_format. It does not clarify default values or parameter interactions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Crawl a website and extract content from multiple pages' with specific verbs and resources. It mentions depth/limit and return formats, distinguishing it from siblings like spider_scrape or spider_links.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a general sense of when to use the tool (crawling multiple pages with depth/limit) but lacks explicit guidance on when not to use it or alternatives. It does not mention siblings like spider_scrape for single pages.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check your available Spider API credit balance. Returns the number of credits remaining on your account.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must cover behavioral traits. It describes a read-only operation that returns a numeric balance. It does not mention potential issues like authentication or rate limits, but for a simple credit check, this is adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very concise with two sentences. The first sentence states the action, the second explains the return value. No unnecessary words or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has no parameters and no output schema, the description is complete. It explains the purpose and the result of the operation. No additional information is needed for an agent to use it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has no parameters, and the schema description coverage is 100%. The description adds no parameter information, which is acceptable as there are no parameters. Baseline of 4 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: checking the available Spider API credit balance and returning the number of credits remaining. The verb 'Check' and the resource 'credit balance' are specific, and the tool is distinct from siblings which handle browsing, crawling, and scraping.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not provide explicit guidance on when to use this tool versus alternatives. However, given the simplicity of the tool and its clear purpose, usage is implied. No exclusions or alternative suggestions are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract all links from a page without fetching their content. Fast way to discover URLs on a site for further processing.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to extract links from | |
| limit | No | Max links to return | |
| request | No | Request type: http (fast), chrome (JS rendering), smart (auto-detect). Default: smart | |
| return_format | No | Output format. Default: raw |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully convey behavioral traits. It mentions 'without fetching their content' and 'fast', but fails to disclose other important behaviors such as error handling, rate limits, auth requirements, or output format. The description is insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no filler words, front-loaded with key information. Efficient and clear.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple with 4 parameters well-documented in schema. However, no output schema exists and the description does not explain the return value (list of URLs). This leaves ambiguity for the agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description does not add significant semantic value beyond the schema; it restates the purpose but does not clarify defaults or behavior of specific parameters like 'request' or 'return_format'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Extract', the resource 'links from a page', and specifies 'without fetching their content'. It distinguishes itself from siblings like spider_scrape and spider_crawl which fetch content or follow links.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains when to use this tool: when you want to discover URLs without fetching content. It implies it's for initial discovery before further processing. However, it does not explicitly list alternatives or conditions when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scrape a single page and extract its content. No link following — fetches and processes one URL. Faster and cheaper than crawling. Supports all output formats and optional screenshot capture.
| Name | Required | Description | Default |
|---|---|---|---|
| tld | No | Follow top-level domain variations | |
| url | Yes | The URL to process. Comma-separate for multiple URLs. | |
| cron | No | Schedule recurring crawls | |
| cache | No | HTTP caching. true/false or {maxAge, allowStale, period} | |
| proxy | No | Proxy pool type. residential (x1.2), mobile (x2), isp/datacenter (x1.2) | |
| binary | No | Return screenshot as binary instead of base64 | |
| budget | No | Page budget per URL path (e.g. {'*': 100, '/blog': 20}) | |
| locale | No | Browser locale (e.g. 'en-US') | |
| cookies | No | HTTP cookies for authenticated scraping | |
| request | No | Request type: http (fast), chrome (JS rendering), smart (auto-detect). Default: smart | |
| session | No | Persist cookies across requests. Default: true | |
| sitemap | No | Discover pages via sitemap.xml | |
| timeout | No | Overall request timeout in milliseconds | |
| metadata | No | Collect page metadata (title, description, keywords) | |
| sitemaps | No | Specific sitemap URLs to use | |
| timezone | No | Browser timezone | |
| viewport | No | Device viewport settings | |
| wait_for | No | Chrome-only readiness gate (request: 'chrome' or 'smart'). Object with any of: selector ({selector, timeout}), idle_network ({timeout}), idle_network0, almost_idle_network0, dom ({selector, timeout}), delay ({timeout}), page_navigations (bool). Each timeout is a Rust Duration: { secs, nanos }. | |
| webhooks | No | Webhook URLs for async events (on_find, on_credits_depleted) | |
| blacklist | No | URL path patterns to exclude (supports regex) | |
| block_ads | No | Block advertisements. Default: true | |
| full_page | No | Capture full scrollable page. Default: true | |
| whitelist | No | URL path patterns to include (supports regex) | |
| automation | No | Browser automation actions to run before extraction (Click, Fill, Wait, Scroll) | |
| cdp_params | No | Chrome DevTools Protocol screenshot options (clip, format, quality) | |
| clean_html | No | Strip unwanted HTML attributes (class, style, etc.) | |
| filter_svg | No | Remove SVG elements from markup before processing | |
| screenshot | No | Enable screenshot capture | |
| subdomains | No | Follow subdomains | |
| user_agent | No | Custom HTTP user agent string | |
| fingerprint | No | Advanced browser fingerprint detection. Default: true | |
| readability | No | Use readability algorithm for cleaner content extraction | |
| storageless | No | Prevent data storage. Default: true | |
| block_images | No | Block images from loading before screenshot | |
| chunking_alg | No | Segment content: bysentence, bylines, bycharacterlength, bywords | |
| country_code | No | ISO country code for geo-located proxy (e.g. 'gb', 'us') | |
| remote_proxy | No | External proxy URL. Saves 50% on data transfer credits | |
| event_tracker | No | Track detailed request/response events | |
| filter_images | No | Remove image elements from markup before processing | |
| preserve_host | No | Preserve the HOST header on redirects | |
| proxy_enabled | No | Enable premium proxies. Multiplies credit cost by 1.5x | |
| return_format | No | Output format. Default: raw | |
| root_selector | No | Root CSS selector to scope extraction (e.g. "#main-content") | |
| full_resources | No | Download all resources including images, CSS, JS | |
| respect_robots | No | Obey robots.txt rules. Default: true | |
| return_cookies | No | Include HTTP response cookies | |
| return_headers | No | Include HTTP response headers | |
| block_analytics | No | Block analytics scripts. Default: true | |
| omit_background | No | Transparent background in screenshot | |
| redirect_policy | No | How to handle redirects. Default: Loose | |
| request_timeout | No | Per-request timeout in milliseconds | |
| exclude_selector | No | CSS selector for elements to exclude from output | |
| external_domains | No | External domains to follow. Use ['*'] to allow all | |
| filter_main_only | No | Keep only main content. Default: enabled | |
| return_json_data | No | Extract JSON-LD and structured data from pages | |
| block_stylesheets | No | Block CSS stylesheets. Default: true | |
| concurrency_limit | No | Max concurrent requests to the target site | |
| disable_intercept | No | Disable request interception | |
| filter_output_svg | No | Remove SVGs from output | |
| return_page_links | No | Include links found on each page in the response | |
| run_in_background | No | Run asynchronously. Requires webhooks or storageless=false | |
| css_extraction_map | No | CSS/XPath selectors for structured extraction. Shape: { '<url-path>': [{ name, selectors: [...] }] }. '/' matches all paths. Results returned under css_extracted. | |
| filter_output_images | No | Remove images from output | |
| filter_output_main_only | No | Remove nav, aside, footer from output |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It mentions speed and cost relative to crawling and optional screenshot, but lacks details on authentication needs, rate limits, or return format. The description is adequate but not rich enough for a tool with 64 parameters.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences with no waste. Front-loaded with the core action and key differentiators. Every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity (64 params, no output schema), the description is sparse. It does not explain return structure or behavior beyond extraction. While the schema covers parameters, the lack of output schema makes the description incomplete for fully informing the agent about results.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds no parameter-specific meaning beyond generic statements like 'Supports all output formats and optional screenshot capture'. It does not enhance the schema's descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it scrapes a single page and extracts content, with explicit 'No link following' distinguishing it from crawling. It specifies 'Faster and cheaper than crawling' and mentions output formats and screenshots, making the purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'No link following' and compares to crawling ('Faster and cheaper than crawling'), guiding the agent to use this for single-page tasks. It could be more explicit about when not to use it (e.g., for multi-page extraction), but the contrast with crawling is effective.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Capture a screenshot of a web page. Returns base64-encoded PNG by default. Supports full-page capture and custom viewports.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to screenshot | |
| binary | No | Return screenshot as binary instead of base64 | |
| locale | No | Locale | |
| cookies | No | HTTP cookies | |
| timeout | No | Timeout in ms | |
| timezone | No | Timezone | |
| viewport | No | Device viewport settings | |
| block_ads | No | Block ads | |
| full_page | No | Capture full scrollable page. Default: true | |
| automation | No | Automation actions to run before taking the screenshot | |
| cdp_params | No | Chrome DevTools Protocol screenshot options (clip, format, quality) | |
| screenshot | No | Enable screenshot capture | |
| fingerprint | No | Advanced fingerprint detection | |
| block_images | No | Block images from loading before screenshot | |
| country_code | No | ISO country code for proxy | |
| proxy_enabled | No | Enable premium proxies | |
| block_analytics | No | Block analytics | |
| omit_background | No | Transparent background in screenshot | |
| block_stylesheets | No | Block stylesheets |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavior. It mentions return type and support for full-page and viewports, but omits details on error handling, performance, authentication requirements, or rate limits. The description is adequate but not comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise, consisting of two short sentences that convey the essential purpose and key features. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 19 parameters and is complex (nested objects, many options), the description is too minimal. It does not mention most configuration options (e.g., ads blocking, cookies, timeouts, CDP params, automation) nor the default behavior of these parameters. An overview of available features would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all parameters. The description adds context about full-page capture (tying to full_page parameter) and custom viewports (viewport parameter), but does not add significant meaning beyond the schema. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool captures a screenshot of a web page and mentions default return format (base64 PNG) and capabilities (full-page, custom viewports). However, it does not distinguish this tool from the sibling spider_browser_screenshot, which likely serves a similar purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool is used when a screenshot is needed, but it provides no guidance on when not to use it or what alternatives exist (e.g., spider_browser_screenshot for interactive browser context). It lacks explicit usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Search the web and optionally fetch full page content from results. Supports location, language, and time range filters. Set fetch_page_content=true to get full page data, not just URLs.
| Name | Required | Description | Default |
|---|---|---|---|
| num | No | Max results to return | |
| tbs | No | Time range: qdr:h (past hour), qdr:d (24h), qdr:w (week), qdr:m (month), qdr:y (year) | |
| url | No | Optional URL context for the search | |
| page | No | Result page number | |
| limit | No | Page crawl limit when fetching results | |
| search | Yes | Search query | |
| cookies | No | HTTP cookies | |
| country | No | Two-letter country code (e.g. 'us') | |
| request | No | Request type: http (fast), chrome (JS rendering), smart (auto-detect). Default: smart | |
| language | No | Two-letter language code (e.g. 'en') | |
| location | No | Location name (e.g. 'United Kingdom') | |
| quick_search | No | Prioritize speed over completeness | |
| search_limit | No | Max result URLs to fetch. 0 for all | |
| proxy_enabled | No | Enable premium proxies | |
| return_format | No | Output format. Default: raw | |
| auto_pagination | No | Auto-paginate to reach exact result count | |
| fetch_page_content | No | Fetch full content from each result page. Default: false |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits fully. It mentions optional content fetching and filtering, but omits details on rate limits, authentication, result pagination, or whether the operation is read-only. Adequate but not thorough.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise with two sentences that front-load the core action and add an optional feature. No extraneous words; every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (17 parameters, no output schema), the description is too minimal. It does not explain return format, error handling, or what kind of results to expect (e.g., list of items with titles, URLs). The brevity leaves significant gaps for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds meaning by highlighting key parameters (location, language, time range, fetch_page_content) and their intended use, providing useful orientation beyond the schema's individual descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool performs web searches and can optionally fetch full page content, with support for location, language, and time range filters. However, it does not explicitly differentiate from sibling tools like spider_ai_search, limiting its distinctiveness.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives, nor any conditions or prerequisites. The only usage hint is when to set fetch_page_content, but no when-not or context for other choices.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Transform HTML content to markdown, text, or other formats without making any web requests. Use when you already have HTML and need to convert it.
| Name | Required | Description | Default |
|---|---|---|---|
| data | Yes | Array of HTML documents to transform | |
| clean | No | Clean output for AI consumption (strip nav, footers) | |
| clean_full | No | Aggressively clean HTML attributes | |
| readability | No | Apply readability preprocessing | |
| return_format | No | Output format. Default: raw |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. However, it does not disclose any behavioral traits beyond the basic transformation, such as side effects, size limits, or output handling. It only restates the purpose.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence with no wasted words. It is appropriately sized for a simple tool but could benefit from a slightly more structured format.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and 5 input parameters, the description is somewhat complete but lacks details on supported formats, error handling, or behavior with invalid input. It provides the essential context but leaves gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description does not add any additional meaning beyond the schema; it merely repeats the transformation concept already clear in the purpose.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Transform' and resource 'HTML content', and explicitly distinguishes itself from tools that make web requests, making the purpose very specific.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description says 'without making any web requests' and 'Use when you already have HTML and need to convert it', which provides clear context for when to use the tool. It does not explicitly name alternatives but the context implies when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Access content from bot-protected websites. Uses advanced anti-bot bypass with fingerprinting and proxy rotation. Costs 10-40 extra credits per successful unblock on top of base scrape cost.
| Name | Required | Description | Default |
|---|---|---|---|
| tld | No | Follow top-level domain variations | |
| url | Yes | The URL to process. Comma-separate for multiple URLs. | |
| cron | No | Schedule recurring crawls | |
| cache | No | HTTP caching. true/false or {maxAge, allowStale, period} | |
| proxy | No | Proxy pool type. residential (x1.2), mobile (x2), isp/datacenter (x1.2) | |
| binary | No | Return screenshot as binary instead of base64 | |
| budget | No | Page budget per URL path (e.g. {'*': 100, '/blog': 20}) | |
| locale | No | Browser locale (e.g. 'en-US') | |
| cookies | No | HTTP cookies for authenticated scraping | |
| request | No | Request type: http (fast), chrome (JS rendering), smart (auto-detect). Default: smart | |
| session | No | Persist cookies across requests. Default: true | |
| sitemap | No | Discover pages via sitemap.xml | |
| timeout | No | Overall request timeout in milliseconds | |
| metadata | No | Collect page metadata (title, description, keywords) | |
| sitemaps | No | Specific sitemap URLs to use | |
| timezone | No | Browser timezone | |
| viewport | No | Device viewport settings | |
| wait_for | No | Chrome-only readiness gate (request: 'chrome' or 'smart'). Object with any of: selector ({selector, timeout}), idle_network ({timeout}), idle_network0, almost_idle_network0, dom ({selector, timeout}), delay ({timeout}), page_navigations (bool). Each timeout is a Rust Duration: { secs, nanos }. | |
| webhooks | No | Webhook URLs for async events (on_find, on_credits_depleted) | |
| blacklist | No | URL path patterns to exclude (supports regex) | |
| block_ads | No | Block advertisements. Default: true | |
| full_page | No | Capture full scrollable page. Default: true | |
| whitelist | No | URL path patterns to include (supports regex) | |
| automation | No | Browser automation actions to run before extraction (Click, Fill, Wait, Scroll) | |
| cdp_params | No | Chrome DevTools Protocol screenshot options (clip, format, quality) | |
| clean_html | No | Strip unwanted HTML attributes (class, style, etc.) | |
| filter_svg | No | Remove SVG elements from markup before processing | |
| screenshot | No | Enable screenshot capture | |
| subdomains | No | Follow subdomains | |
| user_agent | No | Custom HTTP user agent string | |
| fingerprint | No | Advanced browser fingerprint detection. Default: true | |
| readability | No | Use readability algorithm for cleaner content extraction | |
| storageless | No | Prevent data storage. Default: true | |
| block_images | No | Block images from loading before screenshot | |
| chunking_alg | No | Segment content: bysentence, bylines, bycharacterlength, bywords | |
| country_code | No | ISO country code for geo-located proxy (e.g. 'gb', 'us') | |
| remote_proxy | No | External proxy URL. Saves 50% on data transfer credits | |
| event_tracker | No | Track detailed request/response events | |
| filter_images | No | Remove image elements from markup before processing | |
| preserve_host | No | Preserve the HOST header on redirects | |
| proxy_enabled | No | Enable premium proxies. Multiplies credit cost by 1.5x | |
| return_format | No | Output format. Default: raw | |
| root_selector | No | Root CSS selector to scope extraction (e.g. "#main-content") | |
| full_resources | No | Download all resources including images, CSS, JS | |
| respect_robots | No | Obey robots.txt rules. Default: true | |
| return_cookies | No | Include HTTP response cookies | |
| return_headers | No | Include HTTP response headers | |
| block_analytics | No | Block analytics scripts. Default: true | |
| omit_background | No | Transparent background in screenshot | |
| redirect_policy | No | How to handle redirects. Default: Loose | |
| request_timeout | No | Per-request timeout in milliseconds | |
| exclude_selector | No | CSS selector for elements to exclude from output | |
| external_domains | No | External domains to follow. Use ['*'] to allow all | |
| filter_main_only | No | Keep only main content. Default: enabled | |
| return_json_data | No | Extract JSON-LD and structured data from pages | |
| block_stylesheets | No | Block CSS stylesheets. Default: true | |
| concurrency_limit | No | Max concurrent requests to the target site | |
| disable_intercept | No | Disable request interception | |
| filter_output_svg | No | Remove SVGs from output | |
| return_page_links | No | Include links found on each page in the response | |
| run_in_background | No | Run asynchronously. Requires webhooks or storageless=false | |
| css_extraction_map | No | CSS/XPath selectors for structured extraction. Shape: { '<url-path>': [{ name, selectors: [...] }] }. '/' matches all paths. Results returned under css_extracted. | |
| filter_output_images | No | Remove images from output | |
| filter_output_main_only | No | Remove nav, aside, footer from output |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavior. It mentions fingerprinting, proxy rotation, and extra costs, but lacks details on failure modes, return format, or additional latency. This is adequate but not thorough.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loading the purpose and efficiently conveying technique and cost. No redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 64 parameters and no output schema or annotations, the description is too brief. It fails to explain return values, parameter interplay, or failure handling, leaving significant gaps for agent use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the description adds no parameter-specific meaning beyond the schema. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool accesses content from bot-protected websites, using anti-bot bypass, fingerprinting, and proxy rotation. This distinct purpose differentiates it from siblings like spider_scrape and spider_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use for bot-protected websites and mentions extra credit costs, providing some usage context. However, it does not explicitly state when to avoid this tool or recommend alternatives for non-protected sites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v2.1.2spider_ai_browserspider_ai_crawlspider_ai_linksspider_ai_scrapespider_ai_searchspider_browser_clickspider_browser_closespider_browser_contentspider_browser_evaluatespider_browser_fillspider_browser_navigatespider_browser_openspider_browser_screenshotspider_browser_wait_forspider_crawlspider_get_creditsspider_linksspider_scrapespider_screenshotspider_searchspider_transformspider_unblockerScored across 22 tools
Each tool has a clear, distinct purpose, with AI-prefixed, browser-prefixed, and general tools easily differentiated. Descriptions clearly separate AI-guided from manual operations, and browser tools cover specific actions like click, fill, navigate, etc., with no overlap.
All tools follow the 'spider_<category>_<action>' pattern, with consistent verb_noun order for actions. AI tools use 'spider_ai_', browser tools use 'spider_browser_', and general tools use a direct verb after 'spider_', maintaining a predictable and clear naming convention.
22 tools is appropriate for a comprehensive web automation and scraping server. The range covers AI-assisted operations, full browser automation, direct scraping, crawling, search, and utilities, without being excessive or missing key functionality.
The tool set covers the full lifecycle of web interaction: opening sessions, navigating, interacting (click, fill, wait, evaluate), extracting content (scrape, crawl, screenshot, links), and closing. Additionally, AI versions, search, transform, and unblocker utilities address advanced needs, leaving no obvious gaps.
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
Direct access to 60+ scraping and search tools. Extract structured data from Google (Search, Maps, Trends), Amazon, Airbnb, Social Media, and any web page directly into your AI agent.
Live web access for agents: scrape, SERP search, crawl/map, 74 collectors, datasets, proxies.
Reliable web access for AI agents: smart HTTP, rotating proxies, and full-browser rendering.
