Scrapy MCP Server
The Scrapy MCP Server is a robust, enterprise-grade web scraping platform that offers comprehensive data extraction capabilities for commercial use.
Core Scraping Capabilities:
Multiple scraping methods: HTTP requests, Scrapy framework, Selenium, or Playwright with intelligent method selection
Concurrent processing: Scrape multiple URLs simultaneously with exponential backoff retry mechanisms
JavaScript support: Fully render dynamic, JavaScript-heavy websites using complete browser rendering
Advanced data extraction: Configure flexible extraction rules using simple or advanced selectors, or automatically extract structured data like contact information, social media links, product details, and addresses
Link extraction: Specialized link extraction with domain filtering and internal/external link options
Form interaction: Automatically fill and submit various form types including text inputs, checkboxes, and file uploads
Anti-Detection & Performance:
Stealth techniques: Bypass anti-bot measures using undetected-chromedriver, Playwright stealth, random User-Agent rotation, and proxy support
Performance optimization: In-memory caching, rate limiting, and intelligent request handling to prevent server overload
Monitoring tools: Track server metrics including request counts, success rates, cache statistics, and detailed performance monitoring
Enterprise Features:
Ethical compliance: Check robots.txt files for responsible data collection
Error handling: Robust error classification and handling mechanisms
Cache management: Clear scraping results cache and manage server resources
Provides web scraping capabilities using the Scrapy framework for large-scale data extraction, with support for concurrent requests, custom pipelines, and advanced crawling features.
Enables browser automation and JavaScript-heavy website scraping through Selenium WebDriver, with support for form filling, element waiting, and dynamic content extraction.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Scrapy MCP Serverscrape the pricing page from example.com and convert it to markdown"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
✨ Why Negentropy Perceives?
In the vast ecosystem of AI agent projects, the "dirty work" of information perception often degenerates into fragile, unmaintainable chaos over time. Grounded in our core engineering philosophy of Orthogonal Decomposition and Entropy Reduction (Negentropy), we completely quarantine the mess of low-level network communications and format deconstruction. We only inject pure, undisputed certainty into your sandbox:
🕵️ Web Page to Markdown: Facing heavily-rendered SPAs and fortified anti-scraping defenses? The engine comes armed with a built-in 5-tier penetration mechanism (ranging from hyper-concurrency to headless stealth browser rotation). "What You See Is What You Get" — tearing through waterfall setups is a walk in the park.
📑 PDF to Markdown: Stop compromising over misaligned tables and mangled characters. Powered by our proprietary "Engine Arena" mechanism, engaging
Smartmode summons an LLM as the ultimate referee. It coordinates 7 specialized engines (including Docling, PyMuPDF, etc.) performing concurrent deconstruction to precisely extract LaTeX formulas, gnarly table matrices, and deep layout structures.🦾 Heavy-Duty Infrastructure: Abandon toy-grade SDK wrappers. Our core is hardwired with resilient exponential backoffs, multi-layered rate-limiting circuit breakers, and aggressive memory caching mechanisms. Riding on full-duplex
asyncio, it maxes out the absolute throughput limit of a single node.🔌 Native MCP Integration: We firmly embrace the pristine Model Context Protocol specification. Leveraging standard HTTP / STDIO / SSE transports, it abandons redundant glue code for seamless, zero-friction injection into Claude Desktop or Cursor environments.
Related MCP server: scrapy-mcp
Quick Start
1. Millisecond Loading
# We recommend using uv (Python 3.13+ required)
uv add negentropy-perceives2. Ignite the Engine
uv run negentropy-perceives # Defaults to listening on localhost:2992, HTTP mode💡 Advanced Arsenal: Upon first launch, Negentropy Perceives will auto-generate its configuration at
~/.negentropy/perceives.config.yaml. Hidden inside are the switches for high-tier warfare.
3. Witness True Perception
import asyncio
from negentropy.perceives.sdk import NegentropyPerceivesClient
async def perceive_world():
async with NegentropyPerceivesClient() as client:
result = await client.parse_webpage_to_markdown(
url="https://en.wikipedia.org/wiki/Entropy",
)
print("====== Pure Nectar Extracted ======")
print(result.markdown_content[:250], "......\n")
print(f"📊 Pure words retrieved from the noise: {result.word_count}")
asyncio.run(perceive_world())4. Connect the MCP Client
Add the following to your claude_desktop_config.json in Claude Desktop:
{
"mcpServers": {
"negentropy-perceives": {
"type": "http",
"url": "http://localhost:2992/mcp"
}
}
}Supports three transport modes: STDIO (local dev), HTTP (production-recommended), and SSE (compatibility mode). See the User Guide for the comprehensive configuration.
Core Capabilities
Toolkit Overview
Tool | Function | Use Case |
| Discover webpage links, supports domain filtering | Site map discovery, link audits |
| Inspect page metadata (status code, content type, etc.) | Target page pre-flight check |
| Webpage to Markdown | Granular single-page extraction |
| Batch Webpages to Markdown | Knowledge base building, site archives |
| PDF to Markdown | Academic papers, financial reports |
| Batch PDFs to Markdown | Mass document digitization |
Please adhere to the targeted website's Terms of Service (TOS) and sensibly restrict request frequencies. This tool is intended exclusively for legal and compliant data acquisition.
Web Scraping Strategies
Method | Description |
| Smart selection (Recommended) |
| Standard HTTP request, ideal for static pages |
| Browser rendering, seamlessly executes JS |
| Covert Selenium, shatters anti-scraping blocks |
| Stealth Playwright, lightweight anti-detection |
PDF Engines
Engine | Specialty | GPU Acceleration |
Docling | AI layout analysis, table recognition | CUDA / MPS / XPU |
MinerU | Deep learning structure analysis, LaTeX | CUDA / MLX |
Marker | Academic documents, Nougat model | CUDA |
PyMuPDF | Lightning-fast text extraction | — |
PyPDF | Absolute baseline fallback | — |
In
automode, the system cascades through a graceful degradation chain: Docling → MinerU → Marker → PyMuPDF → PyPDF. Activatingsmartmode enlists an LLM to orchestrate a competitive parallel run across engines, ultimately fusing the optimum output.
Architectural Landscape
graph TD
A["SDK Layer<br/>NegentropyPerceivesClient"] -.->|"HTTP Transport"| T["MCP Tool Layer<br/>6 Tools · @app.tool()"]
T --> P["Pipeline Layer<br/>Stage Orchestration · Competition/Fallback"]
T --> B["Processing Engine Layer<br/>Scraping · PDF · Markdown"]
P --> B
B --> C["Infrastructure Layer<br/>RateLimiter · Cache · Metrics · ErrorHandler · Retry"]
C --> D["Configuration Layer<br/>pydantic-settings · Env Vars"]
style A fill:#4c1d95,stroke:#a78bfa,color:#ffffff
style T fill:#1e3a8a,stroke:#3b82f6,color:#ffffff
style P fill:#b45309,stroke:#f59e0b,color:#ffffff
style B fill:#166534,stroke:#22c55e,color:#ffffff
style C fill:#134e4a,stroke:#14b8a6,color:#ffffff
style D fill:#581c87,stroke:#9333ea,color:#ffffffA 5-tier orthogonal architecture: SDK → MCP Tools → Pipeline Orchestration → Processing Engines → Infrastructure, with the Configuration Layer interweaving through everything. Featuring a 10-Stage PDF Pipeline and a 12-Stage WebPage Pipeline that strictly enforce both fallback and competitive execution models.
Documentation Navigator
Document | Content | Who is it for |
Deep dive into 6 tools, MCP Server setup, SDK interfaces, advanced tweaks | All Users | |
5-tier architecture, Pipeline orchestration, engine fallbacks, Smart Mode | Architects / Contributors | |
Environment setup, test framework, CI/CD, PR guidelines | Developers | |
Release history and change logs | Everyone |
Community & Contributions
Beyond the World Wide Web and massive unstructured texts lies an abyss of noise. Only through relentless code evolution can we forge ahead steadily. If you hold the inspiration to pull chaos back into order, please do not hesitate to share:
Before striking your keyboard, flip through the Developer Guide along the way.
Hurl your paradigm-shifting ideas at our Issues or directly submit a Pull Request armed with game-changing power.
Available Tools
10 toolscheck_robots_txtB
Check the robots.txt file for a domain to understand crawling permissions.
This tool helps ensure ethical scraping by checking the robots.txt file of a website to see what crawling rules are in place.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'ethical scraping' and 'crawling rules,' which imply read-only and non-destructive behavior, but it doesn't explicitly state whether this is a read operation, what permissions or rate limits apply, or what happens on errors (e.g., if the robots.txt file is missing). For a tool with no annotations, this leaves significant gaps in understanding its behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately sized and front-loaded: the first sentence states the core purpose, and the second adds value by explaining the ethical context. There's no wasted text, and every sentence contributes meaningfully to understanding the tool's use.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity (one parameter) and the presence of an output schema (which likely covers return values), the description is reasonably complete. It explains what the tool does and why to use it, though it could improve by addressing behavioral aspects like error handling or rate limits. The output schema reduces the need to describe return values in the description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has one parameter ('url') with 0% schema description coverage (no title or description in the schema). The description doesn't add any parameter-specific details beyond implying the 'url' should be a domain for checking robots.txt. Since schema coverage is low, the description doesn't fully compensate by explaining the parameter's format or constraints, but it does provide some context through the tool's purpose.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Check the robots.txt file for a domain to understand crawling permissions.' It specifies the verb ('check'), resource ('robots.txt file'), and goal ('understand crawling permissions'). However, it doesn't explicitly differentiate this from sibling tools like 'get_page_info' or 'scrape_webpage' that might also retrieve web content, though the focus on robots.txt is specific.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides implied usage context: 'This tool helps ensure ethical scraping by checking the robots.txt file...' This suggests it should be used before scraping to comply with rules, but it doesn't explicitly state when to use this tool versus alternatives (e.g., 'scrape_webpage' for general content) or when not to use it (e.g., for non-web domains). The guidance is helpful but not comprehensive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
clear_cacheA
Clear the scraping results cache.
This removes all cached scraping results, forcing fresh requests for all subsequent scraping operations.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses the destructive nature ('removes all cached scraping results') and the behavioral consequence ('forcing fresh requests'), but doesn't mention potential side effects like performance impact, permission requirements, or rate limits. It adequately describes the core behavior but lacks comprehensive operational context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is perfectly front-loaded with the core purpose in the first sentence, followed by a clarifying second sentence about consequences. Both sentences earn their place with no wasted words, making it highly efficient and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simple nature (0 parameters, no annotations, but has output schema), the description is reasonably complete. It explains what the tool does and the immediate effect, though it could benefit from mentioning what the output schema returns. The presence of an output schema reduces the need to describe return values, but some indication of what to expect would enhance completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters with 100% schema description coverage, so the baseline is 4. The description appropriately doesn't discuss parameters since none exist, maintaining focus on the tool's purpose and behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the specific action ('Clear') and target resource ('scraping results cache'), distinguishing it from sibling tools that perform scraping, extraction, or monitoring operations. It precisely defines what the tool does without restating the name.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use this tool ('forcing fresh requests for all subsequent scraping operations'), but doesn't explicitly state when not to use it or mention alternatives. It implies usage in cache management scenarios but lacks explicit exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_linksB
Extract all links from a webpage.
This tool is specialized for link extraction and can filter links by domain, extract only internal links, or exclude specific domains.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions filtering capabilities but doesn't cover critical aspects like whether this is a read-only operation, potential rate limits, authentication needs, error handling, or what the output looks like (though an output schema exists). For a web scraping tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately sized with two sentences: the first states the core purpose, and the second adds specialized capabilities. Every sentence earns its place by providing essential information without redundancy, making it front-loaded and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity (link extraction with filtering), no annotations, and an output schema (which reduces the need to describe return values), the description is somewhat complete but has gaps. It covers the purpose and basic capabilities but lacks behavioral details like error conditions or performance characteristics, making it adequate but not fully comprehensive for informed tool selection.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds some semantic context by mentioning filtering by domain, internal links, and domain exclusion, which aligns with the parameters in the schema. However, with 0% schema description coverage, the schema provides no parameter descriptions, so the description only partially compensates. It doesn't explain the 'url' parameter or provide examples of domain formats, leaving room for improvement given the coverage gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose as 'Extract all links from a webpage' with a specific verb ('extract') and resource ('links from a webpage'). It distinguishes itself from siblings like 'scrape_webpage' or 'extract_structured_data' by focusing solely on link extraction, but doesn't explicitly contrast with these alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context by mentioning filtering capabilities ('filter links by domain, extract only internal links, or exclude specific domains'), suggesting when to use it for specialized link extraction. However, it lacks explicit guidance on when to choose this tool over siblings like 'scrape_webpage' (which might also extract links) or 'extract_structured_data', and doesn't mention any prerequisites or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_structured_dataC
Extract structured data from a webpage using advanced techniques.
Automatically detects and extracts:
Contact information (emails, phone numbers)
Social media links
Addresses
Prices and product information
Article content
data_type can be: all, contact, social, content, products, or addresses
| Name | Required | Description | Default |
|---|---|---|---|
| data_type | No | all | |
| url | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'advanced techniques' but doesn't explain what these entail (e.g., rate limits, authentication needs, potential for blocking, or how it handles dynamic content). For a web extraction tool with no annotations, this leaves significant gaps in understanding operational behavior and risks.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately sized and front-loaded, starting with the core purpose. The bulleted list efficiently details extractable data types, and the final sentence clarifies the data_type parameter. There's minimal waste, though the structure could be slightly improved by integrating the data_type explanation more seamlessly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity (web extraction with 2 parameters), no annotations, and an output schema present, the description is partially complete. It covers the purpose and parameter semantics to some extent but lacks behavioral details and usage guidelines. The output schema likely handles return values, so that gap is mitigated, but overall completeness is limited for effective agent use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds meaning for the 'data_type' parameter by listing its possible values (all, contact, social, etc.) and examples of what each extracts, which is valuable since schema description coverage is 0%. However, it doesn't explain the 'url' parameter beyond what the schema title implies. With 2 parameters and low schema coverage, this partial compensation results in a baseline adequate score.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'extract' and resource 'structured data from a webpage', specifying what the tool does. It lists concrete examples of data types (contact info, social media links, etc.), making the purpose specific. However, it doesn't explicitly distinguish this tool from sibling tools like 'scrape_webpage' or 'extract_links', which might have overlapping functionality.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'scrape_webpage', 'extract_links', and 'scrape_with_stealth' available, there's no indication of scenarios where this tool is preferred, prerequisites, or exclusions. Usage is implied only through the data_type parameter, but not contextualized against other tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
fill_and_submit_formA
Fill and optionally submit a form on a webpage.
This tool can handle various form elements including:
Text inputs
Checkboxes and radio buttons
Dropdown selects
File uploads
Form submission
Useful for interacting with search forms, contact forms, login forms, etc.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. While it mentions the tool can handle various form elements and optional submission, it lacks critical details such as required permissions, potential side effects (e.g., data submission consequences), error handling, or performance considerations like timeouts. This is inadequate for a tool that interacts with webpages and submits data.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately sized and front-loaded, starting with the core purpose in the first sentence, followed by a bulleted list of capabilities and a concise usage note. Every sentence earns its place without redundancy, making it efficient and easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (web form interaction with potential data submission), no annotations, and an output schema that likely handles return values, the description is incomplete. It covers purpose and basic usage but lacks behavioral transparency and detailed parameter guidance, which are critical for safe and effective use. The presence of an output schema mitigates some gaps, but overall completeness is moderate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description adds some value by listing form element types (text inputs, checkboxes, etc.) and use cases, which helps infer the purpose of 'form_data'. However, it does not explain other parameters like 'method', 'submit_button_selector', or 'wait_for_element', leaving significant gaps. With 1 parameter (a nested object with 6 sub-parameters), the description partially compensates but not fully.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose with specific verbs ('fill and optionally submit a form on a webpage') and distinguishes it from siblings like 'scrape_webpage' or 'extract_links' by focusing on form interaction rather than data extraction or navigation. It explicitly mentions the resource (form) and scope (webpage).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use this tool ('useful for interacting with search forms, contact forms, login forms, etc.'), which helps differentiate it from general scraping tools. However, it does not explicitly state when not to use it or name specific alternatives among siblings, such as when to prefer 'scrape_webpage' for non-form interactions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_page_infoA
Get basic information about a webpage (title, description, status).
This is a lightweight tool for quickly checking page accessibility and getting basic metadata without full content extraction.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that the tool is 'lightweight' and for 'quickly checking,' which hints at performance characteristics, but doesn't detail behavioral traits like rate limits, authentication needs, error handling, or what 'status' specifically means (e.g., HTTP status codes). It adds some context but lacks comprehensive behavioral disclosure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately sized and front-loaded: the first sentence states the core purpose, and the second adds usage context. Every sentence earns its place by clarifying the tool's scope and when to use it, with zero wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity (1 parameter) and the presence of an output schema (which handles return values), the description is reasonably complete. It covers purpose and usage guidelines well. However, with no annotations and low schema coverage, it could benefit from more behavioral details, but the output schema mitigates some gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 1 parameter (url) with 0% description coverage, meaning the schema provides no semantic details. The description doesn't add any parameter-specific information beyond what's implied by the tool's purpose (e.g., it doesn't specify URL format requirements or validation). With low schema coverage, the description doesn't compensate adequately, but since there's only one parameter, the baseline is slightly higher.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get basic information about a webpage (title, description, status).' It specifies the verb 'get' and the resource 'webpage' with concrete examples of what information is retrieved. However, it doesn't explicitly differentiate from sibling tools like 'scrape_webpage' or 'extract_structured_data' beyond mentioning it's 'lightweight' and for 'quickly checking page accessibility.'
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use this tool: 'for quickly checking page accessibility and getting basic metadata without full content extraction.' This implies it's suitable for lightweight checks versus more intensive extraction tools. However, it doesn't explicitly name alternatives or state when not to use it, such as for detailed content analysis.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_server_metricsB
Get server performance metrics and statistics.
Returns information about:
Request counts and success rates
Performance metrics
Method usage statistics
Error categories
Cache statistics
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. While it lists the categories of metrics returned, it doesn't address critical behavioral aspects like whether this requires authentication, if it's a read-only operation, what time range the metrics cover, or if there are rate limits. The description is insufficient for a mutation-free tool with zero annotation coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficiently structured with a clear purpose statement followed by a bulleted list of return categories. Every sentence earns its place, though the bulleted list could be slightly more concise by integrating it into the flow rather than as a separate section.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there's an output schema (which handles return value documentation), zero parameters, and no annotations, the description is reasonably complete for its core purpose. However, it lacks important context about when to use it versus siblings and behavioral constraints, making it only minimally adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, and schema description coverage is 100%, so there's no parameter documentation burden. The description appropriately doesn't waste space discussing non-existent parameters, earning a high score for correctly handling this edge case.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose with the verb 'Get' and resource 'server performance metrics and statistics', making it immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'get_page_info' or 'scrape_webpage' which might also provide performance-related data in different contexts, preventing a perfect score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'get_page_info' that might overlap in providing server-related data, there's no indication of when this specific metrics tool is preferred or what contexts it's designed for.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_multiple_webpagesB
Scrape multiple webpages concurrently.
This tool allows you to scrape multiple URLs at once, which is much faster than scraping them one by one. All URLs will be processed concurrently.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It mentions concurrency and speed but lacks critical behavioral details such as rate limits, error handling (e.g., if some URLs fail), authentication needs, or what 'processed concurrently' entails (e.g., thread count, timeouts). For a tool with potential complexity and no annotation coverage, this is a significant gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately sized and front-loaded: the first sentence states the core purpose, and the following sentences add useful context about concurrency and speed. Every sentence earns its place with no wasted words, making it easy to scan and understand quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (concurrent web scraping with configurable methods), no annotations, 0% schema description coverage, but an output schema exists, the description is incomplete. It covers the high-level purpose and benefit but misses details on parameters, behavioral traits, and error handling. The output schema reduces the need to explain return values, but other gaps remain significant.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for undocumented parameters. It mentions 'URLs' but doesn't explain the 'extract_config' or 'method' parameters beyond what the schema provides (e.g., what 'auto' means, how configuration works). With 1 parameter (a nested object with 3 sub-parameters) and no schema descriptions, the description adds minimal value over the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Scrape multiple webpages concurrently.' It specifies the verb (scrape) and resource (multiple webpages) with the key feature of concurrency. However, it doesn't explicitly differentiate from sibling tools like 'scrape_webpage' (singular) or 'scrape_with_stealth' (stealth-focused), missing full sibling differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context by stating it's 'much faster than scraping them one by one,' suggesting it should be used for batch scraping. However, it doesn't provide explicit guidance on when to use this tool versus alternatives like 'scrape_webpage' for single URLs or 'scrape_with_stealth' for stealth needs, nor does it mention any exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_webpageA
Scrape a single webpage and extract its content.
This tool can scrape web pages using different methods:
auto: Automatically choose the best method
simple: Fast HTTP requests (no JavaScript)
scrapy: Robust scraping with Scrapy framework
selenium: Full browser rendering (supports JavaScript)
You can specify extraction rules to get specific data from the page.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It mentions scraping methods and extraction rules but doesn't disclose critical behavioral traits like rate limits, authentication needs, potential for blocking, error handling, or what 'extract its content' entails. For a web scraping tool with zero annotation coverage, this leaves significant gaps in understanding its behavior and risks.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately sized and front-loaded: the first sentence states the core purpose, followed by bullet points for methods and a final sentence on extraction rules. Every sentence earns its place with no wasted words, making it easy to scan and understand quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (web scraping with multiple methods) and no annotations, the description is moderately complete but has gaps. It covers purpose and methods well, and an output schema exists (reducing need to explain returns), but it lacks details on behavioral aspects like limitations or errors. For a tool with siblings like 'scrape_with_stealth', more context on trade-offs would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It adds meaningful context beyond the schema by explaining the four scraping methods (auto, simple, scrapy, selenium) and mentioning extraction rules, which helps interpret the 'extract_config' and 'method' parameters. However, it doesn't detail the 'wait_for_element' parameter or provide examples for extraction rules, leaving some semantic gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose with a specific verb ('scrape') and resource ('single webpage'), and distinguishes it from siblings like 'scrape_multiple_webpages' by specifying 'single'. It also mentions content extraction, which differentiates it from tools like 'extract_links' or 'get_page_info'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context on when to use different scraping methods (auto, simple, scrapy, selenium) based on needs like speed or JavaScript support, but doesn't explicitly state when NOT to use this tool or name alternatives among siblings. It implies usage scenarios but lacks explicit exclusions or comparisons to tools like 'scrape_with_stealth'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_with_stealthB
Scrape a webpage using advanced stealth techniques to avoid detection.
This tool uses sophisticated anti-detection methods including:
Undetected browser automation
Randomized behavior patterns
Human-like interactions
Advanced evasion techniques
Use this for websites with strong anti-bot protection.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It does describe key behavioral traits: 'advanced stealth techniques,' 'anti-detection methods,' and lists specific techniques like undetected browser automation and randomized patterns. However, it lacks details about potential risks, rate limits, authentication needs, or what happens when detection occurs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is appropriately sized with clear front-loading of the main purpose. The bullet-point list of techniques is organized and adds value. However, the final usage guideline sentence could be more integrated with the opening statement for better flow.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of stealth web scraping (which involves evasion techniques and potential ethical/legal considerations), the description is moderately complete. It covers the 'why' (anti-bot protection) and 'how' (stealth techniques) but lacks details about parameters, output format (though output schema exists), error conditions, and limitations. The presence of an output schema helps but doesn't compensate for missing behavioral context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for undocumented parameters. The description mentions 'stealth techniques' and 'anti-detection methods' but provides no information about the actual parameters (url, method, scroll_page, wait_for_element, extract_config). It doesn't explain what these parameters mean or how they relate to the stealth functionality.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose as 'Scrape a webpage using advanced stealth techniques to avoid detection,' which is a specific verb+resource combination. However, it doesn't explicitly differentiate from sibling tools like 'scrape_webpage' or 'scrape_multiple_webpages' beyond mentioning stealth techniques for anti-bot protection.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use this tool: 'Use this for websites with strong anti-bot protection.' This gives practical guidance about the target scenario. However, it doesn't explicitly state when NOT to use it or mention specific alternatives among the sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
10 tool updates
v1.0.0- First observed
check_robots_txt - First observed
clear_cache - First observed
extract_links - First observed
extract_structured_data - First observed
fill_and_submit_form - First observed
get_page_info - First observed
get_server_metrics - First observed
scrape_multiple_webpages - First observed
scrape_webpage - First observed
scrape_with_stealth
TDQS
Most tools have distinct purposes, but there is some overlap between scrape_webpage and scrape_with_stealth, as both scrape single webpages with different methods. The descriptions help clarify the stealth aspect, but an agent might initially confuse them. Other tools like extract_links vs. extract_structured_data are well-differentiated.
All tool names follow a consistent snake_case pattern with clear verb_noun structures, such as check_robots_txt, clear_cache, and scrape_webpage. There are no deviations in naming conventions, making the set predictable and easy to parse.
With 10 tools, the server is well-scoped for web scraping and data extraction tasks. Each tool serves a specific function, from basic scraping to advanced operations like form handling and stealth techniques, without feeling overloaded or sparse.
The toolset covers a broad range of web scraping operations, including single and multi-page scraping, data extraction, form interaction, and performance metrics. A minor gap is the lack of tools for managing scraping sessions or saving/exporting data, but core workflows are well-supported.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
One MCP server for 180+ live web-data APIs returning clean JSON from sites that block scrapers.
- HasDataOAuthcom.hasdata
All HasData scraping tools in one MCP server: Google, TikTok, Instagram, maps, e-commerce and more.
Official Octoparse MCP server for template discovery, cloud tasks, and structured data export.
MCP server for web extraction and rendering via AceDataCloud WebExtrator
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA powerful MCP server that offers web scraping capabilities unrestricted by robots.txt, supports multiple HTTP methods and custom request Settings.2Apache 2.0
- AlicenseAqualityDmaintenanceA headless web-scraping MCP server built on Scrapy, providing tools for polite fetching, CSS/XPath extraction, link/table extraction, sitemap and robots.txt reading, and bounded asynchronous crawls.10MIT
- FlicenseNot gradedqualityDmaintenanceSecure, agent-driven web data extraction MCP server that extracts structured data from websites using APIs, RSS, and HTML without requiring a browser.93-
- FlicenseNot gradedqualityCmaintenanceOpen-source web scraping MCP server with anti-bot bypass, enabling AI models to scrape, crawl, search, and extract data from any website without API keys or limits.-
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/ThreeFish-AI/negentropy-perceives'
If you have feedback or need assistance with the MCP directory API, please join our Discord server