The MCP Web Scrape server is a comprehensive web content extraction and analysis tool that converts web pages into clean, agent-friendly formats with smart caching and ethical compliance.
Content Extraction & Transformation: Convert HTML to clean Markdown/text/JSON with citations (extract_content), extract plain text (extract_text_only), summarize content with customizable formats (summarize_content), translate content (translate_content), convert pages to PDF (convert_to_pdf), and generate word clouds (generate_word_cloud).
Structured Data Extraction: Extract links with filtering (extract_links), images with metadata (extract_images), forms with validation rules (extract_forms), tables with export options (extract_tables), social media links (extract_social_media), contact information (extract_contact_info), heading hierarchy (extract_headings), RSS/Atom feeds (extract_feeds), and structured data including JSON-LD, microdata, RDFa, OpenGraph, and schema.org markup (extract_structured_data, extract_schema_markup).
Content Analysis: Search within pages with regex support (search_content), extract keywords (extract_keywords), analyze readability using multiple metrics (analyze_readability), detect language with confidence scores (detect_language), extract named entities (extract_entities), perform sentiment analysis at document/paragraph/sentence level (sentiment_analysis), classify content into categories (classify_content), and compare content between URLs (compare_content).
SEO & Marketing Tools: Analyze competitors for insights (analyze_competitors), generate optimized meta tags (generate_meta_tags), check broken links and redirects (check_broken_links), analyze page speed with Core Web Vitals (analyze_page_speed), validate HTML structure (validate_html), analyze overall performance including SEO and accessibility (analyze_performance), and generate sitemaps (generate_sitemap).
Security & Privacy: Scan for vulnerabilities including XSS and CSRF (scan_vulnerabilities), check SSL certificates with chain details (check_ssl_certificate), analyze cookies for security flags (analyze_cookies), detect tracking scripts (detect_tracking), and check privacy policy compliance with GDPR, CCPA, COPPA, and PIPEDA (check_privacy_policy).
Monitoring & Tracking: Monitor uptime with configurable intervals (monitor_uptime), track content changes with similarity analysis (monitor_changes, track_changes_detailed), analyze traffic patterns (analyze_traffic_patterns), and benchmark performance against competitors (benchmark_performance).
Utility & Management: Process multiple URLs efficiently in batches (batch_extract), validate robots.txt compliance (validate_robots), check URL accessibility (check_url_status), manage cache with statistics and selective clearing (clear_cache, get_cache_stats), and generate comprehensive reports in JSON, HTML, or Markdown formats (generate_reports).
Key Benefits: Provides clean Markdown output optimized for AI agents, citation links for fact verification, deterministic and cached results with ETag/304 support, ethical scraping with robots.txt respect and rate limiting, and supports both STDIO and HTTP/SSE transports.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@MCP Web Scrape scrape the main text from https://news.ycombinator.com and format it as markdown"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Clean, cached web content for agents—Markdown + citations, robots-aware, ETag/304 caching.
Current Version: 1.0.7
# Extract content from any webpage
npx mcp-web-scrape@1.0.7
# Example: Extract from a news article
> extract_content https://news.ycombinator.com
✅ Extracted 1,247 words with 5 citations
📄 Clean Markdown ready for your AI agentRelated MCP server: crawl-mcp-server
# Extract all forms from a webpage
> extract_forms https://example.com/contact
✅ Found 3 forms with 12 input fields
# Parse tables into structured data
> extract_tables https://example.com/data --format json
✅ Extracted 5 tables with 247 rows
# Find social media profiles
> extract_social_media https://company.com
✅ Found Twitter, LinkedIn, Facebook profiles
# Analyze sentiment of content
> sentiment_analysis https://blog.example.com/article
✅ Sentiment: Positive (0.85), Emotional tone: Optimistic
# Extract named entities
> extract_entities https://news.example.com/article
✅ Found 12 people, 8 organizations, 5 locations
# Check for security vulnerabilities
> scan_vulnerabilities https://mysite.com
✅ No XSS vulnerabilities found, 2 header improvements suggested
# Analyze competitor SEO
> analyze_competitors ["https://competitor1.com", "https://competitor2.com"]
✅ Competitor analysis complete: keyword gaps identified
# Monitor uptime and performance
> monitor_uptime https://mysite.com --interval 300
✅ Uptime: 99.9%, Average response: 245ms
# Generate comprehensive report
> generate_reports https://website.com --metrics ["seo", "performance", "security"]
✅ Generated 15-page analysis report# Install globally
npm install -g mcp-web-scrape@1.0.7
# Try it instantly (latest version)
npx mcp-web-scrape@latest
# Try specific version
npx mcp-web-scrape@1.0.7
# Or start HTTP server
node dist/http.jsAdd to your ~/Library/Application Support/ChatGPT/config.json:
{
"mcpServers": {
"web-scrape": {
"command": "npx",
"args": ["mcp-web-scrape@1.0.7"]
}
}
}Add to your ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"web-scrape": {
"command": "npx",
"args": ["mcp-web-scrape@1.0.7"]
}
}
}Tool | Description |
| Convert HTML to clean Markdown with citations |
| AI-powered content summarization |
| Extract title, description, author, keywords |
| Get all links with filtering options |
| Extract images with alt text and dimensions |
| Search within page content |
| Verify URL accessibility |
| Check robots.txt compliance |
| Parse JSON-LD, microdata, RDFa |
| Compare two pages for changes |
| Process multiple URLs efficiently |
| View cache performance metrics |
| Manage cached content |
Tool | Description |
| Extract form elements, fields, and validation rules |
| Parse HTML tables with headers and structured data |
| Find social media links and profiles |
| Discover emails, phone numbers, and addresses |
| Analyze heading structure (H1-H6) for content hierarchy |
| Discover and parse RSS/Atom feeds |
Tool | Description |
| Convert web pages to PDF format with customizable settings |
| Extract plain text content without formatting or HTML |
| Generate word frequency analysis and word cloud data |
| Translate web page content to different languages |
| Extract important keywords and phrases from content |
Tool | Description |
| Analyze text readability using various metrics (Flesch, Gunning-Fog, etc.) |
| Detect the primary language of web page content |
| Extract named entities (people, places, organizations) |
| Analyze sentiment and emotional tone of content |
| Classify content into categories and topics |
Tool | Description |
| Analyze competitor websites for SEO and content insights |
| Extract and validate schema.org structured data |
| Check for broken links and redirects on pages |
| Analyze page loading speed and performance metrics |
| Generate optimized meta tags for SEO |
Tool | Description |
| Scan pages for common security vulnerabilities |
| Check SSL certificate validity and security details |
| Analyze cookies and tracking mechanisms |
| Detect tracking scripts and privacy concerns |
| Analyze privacy policy compliance and coverage |
Tool | Description |
| Monitor website uptime and availability |
| Advanced change tracking with similarity analysis |
| Analyze website traffic patterns and trends |
| Benchmark performance against competitors |
| Generate comprehensive analysis reports |
Tool | Description |
| Track content changes over time with similarity analysis |
| Measure page performance, SEO, and accessibility metrics |
| Crawl websites to generate comprehensive sitemaps |
| Validate HTML structure, accessibility, and SEO compliance |
Deterministic Results → Same URL always returns identical content
Smart Citations → Every fact links back to its source
Robots Compliant → Respects robots.txt and rate limits
Lightning Fast → ETag/304 caching + persistent storage
Agent-Optimized → Clean Markdown instead of messy HTML
✅ Respects robots.txt by default
✅ Rate limiting prevents server overload
✅ No paywall bypass - ethical scraping only
✅ User-Agent identification for transparency
# Install specific version
npm install -g mcp-web-scrape@1.0.7
# Or use directly (latest)
npx mcp-web-scrape@latest
# Or use specific version
npx mcp-web-scrape@1.0.7# Environment variables
export MCP_WEB_SCRAPE_CACHE_DIR="./cache"
export MCP_WEB_SCRAPE_USER_AGENT="MyBot/1.0"
export MCP_WEB_SCRAPE_RATE_LIMIT="1000"STDIO (default)
mcp-web-scrapeHTTP/SSE
node dist/http.js --port 3000Access cached content as MCP resources:
cache://news.ycombinator.com/path → Cached page content
cache://stats → Cache statistics
cache://robots/news.ycombinator.com → Robots.txt statusWe love contributions! See CONTRIBUTING.md for guidelines.
Good First Issues:
Add new content extractors
Improve error handling
Write more tests
Enhance documentation
MIT © Mahipal
Built with ❤️ for the Model Context Protocol ecosystem
Analyze competitor websites for SEO and content insights
Inputs: urls (required), metrics, useCache.
Analyze cookies set by web pages for privacy and security
Inputs: url (required), includeThirdParty, checkSecurity.
Analyze page loading speed and performance metrics
Inputs: url (required), device, metrics.
Analyze web page performance metrics
Inputs: url (required), metrics, useCache.
Analyze text readability using various metrics
Inputs: url (required), metrics, useCache.
Analyze traffic patterns and user behavior indicators
Inputs: url (required), timeframe, metrics.
Extract content from multiple URLs in a single operation
Inputs: urls (required), format, maxConcurrent, useCache.
Benchmark website performance against competitors and industry standards
Inputs: url (required), competitors, metrics.
Check for broken links and redirects on web pages
Inputs: url (required), checkExternal, timeout, useCache.
Analyze privacy policy content and compliance
Inputs: url (required), regulations, useCache.
Check SSL certificate validity and security details
Inputs: url (required), includeChain.
Check if URL is accessible and get HTTP status codes
Inputs: url (required), followRedirects.
Classify web content into categories and topics
Inputs: url (required), categories, useCache.
Clear cached content entries
Inputs: url.
Compare content between two URLs or cached versions
Inputs: url1 (required), url2 (required), compareType, useCache.
Convert web page content to PDF format
Inputs: url (required), format, includeImages, useCache.
Detect the primary language of web page content
Inputs: url (required), confidence, useCache.
Detect tracking scripts and privacy-related elements
Inputs: url (required), trackerTypes, useCache.
Extract contact information like emails, phones, addresses from web pages
Inputs: url (required), types, useCache.
Extract and clean content from a web page, returning Markdown with citation
Inputs: url (required), format, includeImages, includeLinks, +2 more.
Extract named entities (people, places, organizations) from web content
Inputs: url (required), entityTypes, useCache.
Discover and parse RSS/Atom feeds from web pages
Inputs: url (required), maxItems, includeContent, useCache.
Extract form elements and their structure from web pages
Inputs: url (required), includeHidden, includeDisabled, useCache.
Extract document structure and heading hierarchy from web pages
Inputs: url (required), levels, includeText, useCache.
Extract all images from a web page with metadata
Inputs: url (required), includeAltText, includeDimensions, useCache.
Extract important keywords and phrases from web content
Inputs: url (required), maxKeywords, includePhrases, useCache.
Extract all links from a web page with filtering options
Inputs: url (required), linkType, includeAnchorText, useCache.
Extract and validate schema.org structured data markup
Inputs: url (required), schemaTypes, validate, useCache.
Extract social media links and metadata from web pages
Inputs: url (required), platforms, useCache.
Extract JSON-LD, microdata, and schema.org data
Inputs: url (required), dataTypes, useCache.
Extract and parse HTML tables with optional CSV export
Inputs: url (required), format, includeHeaders, minRows, +1 more.
Extract plain text content without any formatting or HTML
Inputs: url (required), removeWhitespace, useCache.
Generate optimized meta tags for SEO based on content analysis
Inputs: url (required), targetKeywords, useCache.
Generate comprehensive reports combining multiple analysis tools
Inputs: url (required), reportType, format, useCache.
Generate sitemap by crawling website pages
Inputs: url (required), maxDepth, maxPages, includeExternal, +1 more.
Generate word frequency analysis and word cloud data from web content
Inputs: url (required), maxWords, minLength, useCache.
Get detailed cache statistics and usage information
Inputs: includeEntries.
Extract meta tags, title, description, keywords from web pages
Inputs: url (required), useCache.
Monitor web page content changes over time
Inputs: url (required), interval, threshold, useCache.
Monitor website uptime and availability
Inputs: url (required), interval, timeout, expectedStatus.
Scan web pages for common security vulnerabilities
Inputs: url (required), scanTypes, useCache.
Search for specific text patterns within extracted content
Inputs: content (required), query (required), caseSensitive, useRegex, +1 more.
Analyze sentiment and emotional tone of web content
Inputs: url (required), granularity, useCache.
Generate a summary of already extracted content
Inputs: content (required), maxLength, format.
Track detailed changes in web page content with diff analysis
Inputs: url (required), sections, sensitivity, useCache.
Translate web page content to different languages
Inputs: url (required), targetLanguage (required), sourceLanguage, useCache.
Validate HTML structure, accessibility, and SEO
Inputs: url (required), checks, useCache.
Check robots.txt compliance for specific URLs
Inputs: url (required), userAgent.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 48 tools
Most tools have distinct purposes, but there is notable overlap in some areas, such as analyze_performance and analyze_page_speed both focusing on performance metrics, and extract_structured_data and extract_schema_markup both handling structured data extraction. The descriptions help differentiate them, but agents might still face confusion in selecting the right tool for specific tasks.
Tool names follow a highly consistent verb_noun pattern throughout, such as analyze_competitors, extract_content, and validate_html. There are no deviations in naming conventions, making the set predictable and easy to navigate for agents.
With 48 tools, the count is excessive for a web scraping server, leading to potential bloat and complexity. A more focused set of 10-20 tools could cover the domain effectively without overwhelming agents, as many tools could be consolidated or omitted.
The tool set provides comprehensive coverage for web scraping and analysis, including extraction, analysis, monitoring, validation, and reporting. There are no obvious gaps; it supports full workflows from data retrieval to insights, ensuring agents can handle diverse tasks without dead ends.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Unified 15-tool SEO, SERP, clean Markdown & GEO (LLM citation Share-of-Voice) MCP server.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
