Web scraping tools and techniques

Glama

Search for:

Web scraping tools and techniques

View all MCP Servers

Why this server?
This server is an excellent fit as it explicitly states 'Enables AI models to scrape and extract data from any website globally' and highlights its ability to bypass anti-bot systems, which is core functionality for a web crawler (爬虫).
Thordata MCP Server
xja1023789-collab
-
security
A
license
-
quality
Enables AI models to scrape and extract data from any website globally using Thordata's 195+ country proxy network. Bypasses anti-bot systems and renders JavaScript content, outputting structured data in Markdown, HTML, or Links format.
Last updated -
MIT License
Why this server?
This server is a direct match, as its description mentions 'Advanced search and retrieval for web crawler data' and specifies support for various 'crawlers', directly addressing the '爬虫' query.
mcp-server-webcrawl
pragmar
-
security
F
license
-
quality
Bridge the gap between your web crawl and AI language models. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content. Supports WARC, wget, InterroBot, Katana, and SiteOne crawlers.
Last updated -
26
Python
Why this server?
The server directly offers 'web scraping and crawling capabilities for LLM clients' and lists popular automation tools like Playwright and Puppeteer, making it a strong match for '爬虫'.
AnyCrawl MCP Server
any4ai
-
security
F
license
-
quality
Enables web scraping and crawling capabilities for LLM clients, supporting single-page scraping, multi-page website crawling, and web search with multiple engines (Playwright, Cheerio, Puppeteer) and flexible output formats including markdown, HTML, text, and screenshots.
Last updated -
16
4
Why this server?
This server is a powerful match, described as a 'web scraping MCP server built on Scrapy' that includes 'anti-detection techniques' and 'concurrent crawling', which are advanced features for web crawling (爬虫).
Scrapy MCP Server
ThreeFish-AI
-
security
F
license
-
quality
A powerful web scraping MCP server built on Scrapy and FastMCP that supports multiple scraping methods (HTTP, Scrapy, browser automation), anti-detection techniques, form handling, and concurrent crawling. Designed for commercial environments with enterprise-grade features like intelligent retry mechanisms, performance monitoring, and configurable data extraction.
Last updated -
10
1
MIT License
Why this server?
This server enables 'undetectable browser automation' and 'real-world web scraping' by bypassing anti-bot systems, making it highly relevant for discreet and effective web crawling (爬虫).
Stealth Browser MCP
vibheksoni
-
security
A
license
-
quality
Enables AI agents to perform undetectable browser automation that bypasses Cloudflare, antibots, and social media blocks. Provides 105 tools for element extraction, network debugging, and real-world web scraping with a 98.7% success rate on protected sites.
Last updated -
74
MIT License
Why this server?
The server's description, 'Enables reverse engineering of web applications and chat interfaces through browser automation, network traffic capture, and streaming API discovery', is indicative of sophisticated web crawling (爬虫) and data extraction methods.
WebScout MCP
pyscout
-
security
F
license
-
quality
Enables reverse engineering of web applications and chat interfaces through browser automation, network traffic capture, and streaming API discovery. Provides comprehensive tools for analyzing network patterns, capturing streaming responses, and automating complex web interactions.
Last updated -
14
2
Why this server?
This server explicitly mentions 'intelligent web scraping through a browser automation tool' that can 'extract content from various websites', directly aligning with the functionality of a web crawler (爬虫).
Browser Automation MCP Server
Raghu6798
-
security
F
license
-
quality
Enables intelligent web scraping through a browser automation tool that can search Google, navigate to webpages, and extract content from various websites including GitHub, Stack Overflow, and documentation sites.
Last updated -
1
Why this server?
This server explicitly states that it 'enables AI assistants to scrape web content with high accuracy and flexibility', directly fitting the user's search for a '爬虫'.
Prysm MCP Server
pinkpixel-dev
A
security
A
license
A
quality
A Model Context Protocol server enabling AI assistants to scrape web content with high accuracy and flexibility, supporting multiple scraping modes and content formatting options.
Last updated -
4
29
2
MIT License
Why this server?
As a 'web scraping server' offering 'content extraction rules' and support for dynamic websites, this server is a direct and strong match for the '爬虫' query.
Web Scraper MCP Server
naku111
A
security
A
license
A
quality
A TypeScript-based web scraping server built on the Model Context Protocol that offers multiple export formats, content extraction rules, and support for both static and dynamic (SPA) websites.
Last updated -
7
14
1
MIT License
Why this server?
This tool is focused on 'web content extraction' and mentions 'polite crawling capabilities', which are key aspects of a web crawler (爬虫) designed for efficient data gathering.
read-website-fast
just-every
A
security
A
license
A
quality
Fast, token-efficient web content extraction tool that converts websites to clean Markdown for AI agents, featuring smart caching, content extraction with Mozilla Readability, and polite crawling capabilities.
Last updated -
1
148
91
MIT License