Web scraping and crawling tools

Glama

Search for:

Web scraping and crawling tools

View all MCP Servers

Why this server?
This server is an excellent fit as it explicitly enables AI models to 'scrape and extract data from any website globally,' bypassing anti-bot systems, which is the core function of a web crawler.
Thordata MCP Server
xja1023789-collab
-
security
A
license
-
quality
Enables AI models to scrape and extract data from any website globally using Thordata's 195+ country proxy network. Bypasses anti-bot systems and renders JavaScript content, outputting structured data in Markdown, HTML, or Links format.
Last updated -
MIT License
Why this server?
This server directly addresses the request by providing comprehensive 'web scraping and crawling capabilities for LLM clients' using tools like Playwright and Puppeteer.
AnyCrawl MCP Server
any4ai
-
security
F
license
-
quality
Enables web scraping and crawling capabilities for LLM clients, supporting single-page scraping, multi-page website crawling, and web search with multiple engines (Playwright, Cheerio, Puppeteer) and flexible output formats including markdown, HTML, text, and screenshots.
Last updated -
16
4
Why this server?
Focuses on AI-powered web scraping and content conversion into LLM-friendly Markdown, which aligns perfectly with the goal of building a web crawler tool.
ScrapeGraph MCP Serverofficial
ScrapeGraphAI
A
security
A
license
A
quality
A production-ready Model Context Protocol server that enables language models to leverage AI-powered web scraping capabilities, offering tools for transforming webpages to markdown, extracting structured data, and executing AI-powered web searches.
Last updated -
5
39
MIT License
Why this server?
Explicitly defined as a 'web scraping server' supporting multiple export formats, making it a direct solution for a web crawler requirement.
Web Scraper MCP Server
naku111
A
security
A
license
A
quality
A TypeScript-based web scraping server built on the Model Context Protocol that offers multiple export formats, content extraction rules, and support for both static and dynamic (SPA) websites.
Last updated -
7
14
1
MIT License
Why this server?
This server is designed for high-performance web crawling and information retrieval, integrating web content analysis for AI assistants.
Crawl4AI MCP Server
weidwonder
-
security
A
license
-
quality
Crawl4AI MCP Server is an intelligent information retrieval server offering robust search capabilities and LLM-optimized web content understanding, utilizing multi-engine search and intelligent content extraction to efficiently gather and comprehend internet information.
Last updated -
118
MIT License
Why this server?
Uses the Playwright framework to enable headless 'browser automation and web page interactions,' a common technique used to scrape data from dynamic, JavaScript-heavy websites.
Playwright MCP
mattreya
-
security
A
license
-
quality
Enables LLMs to perform browser automation and web page interactions using Playwright's accessibility tree instead of screenshots. Provides fast, deterministic web automation through structured data without requiring vision models.
Last updated -
524,380
Apache 2.0
Why this server?
Specifically geared towards handling and retrieving data from various web crawler outputs (WARC, wget, etc.), confirming its relevance to web scraping/crawling tasks.
mcp-server-webcrawl
pragmar
-
security
F
license
-
quality
Bridge the gap between your web crawl and AI language models. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content. Supports WARC, wget, InterroBot, Katana, and SiteOne crawlers.
Last updated -
26
Python
Why this server?
Scraping often results in messy HTML; this tool specializes in cleaning and transforming raw webpage content into usable, LLM-optimized Markdown.
Mozilla Readability Parser MCP Server
emzimmer
A
security
A
license
A
quality
Extracts and transforms webpage content into clean, LLM-optimized Markdown. Returns article title, main content, excerpt, byline and site name. Uses Mozilla's Readability algorithm to remove ads, navigation, footers and non-essential elements while preserving the core content structure.
Last updated -
1
16
14
MIT License
Why this server?
Covers both browser automation and web reverse engineering, necessary skills for sophisticated web crawling and data extraction.
WebScout MCP
pyscout
-
security
F
license
-
quality
Enables reverse engineering of web applications and chat interfaces through browser automation, network traffic capture, and streaming API discovery. Provides comprehensive tools for analyzing network patterns, capturing streaming responses, and automating complex web interactions.
Last updated -
14
2
Why this server?
Offers web browser automation capabilities using Selenium WebDriver, providing an alternative, reliable method for simulating human browsing behavior during crawling.
MCP Selenium WebDriver
Nixon-Suarez
-
security
F
license
-
quality
A complete Model Context Protocol server that implements Selenium WebDriver for browser automation with advanced detection evasion capabilities, multi-browser support, and comprehensive web interaction functionalities.
Last updated -

ScrapeGraph MCP Serverofficial