Web crawling techniques and tools

Glama

Search for:

Web crawling techniques and tools

View all MCP Servers

Why this server?
Explicitly enables AI models to 'scrape and extract data from any website globally' and bypasses anti-bot systems, which is core to web crawling/scraping.
Thordata MCP Server
xja1023789-collab
-
security
A
license
-
quality
Enables AI models to scrape and extract data from any website globally using Thordata's 195+ country proxy network. Bypasses anti-bot systems and renders JavaScript content, outputting structured data in Markdown, HTML, or Links format.
Last updated -
MIT License
Why this server?
Enables 'web scraping and crawling capabilities for LLM clients,' supporting single-page scraping and multi-page website crawling.
AnyCrawl MCP Server
any4ai
-
security
F
license
-
quality
Enables web scraping and crawling capabilities for LLM clients, supporting single-page scraping, multi-page website crawling, and web search with multiple engines (Playwright, Cheerio, Puppeteer) and flexible output formats including markdown, HTML, text, and screenshots.
Last updated -
16
4
Why this server?
Provides 'Advanced search and retrieval for web crawler data' and lists support for several popular web crawler outputs, directly matching the search term.
mcp-server-webcrawl
pragmar
-
security
F
license
-
quality
Bridge the gap between your web crawl and AI language models. With mcp-server-webcrawl, your AI client filters and analyzes web content under your direction or autonomously, extracting insights from your web content. Supports WARC, wget, InterroBot, Katana, and SiteOne crawlers.
Last updated -
26
Python
Why this server?
Offers 'robust search capabilities and LLM-optimized web content understanding,' specifically designed for intelligent information retrieval and scraping.
Crawl4AI MCP Server
weidwonder
-
security
A
license
-
quality
Crawl4AI MCP Server is an intelligent information retrieval server offering robust search capabilities and LLM-optimized web content understanding, utilizing multi-engine search and intelligent content extraction to efficiently gather and comprehend internet information.
Last updated -
118
MIT License
Why this server?
Enables AI assistants to 'scrape web content with high accuracy and flexibility,' supporting multiple scraping modes and content formatting options.
Prysm MCP Server
pinkpixel-dev
A
security
A
license
A
quality
A Model Context Protocol server enabling AI assistants to scrape web content with high accuracy and flexibility, supporting multiple scraping modes and content formatting options.
Last updated -
4
29
2
MIT License
Why this server?
Enables web search, 'scraping, crawling, searching, and data extraction' through the Firecrawl API, providing essential web crawling functionality.
Firecrawl MCP Server
mcma123
-
security
A
license
-
quality
A Model Context Protocol server that enables AI assistants to perform advanced web scraping, crawling, searching, and data extraction through the Firecrawl API.
Last updated -
39,239
MIT License
Why this server?
Focuses on advanced web crawling by enabling 'undetectable browser automation that bypasses Cloudflare, antibots,' specifically for web scraping tasks.
Stealth Browser MCP
vibheksoni
-
security
A
license
-
quality
Enables AI agents to perform undetectable browser automation that bypasses Cloudflare, antibots, and social media blocks. Provides 105 tools for element extraction, network debugging, and real-world web scraping with a 98.7% success rate on protected sites.
Last updated -
74
MIT License
Why this server?
A powerful server dedicated to 'web scraping' built on the Scrapy framework, supporting multiple scraping methods and concurrent crawling.
Scrapy MCP Server
ThreeFish-AI
-
security
F
license
-
quality
A powerful web scraping MCP server built on Scrapy and FastMCP that supports multiple scraping methods (HTTP, Scrapy, browser automation), anti-detection techniques, form handling, and concurrent crawling. Designed for commercial environments with enterprise-grade features like intelligent retry mechanisms, performance monitoring, and configurable data extraction.
Last updated -
10
1
MIT License
Why this server?
Enables 'web search and webpage scraping' and executing batch webpage scraping and content extraction using browser automation.
Spider MCP
Bosegluon2
A
security
A
license
A
quality
Enables web searching and webpage scraping using pure crawler technology without requiring official APIs. Supports Bing web and news search, batch webpage scraping, and content extraction through Puppeteer automation.
Last updated -
4
1
MIT License
Why this server?
Provides 'Web crawling and RAG implementation' that enables AI agents to scrape websites and perform semantic search over the crawled content.
Crawl4AI RAG MCP Server
coleam00
-
security
A
license
-
quality
Web crawling and RAG implementation that enables AI agents to scrape websites and perform semantic search over the crawled content, storing everything in Supabase for persistent knowledge retrieval.
Last updated -
1,742
MIT License