A
licenseB
qualityD
maintenanceA local web scraping MCP server with RAG capabilities that provides intelligent web search, content extraction, and screenshot tools without requiring API keys.
4
8 npm
MIT
freecrawl
by RistrixOP
Provides anti-bot bypass for Akamai-protected websites using patchright, enabling scraping of Akamai-challenged pages.
Provides anti-bot bypass for Cloudflare-protected websites using flaresolverr, enabling scraping of Cloudflare-challenged pages.
Integrates DuckDuckGo search to allow the server to perform web searches as part of the Search feature.
Provides OpenAI-compatible tool definitions for use with LLMs, allowing AI models to invoke scraping, crawling, search, and other actions via function calling. Also supports LLM-based extraction and summarization.
Integrates SearXNG as an alternative search engine for the Search feature, providing flexible search capabilities.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@freecrawlscrape https://example.com to markdown"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
🔥 Freecrawl
Open-source web scraping API with anti-bot bypass. Drop-in Firecrawl alternative. AI-first: MCP server, OpenAI tool definitions, REST API — any AI model can use it. No API keys, no credits, no limits.
Quick Start
pip install -r requirements.txt
patchright install chromium
python -m freecrawl serve # REST API on :9100Related MCP server: scraperapi-mcp-server
AI Integration
MCP Server (Claude, Cursor, any MCP client)
{"mcpServers":{"freecrawl":{"command":"python","args":["-m","freecrawl.mcp_server"]}}}OpenAI Tool Definitions
from freecrawl.ai_tools import get_tools
tools = get_tools() # 9 tool definitions, pass to any LLMREST API
curl -X POST http://localhost:9100/v1/scrape \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","formats":["markdown"]}'Features
Feature | Status |
Scrape (URL→Markdown/HTML/text) | ✅ |
Anti-bot bypass (Cloudflare, DataDome) | ✅ |
Auto-escalation engine chain | ✅ |
Crawl (BFS + sitemap) | ✅ |
Search (DuckDuckGo/SearXNG) | ✅ |
Map (fast URL discovery) | ✅ |
Batch scrape (parallel) | ✅ |
LLM extraction (structured JSON) | ✅ |
Change tracking (diff) | ✅ |
PDF parsing | ✅ |
CSS extraction | ✅ |
Screenshots | ✅ |
JS actions (click/scroll/type) | ✅ |
Product/Menu/Branding extractors | ✅ |
Summary (LLM) | ✅ |
Proxy rotation | ✅ |
MCP server | ✅ |
OpenAI tool definitions | ✅ |
Docker deployment | ✅ |
CLI tool | ✅ |
Architecture
Engine Chain (auto-escalation):
httpx (fast) → flaresolverr (Cloudflare) → patchright (DataDome/Akamai)License
MIT
This server cannot be deployed
Maintenance
ActivityStale
ResponsivenessNo issues
Related MCP Connectors
Zenrows MCP server — Fetch, Extract, Batch, and Browser Sessions for AI coding assistants
Free web search for AI agents. No API key required. Hosted MCP in active development.
One MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked.…
All HasData scraping tools in one MCP server: Google, TikTok, Instagram, maps, e-commerce and more.
Related MCP Servers
scraperapi-mcp-serverofficial
AlicenseAqualityCmaintenanceA local MCP server that lets AI agents bypass bot detection, geo-restrictions, and JavaScript rendering challenges when scraping the web, backed by ScraperAPI's services285MIT- -licenseNot gradedqualityCmaintenanceSelf-hosted MCP server that provides web scraping and crawling tools, integrating seamlessly with AI frameworks like OpenAI Agents SDK, Cursor, and Claude Code.4-
- AlicenseNot gradedqualityDmaintenanceAn MCP server that gives AI tools a reliable way to fetch content from the web, handling anti-bot protection and JavaScript-rendered pages.1,268 npmMIT