F
licenseB
qualityD
maintenanceAn MCP Server for Web scraping and Crawling, built using Crawl4AI
Last updated
2
24
freecrawl
by RistrixOP
Provides anti-bot bypass for Akamai-protected websites using patchright, enabling scraping of Akamai-challenged pages.
Provides anti-bot bypass for Cloudflare-protected websites using flaresolverr, enabling scraping of Cloudflare-challenged pages.
Integrates DuckDuckGo search to allow the server to perform web searches as part of the Search feature.
Provides OpenAI-compatible tool definitions for use with LLMs, allowing AI models to invoke scraping, crawling, search, and other actions via function calling. Also supports LLM-based extraction and summarization.
Integrates SearXNG as an alternative search engine for the Search feature, providing flexible search capabilities.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@freecrawlscrape https://example.com to markdown"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
🔥 Freecrawl
Open-source web scraping API with anti-bot bypass. Drop-in Firecrawl alternative. AI-first: MCP server, OpenAI tool definitions, REST API — any AI model can use it. No API keys, no credits, no limits.
Quick Start
pip install -r requirements.txt
patchright install chromium
python -m freecrawl serve # REST API on :9100Related MCP server: ISIS MCP
AI Integration
MCP Server (Claude, Cursor, any MCP client)
{"mcpServers":{"freecrawl":{"command":"python","args":["-m","freecrawl.mcp_server"]}}}OpenAI Tool Definitions
from freecrawl.ai_tools import get_tools
tools = get_tools() # 9 tool definitions, pass to any LLMREST API
curl -X POST http://localhost:9100/v1/scrape \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","formats":["markdown"]}'Features
Feature | Status |
Scrape (URL→Markdown/HTML/text) | ✅ |
Anti-bot bypass (Cloudflare, DataDome) | ✅ |
Auto-escalation engine chain | ✅ |
Crawl (BFS + sitemap) | ✅ |
Search (DuckDuckGo/SearXNG) | ✅ |
Map (fast URL discovery) | ✅ |
Batch scrape (parallel) | ✅ |
LLM extraction (structured JSON) | ✅ |
Change tracking (diff) | ✅ |
PDF parsing | ✅ |
CSS extraction | ✅ |
Screenshots | ✅ |
JS actions (click/scroll/type) | ✅ |
Product/Menu/Branding extractors | ✅ |
Summary (LLM) | ✅ |
Proxy rotation | ✅ |
MCP server | ✅ |
OpenAI tool definitions | ✅ |
Docker deployment | ✅ |
CLI tool | ✅ |
Architecture
Engine Chain (auto-escalation):
httpx (fast) → flaresolverr (Cloudflare) → patchright (DataDome/Akamai)License
MIT
This server cannot be installed
Maintenance
–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Servers
- AlicenseBqualityDmaintenanceA local web scraping MCP server with RAG capabilities that provides intelligent web search, content extraction, and screenshot tools without requiring API keys.Last updated428MIT
scraperapi-mcp-serverofficial
AlicenseAqualityBmaintenanceA local MCP server that lets AI agents bypass bot detection, geo-restrictions, and JavaScript rendering challenges when scraping the web, backed by ScraperAPI's servicesLast updated285MIT- -license-qualityCmaintenanceSelf-hosted MCP server that provides web scraping and crawling tools, integrating seamlessly with AI frameworks like OpenAI Agents SDK, Cursor, and Claude Code.Last updated4
Related MCP Connectors
Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili…
SEO MCP server: crawl your site, find AI-visibility gaps, and ship the fix from your coding agent.
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/RistrixOP/freecrawl'
If you have feedback or need assistance with the MCP directory API, please join our Discord server