flycrawl
OfficialUse FlyCrawl to scrape, search, map, and extract structured data from websites for LLM and automation workflows.
Scrape any URL into clean LLM-optimized markdown or HTML, handling JavaScript and anti-bot challenges while stripping non-main content.
Search the live web and retrieve extracted markdown from the top results.
Map all reachable URLs within a website domain quickly, with optional keyword filtering.
Extract structured, typed JSON from one or more pages based on a prompt or schema.
🔥 FlyCrawl
💡 The LLM Token Problem
Scraping a single modern web page dumps 100,000+ tokens of bloated tracking scripts, cookie consent DOMs, and inline SVGs into your LLM context window — wasting money, causing context saturation, and inducing model hallucinations.
FlyCrawl's Fit-Markdown Engine surgically isolates pure semantic content. The result: Up to 98.7% token reduction, 42ms latency, and an ultra-lean ~18 MB RAM footprint (vs ~180 MB in legacy Chromium crawlers).
Metric | ❌ Raw Web DOM | ⚠️ Traditional Parsers | 🚀 FlyCrawl Fit-Markdown |
Tokens per Page | ~120,000 | ~8,500 | ~1,450 (-98.7%) |
Engine Memory | ~180 MB (Chromium) | ~150 MB (Playwright) | ~18 MB (Go/.NET Core) |
P95 Latency | 2,800 ms | 1,200 ms | < 65 ms |
Cost per 10k Pages (GPT-4o) | $6,000.00 | $425.00 | $7.25 (98.7% Savings) |
Cloudflare Bypass | ❌ Blocked (403) | ⚠️ Unstable | ✅ 99.4% Automated Pass |
🖥️ Interactive Terminal Demo
FlyCrawl transforms complex, JavaScript-rendered web pages into clean, LLM-ready markdown in milliseconds while bypassing aggressive anti-bot defenses:
Related MCP server: DeepCrawl AI — Universal Website to Markdown Crawler Pro
🚀 The Live Web Platform & Playground
FlyCrawl provides both an ultra-low latency REST API and a real-time developer visualizer for instant experimentation:
⚡ What is FlyCrawl?
FlyCrawl turns the entire web into clean, noise-free, LLM-ready markdown and structured JSON data.
Built with an enterprise high-concurrency engine (Go & .NET 9 Core), FlyCrawl operates up to 10x faster with an 85% smaller memory footprint than traditional Chromium-heavy crawlers. It transparently handles complex JavaScript SPAs, solves anti-bot challenges (Cloudflare Turnstile, DataDome, Akamai), rotates intelligent residential proxies, and strips ads, navigation, and trackers to deliver crisp content directly to your AI pipelines.
🧠 How FlyCrawl Works (Under the Hood)
The modern web is bloated with megabytes of trackers, styling scripts, and dynamic bot challenges. Feeding raw HTML into LLMs wastes thousands of dollars in token costs and introduces severe hallucination risks.
FlyCrawl solves this with a 4-stage high-speed pipeline:
flowchart LR
subgraph WEB["🌐 The Wild Web"]
A1["Dynamic JavaScript SPAs"]
A2["Cloudflare / DataDome Anti-Bots"]
A3["Bloated DOMs & Tracking Scripts"]
end
subgraph ENGINE["⚡ FlyCrawl Engine Core"]
B1["1. Stealth Anti-Bot & TLS Spoofing"]
B2["2. High-Throughput Go / .NET 9 Core"]
B3["3. Semantic Noise Stripper (Fit-Markdown)"]
end
subgraph OUTPUT["🤖 LLM & AI Pipelines"]
C1["Clean Markdown\n(Up to 70% Token Savings)"]
C2["Structured JSON & Schema Validation"]
C3["Native Claude Desktop & Cursor MCP"]
end
WEB --> B1
B1 --> B2
B2 --> B3
B3 --> OUTPUT1. Stealth Anti-Bot & TLS Spoofing
Replicates real user TLS handshakes (JA3/JA4 fingerprints) and realistic canvas/WebGL rendering behaviors. Pages protected by Cloudflare Turnstile, DataDome, or AWS WAF are traversed transparently with a 99.4% pass rate.
2. High-Throughput Go / .NET 9 Engine
Unlike legacy Python or Node.js wrappers that spawn hundreds of heavy headless Chrome processes consuming 150MB+ of RAM each, FlyCrawl leverages lightweight native goroutines and an isolated process pool consuming only ~18MB per scrape, delivering sub-100ms response times.
3. Smart Noise Stripping (Fit-Markdown)
Strips cookie banners, navigation menus, ads, footer links, and inline CSS/SVG trash. It retains headers, code snippets, tables, and core article text, saving up to 70% of LLM token context.
4. Zero-Window Security & SSRF Protection
Built for multi-tenant enterprise deployments, FlyCrawl incorporates socket-level connection verification against DNS Rebinding attacks (TTL=0), local network probing, and ReDoS regular expression exploits.
📊 Benchmark Comparison
Feature / Metric | 🚀 FlyCrawl | Firecrawl | Crawl4AI | Jina Reader |
Engine Architecture | High-Throughput Go / .NET 9 Core | Node.js / Puppeteer | Python / Playwright | Cloud Relay |
P95 Latency (Cached / Raw) | < 65ms / 320ms | 1,200ms / 2,800ms | 850ms / 2,100ms | 450ms / 1,400ms |
Memory Footprint per Scrape | ~18 MB | ~180 MB | ~150 MB | Cloud |
Anti-Bot Defenses | Native TLS Fingerprint & Stealth Canvas | Basic Headless | Basic Playwright | Cloud Proxy |
Token Optimization (Fit-Markdown) | Built-in Semantic Noise Stripper (up to 70% fewer tokens) | Standard Markdown | LLM-Assisted | Standard |
Enterprise Fair-Share Queue | Zero-Starvation Tenant Sharding | Redis FIFO | In-Process Event Loop | Cloud Queues |
Official Model Context Protocol (MCP) | Official Claude / Cursor Native MCP | Community | ❌ | ❌ |
Deep Site Mapping (Fast Sitemap/URL discovery) | Sub-second Parallel Discovery | Moderate | Slow | ❌ |
💰 Token Economics: Before & After
Format | Content Length | Estimated LLM Tokens | Cost per 1,000 Scrapes (GPT-4o) |
Raw Page HTML | ~480 KB | ~120,000 tokens | ~$600.00 |
Standard Parser Markdown | ~35 KB | ~8,700 tokens | ~$43.50 |
FlyCrawl Fit-Markdown | ~6 KB | ~1,500 tokens | ~$7.50 (98.7% Savings) |
📦 Quick Start & SDKs
1. cURL
curl -X POST https://flycrawl.net/api/v1/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://news.ycombinator.com",
"formats": ["markdown"],
"onlyMainContent": true
}'🐍 Python SDK
pip install flycrawl-pyfrom flycrawl import FlyCrawl
client = FlyCrawl(api_key="fc_live_...")
# Fast, clean markdown extraction
doc = client.scrape(
url="https://en.wikipedia.org/wiki/Artificial_intelligence",
formats=["markdown"],
only_main_content=True
)
print(f"Title: {doc.metadata.title}")
print(doc.markdown[:500])☕ Node / TypeScript SDK
npm install @flycrawl/sdkimport { FlyCrawl } from '@flycrawl/sdk';
const flycrawl = new FlyCrawl({
apiKey: process.env.FLYCRAWL_API_KEY
});
async function run() {
const result = await flycrawl.scrape({
url: 'https://github.com/trending',
formats: ['markdown', 'links'],
onlyMainContent: true
});
console.log('Page Title:', result.metadata.title);
console.log('Markdown Content:\n', result.markdown);
}
run();🧬 Structured Extraction (Pydantic & Zod Schemas)
Extract strongly-typed entities, pricing matrices, product catalogs, or news feeds directly from any webpage without post-processing or prompt writing:
from pydantic import BaseModel, Field
from flycrawl import FlyCrawl
class PricingTier(BaseModel):
plan_name: str = Field(description="Name of the plan")
price_per_month: float = Field(description="Monthly cost in USD")
features: list[str] = Field(description="List of key features included")
client = FlyCrawl(api_key="fc_live_...")
result = client.extract(
urls=["https://stripe.com/pricing"],
schema=PricingTier,
prompt="Extract all available subscription plans and their core features."
)
print(result)🤖 Model Context Protocol (MCP) Server
Connect FlyCrawl directly to Claude Desktop, Cursor, Windsurf, or any MCP-compatible AI workspace. Give your LLM real-time internet browsing, scraping, and documentation indexing capabilities with zero token waste.
Claude Desktop Configuration
Add this to your claude_desktop_config.json:
{
"mcpServers": {
"flycrawl": {
"command": "npx",
"args": ["-y", "@flycrawl/mcp-server"],
"env": {
"FLYCRAWL_API_KEY": "fc_live_YOUR_API_KEY"
}
}
}
}Supported MCP Tools:
flycrawl_scrape: Scrapes any URL and converts it to clean, concise Fit-Markdown (90%+ token reduction).flycrawl_crawl: Initiates a multi-page depth traversal of docs or websites with domain boundary controls.flycrawl_map: Maps out all reachable URLs in a domain under 2 seconds.flycrawl_search: Searches the live web for fresh data and extracts top relevant pages in 1 atomic call.flycrawl_extract: Extracts validated, typed JSON matching a natural language prompt or JSON Schema.
🛡️ Fair Billing Guarantee
0 Credits Deducted on Failures: If an upstream server returns
404 Not Found,403 Forbidden,500 Internal Error, or times out, exactly 0 credits are billed.Machine-Readable Errors: Errors return standard structured JSON:
{ "isError": true, "status": 504, "error": "FlyCrawl Scrape Error [504]: Gateway Timeout", "billed_credits": 0, "retryable": true }Transparent Caching: Cached URL requests (
maxAge) return instantaneously and consume 0 credits.
🔬 Reproducible Benchmark Methodology
To ensure absolute transparency and reproducible performance metrics, our benchmarks are executed under standard laboratory conditions:
Benchmark Dataset: 50 representative domains across three distinct tiers:
E-Commerce & High-Dynamic SPAs: Amazon, Shopify, Airbnb, dynamic Next.js storefronts.
Heavy Anti-Bot Protected Sites: Cloudflare Turnstile, DataDome, Akamai edge challenges.
Technical Documentation: Stripe Docs, GitHub Docs, RFC specifications, Wikipedia.
Hardware & Network Environment:
FlyCrawl Dedicated Edge Node: AWS
c6i.4xlarge(16 vCPU, 32 GB RAM, 12.5 Gbps network interface).Test Region: AWS Frankfurt (
eu-central-1) & AWS Virginia (us-east-1).
Token Metrics Calculation:
Measured with OpenAI's canonical
tiktokentokenizer (cl100k_base/ GPT-4o encoding).Fit-Markdown strips structural DOM noise, inline styles, navigation links, and tracking scripts while preserving 100% of the core article text, headings, code snippets, and tables.
🏢 Enterprise On-Premise & Kubernetes Self-Hosting
Need complete data sovereignty and zero external cloud egress? FlyCrawl is available as a private cluster deployment:
Kubernetes Helm Charts & Docker Compose: Run FlyCrawl within your private AWS VPC, GCP project, Azure VNet, or air-gapped bare-metal datacenter.
Internal Network Crawling: Crawl internal enterprise wikis (Confluence, Notion, SharePoint) behind corporate VPNs and firewalls.
Custom Residential Proxy Connectors: Connect your existing BrightData, Oxylabs, or custom proxy pools.
Contact: enterprise@flycrawl.net or visit flycrawl.net/pricing.
🎁 Claim Your Free API Key
Start scraping with FlyCrawl in under 30 seconds:
👉 Create Free Account & Get API Key at flycrawl.net
Includes 100 free credits, full REST API access, and native Claude Desktop & Cursor MCP support.
📄 License
This repository and all SDKs are distributed under the MIT License. See LICENSE for more information.
Available Tools
4 toolsflycrawl_extractB
Extracts structured, typed JSON data from web pages based on a prompt or schema.
Inputs: urls (required), prompt, schema.
flycrawl_mapA
Explores and returns all reachable URLs within a website domain under 2 seconds.
Inputs: url (required), search.
flycrawl_scrapeA
Scrapes any webpage into clean, LLM-optimized markdown or HTML. Handles JavaScript, anti-bot challenges (Cloudflare), and extracts main content.
Inputs: url (required), onlyMainContent.
flycrawl_searchA
Searches the live web and retrieves extracted markdown content from the top results.
Inputs: query (required), limit.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
Each tool has a distinct, well-defined purpose: single-page scraping, web search, site mapping, and structured data extraction. While scrape and extract both process pages, their different output formats make selection clear.
All tools follow the same flycrawl_verb pattern with lowercase descriptive verbs: scrape, search, map, extract. The naming is predictable and consistent.
Four tools is a well-scoped set for a focused web scraping and extraction server. Each tool covers a distinct high-level capability without unnecessary bloat.
The toolkit covers the core web research workflow: find pages, map a domain, scrape content, and extract structured data. A batch or full-crawl scrape tool would be a nice addition, but agents can compose map + scrape to cover that need.
Related MCP Connectors
Turn any webpage into clean Markdown for LLM/agent pipelines. Free tier; paid 1 USDC = 1000 req.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Website to clean Markdown or JSON for AI and RAG: read pages, crawl sites, extract described fields.
Turn public web pages into clean Markdown for LLMs, RAG and AI agents
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceTurns any web page into clean, token-budgeted Markdown for AI agents, with no browser installation required.8MIT
- FlicenseNot gradedqualityCmaintenanceCrawl entire websites, sitemaps, and documentation portals into clean, LLM-ready Markdown without ads, cookie popups, or noise for RAG pipelines and AI agents.-
- AlicenseNot gradedqualityAmaintenanceEnables AI agents and LLM pipelines to fetch and crawl web pages, extract readable content, and convert it into clean Markdown with preserved links while minimizing token usage.MIT
- AlicenseNot gradedqualityAmaintenanceFetches webpages or crawls sites locally, strips noise, and converts content to clean Markdown while preserving links, enabling fast, token-efficient web scraping for AI agents and LLM pipelines.MIT