💡 The LLM Token Problem
Scraping a single modern web page dumps 100,000+ tokens of bloated tracking scripts, cookie consent DOMs, and inline SVGs into your LLM context window — wasting money, causing context saturation, and inducing model hallucinations.
FlyCrawl's Fit-Markdown Engine surgically isolates pure semantic content. The result: Up to 98.7% token reduction, 42ms latency, and an ultra-lean ~18 MB RAM footprint (vs ~180 MB in legacy Chromium crawlers).
Metric | ❌ Raw Web DOM | ⚠️ Traditional Parsers | 🚀 FlyCrawl Fit-Markdown |
Tokens per Page | ~120,000 | ~8,500 | ~1,450 (-98.7%) |
Engine Memory | ~180 MB (Chromium) | ~150 MB (Playwright) | ~18 MB (Go/.NET Core) |
P95 Latency | 2,800 ms | 1,200 ms | < 65 ms |
Cost per 10k Pages (GPT-4o) | $6,000.00 | $425.00 | $7.25 (98.7% Savings) |
Cloudflare Bypass | ❌ Blocked (403) | ⚠️ Unstable | ✅ 99.4% Automated Pass |
FlyCrawl transforms complex, JavaScript-rendered web pages into clean, LLM-ready markdown in milliseconds while bypassing aggressive anti-bot defenses:
Related MCP server: markd-mcp
FlyCrawl provides both an ultra-low latency REST API and a real-time developer visualizer for instant experimentation:
FlyCrawl turns the entire web into clean, noise-free, LLM-ready markdown and structured JSON data.
Built with an enterprise high-concurrency engine (Go & .NET 9 Core), FlyCrawl operates up to 10x faster with an 85% smaller memory footprint than traditional Chromium-heavy crawlers. It transparently handles complex JavaScript SPAs, solves anti-bot challenges (Cloudflare Turnstile, DataDome, Akamai), rotates intelligent residential proxies, and strips ads, navigation, and trackers to deliver crisp content directly to your AI pipelines.
The modern web is bloated with megabytes of trackers, styling scripts, and dynamic bot challenges. Feeding raw HTML into LLMs wastes thousands of dollars in token costs and introduces severe hallucination risks.
FlyCrawl solves this with a 4-stage high-speed pipeline:
flowchart LR
subgraph WEB["🌐 The Wild Web"]
A1["Dynamic JavaScript SPAs"]
A2["Cloudflare / DataDome Anti-Bots"]
A3["Bloated DOMs & Tracking Scripts"]
end
subgraph ENGINE["⚡ FlyCrawl Engine Core"]
B1["1. Stealth Anti-Bot & TLS Spoofing"]
B2["2. High-Throughput Go / .NET 9 Core"]
B3["3. Semantic Noise Stripper (Fit-Markdown)"]
end
subgraph OUTPUT["🤖 LLM & AI Pipelines"]
C1["Clean Markdown\n(Up to 70% Token Savings)"]
C2["Structured JSON & Schema Validation"]
C3["Native Claude Desktop & Cursor MCP"]
end
WEB --> B1
B1 --> B2
B2 --> B3
B3 --> OUTPUTReplicates real user TLS handshakes (JA3/JA4 fingerprints) and realistic canvas/WebGL rendering behaviors. Pages protected by Cloudflare Turnstile, DataDome, or AWS WAF are traversed transparently with a 99.4% pass rate.
Unlike legacy Python or Node.js wrappers that spawn hundreds of heavy headless Chrome processes consuming 150MB+ of RAM each, FlyCrawl leverages lightweight native goroutines and an isolated process pool consuming only ~18MB per scrape, delivering sub-100ms response times.
Strips cookie banners, navigation menus, ads, footer links, and inline CSS/SVG trash. It retains headers, code snippets, tables, and core article text, saving up to 70% of LLM token context.
Built for multi-tenant enterprise deployments, FlyCrawl incorporates socket-level connection verification against DNS Rebinding attacks (TTL=0), local network probing, and ReDoS regular expression exploits.
Feature / Metric | 🚀 FlyCrawl | Firecrawl | Crawl4AI | Jina Reader |
Engine Architecture | High-Throughput Go / .NET 9 Core | Node.js / Puppeteer | Python / Playwright | Cloud Relay |
P95 Latency (Cached / Raw) | < 65ms / 320ms | 1,200ms / 2,800ms | 850ms / 2,100ms | 450ms / 1,400ms |
Memory Footprint per Scrape | ~18 MB | ~180 MB | ~150 MB | Cloud |
Anti-Bot Defenses | Native TLS Fingerprint & Stealth Canvas | Basic Headless | Basic Playwright | Cloud Proxy |
Token Optimization (Fit-Markdown) | Built-in Semantic Noise Stripper (up to 70% fewer tokens) | Standard Markdown | LLM-Assisted | Standard |
Enterprise Fair-Share Queue | Zero-Starvation Tenant Sharding | Redis FIFO | In-Process Event Loop | Cloud Queues |
Official Model Context Protocol (MCP) | Official Claude / Cursor Native MCP | Community | ❌ | ❌ |
Deep Site Mapping (Fast Sitemap/URL discovery) | Sub-second Parallel Discovery | Moderate | Slow | ❌ |
Format | Content Length | Estimated LLM Tokens | Cost per 1,000 Scrapes (GPT-4o) |
Raw Page HTML | ~480 KB | ~120,000 tokens | ~$600.00 |
Standard Parser Markdown | ~35 KB | ~8,700 tokens | ~$43.50 |
FlyCrawl Fit-Markdown | ~6 KB | ~1,500 tokens | ~$7.50 (98.7% Savings) |
curl -X POST https://flycrawl.net/api/v1/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://news.ycombinator.com",
"formats": ["markdown"],
"onlyMainContent": true
}'pip install flycrawl-pyfrom flycrawl import FlyCrawl
client = FlyCrawl(api_key="fc_live_...")
# Fast, clean markdown extraction
doc = client.scrape(
url="https://en.wikipedia.org/wiki/Artificial_intelligence",
formats=["markdown"],
only_main_content=True
)
print(f"Title: {doc.metadata.title}")
print(doc.markdown[:500])npm install @flycrawl/sdkimport { FlyCrawl } from '@flycrawl/sdk';
const flycrawl = new FlyCrawl({
apiKey: process.env.FLYCRAWL_API_KEY
});
async function run() {
const result = await flycrawl.scrape({
url: 'https://github.com/trending',
formats: ['markdown', 'links'],
onlyMainContent: true
});
console.log('Page Title:', result.metadata.title);
console.log('Markdown Content:\n', result.markdown);
}
run();Extract strongly-typed entities, pricing matrices, product catalogs, or news feeds directly from any webpage without post-processing or prompt writing:
from pydantic import BaseModel, Field
from flycrawl import FlyCrawl
class PricingTier(BaseModel):
plan_name: str = Field(description="Name of the plan")
price_per_month: float = Field(description="Monthly cost in USD")
features: list[str] = Field(description="List of key features included")
client = FlyCrawl(api_key="fc_live_...")
result = client.extract(
urls=["https://stripe.com/pricing"],
schema=PricingTier,
prompt="Extract all available subscription plans and their core features."
)
print(result)Connect FlyCrawl directly to Claude Desktop, Cursor, Windsurf, or any MCP-compatible AI workspace. Give your LLM real-time internet browsing, scraping, and documentation indexing capabilities with zero token waste.
Add this to your claude_desktop_config.json:
{
"mcpServers": {
"flycrawl": {
"command": "npx",
"args": ["-y", "@flycrawl/mcp-server"],
"env": {
"FLYCRAWL_API_KEY": "fc_live_YOUR_API_KEY"
}
}
}
}flycrawl_scrape: Scrapes any URL and converts it to clean, concise markdown.
flycrawl_crawl: Initiates a multi-page crawl of docs or articles.
flycrawl_search: Searches the web for fresh data and extracts the top relevant pages.
flycrawl_map: Maps out all reachable URLs in a domain under 2 seconds.
Start scraping with FlyCrawl in under 30 seconds:
👉 Create Free Account & Get API Key at flycrawl.net
Includes 100 free credits, full REST API access, and native Claude Desktop & Cursor MCP support.
This repository and all SDKs are distributed under the MIT License. See LICENSE for more information.
Extracts structured, typed JSON data from web pages based on a prompt or schema.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | List of URLs to extract information from. | |
| prompt | No | Description of what to extract. | |
| schema | No | Optional JSON Schema definition for the extraction. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description carries the full burden of behavioral disclosure. It briefly notes that output is 'typed JSON data' but does not mention whether the tool renders pages, handles dynamic content, requires authentication, or behaves in any other way beyond extraction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single focused sentence with no redundant or speculative content. It uses front-loaded phrasing and every word serves a purpose, making it easy for an agent to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a three-parameter tool with 100% schema coverage, the description is adequate for basic invocation. However, it lacks usage context, behavioral details, and a clear distinction from sibling tools, leaving some gaps in the full picture an agent would need for confident selection.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides descriptions for all three parameters, so the baseline is 3. The description adds a small clarifying point that extraction is 'based on a prompt or schema,' which helps understand the relationship between the two optional parameters, but no additional semantics are provided.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Extracts') and clearly identifies the resource ('web pages') and output ('structured, typed JSON data'). It conveys the tool's core purpose effectively, though it does not explicitly mention or differentiate from sibling tools like flycrawl_scrape.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use flycrawl_extract versus flycrawl_scrape, flycrawl_search, or flycrawl_map. The description only states what the tool does, leaving the agent to infer selection criteria from the tool name and sibling list.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Explores and returns all reachable URLs within a website domain under 2 seconds.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The target website domain to map. | |
| search | No | Optional keyword to filter discovered URLs. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that the tool crawls and returns URLs, but it does not explain output format, crawl limits, what 'reachable' means, failure modes, or rate-limit behavior. The 'under 2 seconds' qualifier is a performance claim rather than meaningful behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single focused sentence that states the core purpose up front. No redundant clauses or unnecessary details are included, although the time qualifier adds limited value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the basic purpose and return concept, and the schema covers both parameters. However, with no output schema and no annotations, an agent still lacks details about the response structure, potential limitations, and exact crawling behavior beyond 'all reachable URLs.'
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with both url and search described in the input schema. The tool description adds no extra parameter nuance beyond the schema, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb and resource: it explores and returns all reachable URLs within a website domain. This clearly distinguishes the map behavior from the sibling scrape/search/extract tools, even without sibling descriptions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use is implied: call this when you need all reachable URLs for a domain. However, it provides no explicit when-to-use guidance, no exclusions, and does not mention any alternatives among the sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scrapes any webpage into clean, LLM-optimized markdown or HTML. Handles JavaScript, anti-bot challenges (Cloudflare), and extracts main content.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The complete URL of the webpage to scrape. | |
| onlyMainContent | No | Whether to strip navigation, headers, footers, and ads (default: true). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of behavioral disclosure. It usefully reveals that the tool executes JavaScript, handles anti-bot challenges such as Cloudflare, and extracts main content, which are meaningful behavioral traits beyond a generic 'scrape' label. It does not cover failure modes or edge cases, but the key execution behaviors are disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single compact sentence that front-loads the core action and output format, then adds key capabilities without wasted words. Every clause contributes useful information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has only two parameters, both fully documented in the schema, and the description states the output format (markdown or HTML) and processing behavior. Since there is no output schema, explicitly stating that the result is clean markdown or HTML provides enough context for an agent to invoke it correctly. Slight gap: no mention of response structure or error conditions, but this is minor for such a simple tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already covers both parameters with 100% description coverage, so the baseline applies. The description's mention of 'extracts main content' loosely aligns with onlyMainContent, but it does not add meaning beyond the schema's own description of stripping navigation, headers, footers, and ads.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Scrapes') with a clear resource ('any webpage') and a concrete output ('clean, LLM-optimized markdown or HTML'). It also mentions 'extracts main content,' which further clarifies the tool's function and distinguishes it from the sibling search/map/extract tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no explicit guidance about when to use this tool versus flycrawl_search, flycrawl_map, or flycrawl_extract. The description implies it is for fetching page content, but it does not state when not to use it or which sibling would be more appropriate for other tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Searches the live web and retrieves extracted markdown content from the top results.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of search results to return (default: 5). | |
| query | Yes | The search query to execute on the web. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description carries the full behavioral disclosure burden. It does add value by stating that output is extracted markdown content and that only top results are returned. It does not disclose limits, rate behavior, error cases, or whether the returned content is truncated, but the core behavior is transparent enough for a simple search tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that leads with the primary action and packs the key output characteristic into a compact statement. Every word earns its place, and there is no redundant filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a low-complexity tool with two well-described parameters, the description covers the purpose and general output type. However, there is no output schema and the description does not specify the exact result structure, such as whether each result includes a title or URL alongside the markdown, or how results are ordered. This leaves some ambiguity for an agent invoking the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with clear descriptions for both 'query' and 'limit'. The tool description adds general output context but no parameter-specific detail beyond what the schema already provides, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Searches') and identifies a clear resource ('the live web'), with a concrete outcome ('retrieved extracted markdown content from the top results'). It is distinguishable from siblings like flycrawl_scrape because it centers on search rather than a specific page, though it does not explicitly name those alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool: when you need live web search results with markdown content. However, it provides no explicit guidance on when not to use it or which sibling tool to prefer, leaving the agent to infer the boundary between search and the other flycrawl tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0flycrawl_extractflycrawl_mapflycrawl_scrapeflycrawl_searchScored across 4 tools
Each tool has a distinct, well-defined purpose: single-page scraping, web search, site mapping, and structured data extraction. While scrape and extract both process pages, their different output formats make selection clear.
All tools follow the same flycrawl_verb pattern with lowercase descriptive verbs: scrape, search, map, extract. The naming is predictable and consistent.
Four tools is a well-scoped set for a focused web scraping and extraction server. Each tool covers a distinct high-level capability without unnecessary bloat.
The toolkit covers the core web research workflow: find pages, map a domain, scrape content, and extract structured data. A batch or full-crawl scrape tool would be a nice addition, but agents can compose map + scrape to cover that need.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Clean Markdown and AI-readability scoring for any URL. Built for AI agents.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Any web page -> clean markdown. Firecrawl-lite, 0.001 USDC/extract (x402, Base), free demo.