LookaCrawler
README.md
<div align="center">
# 🕷️ LookaCrawler
**Free, open-source, token-efficient local alternative to Firecrawl with native Model Context Protocol (MCP) Server for LLMs.**
[](LICENSE)
[](https://bun.sh)
[](https://www.typescriptlang.org)
[](https://modelcontextprotocol.io)
[](https://github.com/lucasmartins-ai/lookacrawler)
[](SECURITY.md)
</div>
---
## 📌 Why LookaCrawler?
Web crawling for Large Language Models (LLMs) is broken by default: modern web pages contain massive HTML bloat (scripts, tracking pixels, nested divs, navigation headers, stylesheets), costing thousands of wasted tokens per page.
**LookaCrawler** is an open-source, token-optimized local crawler that cuts a median 90% of tokens on real pages (21–98% across 8 sites, see `benchmark-live.ts`), extracts clean Markdown, bypasses anti-bot barriers with stealth Playwright drivers, and exposes a native **Model Context Protocol (MCP) Server** ready for **Claude Desktop, Cursor, and Antigravity**.
---
## 🥊 Comparison: LookaCrawler vs Alternatives
| Feature | 🕷️ LookaCrawler | 🔥 Firecrawl (Cloud) | ⚡ Jina Reader |
| :--- | :---: | :---: | :---: |
| **Pricing / Cost** | **$0.00 (100% Free Open Source)** | $16 to $99+/month | Rate-limited API |
| **Token Reduction** | **Median 90% fewer tokens (real pages) + Footnotes** | Standard Markdown | Basic Markdown |
| **Autonomous Crawling** | **Native `map` & `crawl` (BFS + URL patterns)** | Cloud Crawler | Single-page only |
| **Pre-Crawl Actions** | **Native Playwright (click, scroll, fill)** | Paid Addon | None |
| **Link Formatting** | **Inline, References Footnotes, Strip** | Inline only | Inline only |
| **Data Privacy** | **100% Local (Zero Telemetry)** | Cloud Provider | Cloud API |
| **MCP Integration** | **Native Tools + Resources + Prompts** | Community Wrapper | None |
| **Stealth & Anti-Bot** | **Real Chrome + Stealth Fingerprint** | Cloud Proxies | Basic Headers |
| **Local SQLite Cache** | **Built-in (24h TTL cache)** | Redis / Paid Addon | None |
| **JS SPA Support** | **Playwright + Chrome Pool** | Cloud Headless | Headless |
---
## 🚀 Key Features
- **Token Economy First:** Automatically prunes scripts, styles, inline SVGs, tracking tags, navigations, footers, redundant forms, and boilerplate containers with high link density (>80%).
- **Advanced Link & Image Formatting:**
- `link_format`: Choose between `inline` (standard markdown), `references` (footnote citations `[1]`, saving ~25% tokens on repetitive URLs), or `strip` (pure text).
- `image_mode`: Choose between `ignore` (zero tokens), `alt_only` (preserves semantic context without URL bloat), or `markdown` (full ``).
- **Autonomous Mapping & Recursive Crawling:**
- `map_website`: Inspects `/robots.txt`, sitemaps, and root anchors to discover all pages in a domain.
- `crawl_website`: Breadth-first autonomous crawling with max depth, max pages, URL pattern filters (substring, `*` wildcard), and real-time token accounting.
- **Pre-Crawl Browser Actions:** Automate clicks, scrolls, typing, and waits in Playwright before extracting content (dismiss cookie banners, scroll for infinite loading, expand accordions).
- **Dual Hybrid Crawling Engine:**
- `fast`: Ultra-fast native HTTP GET with backoff. Auto-escalates to `deep` if an anti-bot challenge is encountered.
- `deep`: Headless Playwright engine launching real Google Chrome with stealth patches (`navigator.webdriver` cleared, WebGL spoofed, CDP leaks stripped) to transparently crawl Cloudflare/Turnstile-protected pages.
- **Native MCP Ecosystem:**
- **Tools:** `extract_web_content`, `crawl_website`, `map_website`, `batch_extract_web_content`, `extract_structured_data`, `extract_contacts`, `search_places`, `discover_local_leads`, `deep_enrich_lead`.
- **Resources:** Live telemetry at `crawler://metrics` and cache analytics at `crawler://cache/stats`.
- **Prompts:** Pre-engineered templates `crawl-and-summarize` and `compare-pages`.
- **Entity Extraction:** `contacts` pulls emails, phone numbers, WhatsApp links and social profiles; `places` returns structured Google Maps business records with formula-injection-safe CSV output.
- **B2B Lead Discovery:** `maps` (Google Maps engine) plus `enrich`, which crawls a company's own site for verified emails, socials and decision makers.
- **Local SQLite Caching:** Stores extracted Markdown in `crawler_cache.sqlite` to eliminate duplicate network calls.
- **Structured JSON & Metadata Extraction:** Extracts Open Graph tags (`og:title`, `og:description`), publication dates, canonical URLs, and custom CSS selectors.
---
## 🔌 1-Click MCP Setup (Claude Desktop & Cursor)
Add LookaCrawler to your `claude_desktop_config.json` or Cursor MCP settings:
```json
{
"mcpServers": {
"lookacrawler": {
"command": "bun",
"args": ["run", "/absolute/path/to/lookacrawler/index.ts"]
}
}
}
```
Now you can prompt Claude or Cursor:
> *"Crawl https://example.com/docs and extract the API documentation using LookaCrawler."*
---
## 📦 Quick Start & CLI Usage
### 1. Installation
Requires **Bun 1.1+** (high-performance runtime with native SQLite):
```bash
# Clone the repository
git clone https://github.com/lucasmartins-ai/lookacrawler.git
cd lookacrawler
# Install dependencies
bun install
```
### 2. CLI Commands
```bash
# Single URL fast Markdown extraction
bun run cli.ts extract https://news.ycombinator.com --mode fast
# Single URL fast Markdown extraction with reference footnotes
bun run cli.ts extract https://example.com --link-format references --output page.md
# Headless Playwright deep extraction with CSS selector target
bun run cli.ts extract https://example.com --mode deep --selector "main" --json
# Discover all website URLs and sitemaps
bun run cli.ts map https://example.com --max-urls 500
# Recursively crawl documentation with URL pattern filtering and token accounting
bun run cli.ts crawl https://example.com/docs --max-depth 2 --max-pages 15 --include /docs/ --link-format references
# Batch concurrent multi-URL crawling
bun run cli.ts batch https://site1.com https://site2.com --concurrency 4
# Structured JSON schema extraction
bun run cli.ts structured https://example.com --schema '{"title":"h1","links":"a"}'
# Extract emails, phones, WhatsApp links and social profiles from a site
bun run cli.ts contacts https://example.com
# Search local businesses on Google Maps (browser-driven)
bun run cli.ts places "dentists in Bristol" --limit 5 --csv -o places.csv
# B2B lead discovery via the sibling google-maps-scraper engine
# (set MAPS_LEADS_BIN if the binary is not in a standard path)
bun run cli.ts maps "dental clinic" "Bristol" --limit 20
# Enrich a company website: emails, socials, decision makers, short summary
bun run cli.ts enrich https://example.com --title "Example Ltd" --mode fast
# Enrich many leads at once from a JSON array of {title, website}
bun run cli.ts enrich-batch leads.json --concurrency 3 -o enriched.json
# Start MCP Server via SSE on port 3000
bun run cli.ts serve --transport sse --port 3000
```
> **Behind a TLS-intercepting proxy?** Export `LOOKACRAWLER_INSECURE_TLS=true` to
> accept the proxy's forged certificate. Off by default — never enable it for
> untrusted networks.
> **Rotating proxies?** Export `LOOKACRAWLER_PROXIES` with a comma or
> whitespace separated list of URLs. Every request without an explicit
> `--proxy` pulls the next entry, so one blocked IP is not fatal for the run.
> **Local proxy (Tor, mitmproxy, a sidecar)?** A `proxy` passed per request
> must be a public host. List trusted local proxies in
> `LOOKACRAWLER_ALLOWED_PROXIES` (comma separated, exact URLs); entries in
> `LOOKACRAWLER_PROXIES` are trusted too.
#### SSE server security
| Variable | Effect |
| --- | --- |
| `LOOKACRAWLER_SSE_TOKEN` | Bearer token required on every request. If the server is bound to a non-loopback address (e.g. `--host 0.0.0.0`) and this is unset, a random token is generated and printed at startup. |
| `LOOKACRAWLER_SSE_ALLOWED_ORIGINS` | Comma-separated web origins allowed to call the server from a browser (CORS). Requests with any other `Origin` header are refused. CLI/desktop MCP clients are unaffected. |
| `LOOKACRAWLER_SSE_ALLOW_UNAUTHENTICATED` | `true` disables the token requirement on non-loopback binds. Only for isolated networks. |
| `LOOKACRAWLER_SSE_MAX_SESSIONS` | Maximum concurrent SSE clients (each gets its own MCP session). Default `16`; further connections get `503` until one disconnects. |
| `LOOKACRAWLER_MAX_BROWSERS` | Maximum concurrent Chromium processes (one per distinct proxy). Default `4`; idle ones are closed after `LOOKACRAWLER_BROWSER_IDLE_MS` (default 120000). |
Every outbound connection goes through a built-in loopback egress guard that
resolves each destination once, refuses private/link-local/loopback addresses
and connects to exactly the IP it checked. This covers redirects and requests a
page makes on its own (fetch, XHR, WebSocket) in `deep` mode, and prevents DNS
rebinding. With a proxy (`--proxy`, `LOOKACRAWLER_PROXIES`) the guard sits in
front of it: it validates the destination, then asks the proxy to connect to
that IP (HTTP `CONNECT`, SOCKS5 with optional user/password, or SOCKS4), so a
proxy — even a trusted local one — can never be steered at an internal host.
Side effect: DNS is resolved on this machine, not at the proxy, so geo-DNS/CDN
edges follow your location rather than the proxy's, and SOCKS4 cannot reach
IPv6-only sites. `LOOKACRAWLER_ALLOW_LOCAL=true` turns all of this off and talks
to the proxy directly (for local development only).
### 3. Google Maps engine (optional)
The `maps` command and the `discover_local_leads` MCP tool delegate to a sibling
Google Maps scraper binary. It is a separate Go project
([`google-maps-scraper`](https://github.com/coelhobugado/google-maps-scraper),
MIT) and is **not** vendored into this repository.
Place the compiled `maps-leads` binary in one of these locations, or point
`MAPS_LEADS_BIN` at it:
```
./google-maps-scraper/maps-leads # repo root
../google-maps-scraper/maps-leads # sibling directory
$MAPS_LEADS_BIN # anywhere
```
Without it, `maps` fails fast with a clear message; `extract`, `crawl`,
`contacts`, `places` and `enrich` are unaffected. Pass `--email` to let the
scraper extract public emails during discovery.
### 4. Docker Deployment
```bash
# Build and run Docker container
docker build -t lookacrawler .
docker run -p 3000:3000 -e LOOKACRAWLER_SSE_TOKEN="$(openssl rand -hex 24)" lookacrawler
```
The container listens on `0.0.0.0`, so a token is always required. Without
`LOOKACRAWLER_SSE_TOKEN` a random one is generated on each start and printed in
the container logs.
---
## 🧪 Architecture & Testing
```text
Incoming URL ──► [Local SQLite Cache Check] ──(Hit)──► Return Cached Markdown
│ (Miss)
▼
[Fast HTTP GET Request] ──(Blocked?)──► [Auto-Escalate to Deep Stealth]
│ │
▼ ▼
[HTML DOM Tree Parser] ◄──────────────────────────┘
│
▼
[Aggressive Token Noise Pruner]
(Strips SVG, Nav, Ads, Tracking, CSS, JS)
│
▼
[Mozilla Readability Engine]
│
▼
[Turndown Markdown Converter] ──► Return Clean LLM Markdown
```
Run test suite:
```bash
bun test
```
---
## ⭐ Star & Support
If LookaCrawler saves you API fees and token costs:
- ⭐ **Star this repository** to help other developers find it!
- 💡 **Open an Issue / PR** for new stealth bypasses or crawler features.
---
## Built by LookADev
[`lookacrawler`](https://github.com/lucasmartins-ai/lookacrawler) is built and maintained by [LookADev](https://lookadev.com), an engineering studio specializing in AI agents, web architecture, and token optimization.
**Start a project → [lookadev.com](https://lookadev.com)** · **Email: [lucas@lookadev.com](mailto:lucas@lookadev.com)**
## 📄 License
Open-source software licensed under the [MIT License](LICENSE).
TDQS
A3.6/5.0
Scored across 9 tools
Disambiguation3/5
Most tools have distinct roles, but search_places and discover_local_leads both appear to perform Google Maps business discovery with nearly identical output fields, making selection ambiguous. deep_enrich_lead also overlaps with extract_contacts and crawl_website, though its lead-focused summary provides some differentiation.
Naming Consistency4/5
All names use snake_case and generally follow a verb-first pattern, which is readable and predictable. Minor deviations like batch_extract_web_content and deep_enrich_lead are understandable but slightly less consistent with the core verb_noun convention.
Tool Count5/5
Nine tools is well-scoped for a web crawling, extraction, and lead-generation server. Each tool covers a meaningful capability without excessive fragmentation.
Completeness4/5
The surface covers discovery, single and batch content extraction, structured data, recursive crawling, contact extraction, local search, lead discovery, and enrichment. A few adjacent capabilities such as generic web search or document/PDF extraction are missing, but core crawler and lead workflows are well represented.
Maintenance
ActivityActive
ResponsivenessUnresponsive