LookaCrawler
🕷️ LookaCrawler
Firecrawl의 무료 오픈소스 토큰 효율적 로컬 대안으로, LLM을 위한 네이티브 Model Context Protocol (MCP) 서버를 제공합니다.
📌 왜 LookaCrawler인가?
LLM(대규모 언어 모델)을 위한 웹 크롤링은 기본적으로 결함이 있습니다. 현대 웹 페이지는 방대한 HTML 블로트(스크립트, 추적 픽셀, 중첩된 div, 내비게이션 헤더, 스타일시트)를 포함하고 있어 페이지당 수천 개의 토큰을 낭비합니다.
LookaCrawler는 웹 블로트의 73% 이상 ~ 90%를 제거하고, 깨끗한 Markdown을 추출하며, 스텔스 Playwright 드라이버로 안티봇 장벽을 우회하고, Claude Desktop, Cursor, Antigravity에서 바로 사용할 수 있는 네이티브 Model Context Protocol (MCP) 서버를 제공하는 오픈소스 토큰 최적화 로컬 크롤러입니다.
Related MCP server: TheCrawler
🥊 비교: LookaCrawler vs 대안
기능 | 🕷️ LookaCrawler | 🔥 Firecrawl (Cloud) | ⚡ Jina Reader |
가격 / 비용 | $0.00 (100% 무료 오픈소스) | 월 $16~$99+ | 속도 제한 API |
토큰 절감 | 73% 이상 ~ 90% 적극적 정리 | 표준 Markdown | 기본 Markdown |
데이터 프라이버시 | 100% 로컬 (제로 텔레메트리) | 클라우드 제공자 | 클라우드 API |
MCP 통합 | 네이티브 1-Click 서버 (stdio & SSE) | 커뮤니티 래퍼 | 없음 |
스텔스 & 안티봇 | 실제 Chrome + 스텔스 핑거프린트 | 클라우드 프록시 | 기본 헤더 |
로컬 SQLite 캐시 | 내장 (24시간 TTL 캐시) | Redis / 유료 애드온 | 없음 |
JS SPA 지원 | Playwright + Chrome 풀 | 클라우드 헤드리스 | 헤드리스 |
🚀 주요 기능
토큰 경제 우선: Markdown을 LLM에 전달하기 전에 스크립트, 스타일, 인라인 SVG, 추적 태그, 내비게이션, 푸터, 중복 폼을 자동으로 정리합니다.
이중 하이브리드 크롤링 엔진:
fast: 백오프가 적용된 초고속 네이티브 HTTP GET. 안티봇 챌린지가 감지되면 자동으로deep으로 전환합니다.deep: 스텔스 패치(navigator.webdriver제거, WebGL 스푸핑, CDP 누출 제거)가 적용된 실제 Google Chrome을 실행하는 헤드리스 Playwright 엔진으로, Cloudflare/Turnstile로 보호되는 페이지를 투명하게 크롤링합니다.
네이티브 MCP 서버 프로토콜: Claude Desktop, Cursor, Windsurf, Antigravity에서 사용할 수 있는 JSON-RPC 2.0 stdio 및 SSE 전송.
로컬 SQLite 캐싱: 추출된 Markdown을
crawler_cache.sqlite에 저장하여 중복 네트워크 호출을 제거합니다.구조화된 JSON 및 메타데이터 추출: Open Graph 태그(
og:title,og:description), 게시 날짜, 표준 URL, 사용자 정의 CSS 선택자를 추출합니다.
🔌 1-Click MCP 설정 (Claude Desktop & Cursor)
claude_desktop_config.json 또는 Cursor MCP 설정에 LookaCrawler를 추가하세요:
{
"mcpServers": {
"lookacrawler": {
"command": "bun",
"args": ["run", "/absolute/path/to/lookacrawler/index.ts"]
}
}
}이제 Claude 또는 Cursor에 다음과 같이 요청할 수 있습니다:
"LookaCrawler를 사용하여 https://example.com/docs를 크롤링하고 API 문서를 추출하세요."
📦 빠른 시작 및 CLI 사용법
1. 설치
Bun 1.1+ (또는 Node.js 20+)가 필요합니다:
# Clone the repository
git clone https://github.com/lucasmartins-ai/lookacrawler.git
cd lookacrawler
# Install dependencies
bun install2. CLI 명령어
# Single URL fast Markdown extraction
bun run cli.ts extract https://news.ycombinator.com --mode fast
# Headless Playwright deep extraction with CSS selector target
bun run cli.ts extract https://example.com --mode deep --selector "main" --json
# Batch concurrent multi-URL crawling
bun run cli.ts batch https://site1.com https://site2.com --concurrency 4
# Structured JSON schema extraction
bun run cli.ts structured https://example.com --schema '{"title":"h1","links":"a"}'
# Start MCP Server via SSE on port 3000
bun run cli.ts serve --transport sse --port 30003. Docker 배포
# Build and run Docker container
docker build -t lookacrawler .
docker run -p 3000:3000 lookacrawler🧪 아키텍처 및 테스트
Incoming URL ──► [Local SQLite Cache Check] ──(Hit)──► Return Cached Markdown
│ (Miss)
▼
[Fast HTTP GET Request] ──(Blocked?)──► [Auto-Escalate to Deep Stealth]
│ │
▼ ▼
[HTML DOM Tree Parser] ◄──────────────────────────┘
│
▼
[Aggressive Token Noise Pruner]
(Strips SVG, Nav, Ads, Tracking, CSS, JS)
│
▼
[Mozilla Readability Engine]
│
▼
[Turndown Markdown Converter] ──► Return Clean LLM Markdown테스트 스위트 실행:
bun test⭐ Star & 지원
LookaCrawler가 API 비용과 토큰 비용을 절약해 준다면:
⭐ 이 저장소에 Star를 눌러 다른 개발자들이 이 프로젝트를 찾을 수 있게 도와주세요!
💡 새로운 스텔스 우회 기법이나 크롤러 기능을 위해 Issue / PR을 열어주세요.
제작: LookADev
lookacrawler는 AI 에이전트, 웹 아키텍처, 토큰 최적화를 전문으로 하는 엔지니어링 스튜디오 LookADev가 만들고 유지 관리합니다.
프로젝트 시작하기 → lookadev.com · 이메일: lucas@lookadev.com
📄 라이선스
MIT 라이선스에 따라 배포되는 오픈소스 소프트웨어입니다.
Available Tools
9 toolsbatch_extract_web_contentA
Batch extract token-optimized Markdown from multiple website URLs concurrently, with aggregate token statistics.
What it does Runs the same fast/deep pipeline as extract_web_content across an array of up to 20 URLs in parallel, then returns per-URL results plus a summary (urls attempted, succeeded, failed, total tokens, and per-page token counts).
How to use
Use when you need many pages at once — docs, a list of product pages, a set of sources to compare.
mode applies to every URL. concurrency (1-10) caps parallel workers; keep it modest on fast machines / low bandwidth.
css_selector is applied to all URLs. headers/cookies/proxy apply globally to every request.
Each URL is independent: a single failure does not abort the batch (results carry an "error" field for that URL).
Returns Structured JSON with per-url markdown/tokens/error and aggregate totals. Incomplete or empty batches still return a valid result object.
Inputs: urls (required), mode, proxy, cookies, +4 more.
crawl_websiteB
Recursively crawl an entire website or documentation section with depth limits, URL pattern filtering, and token accounting.
Inputs: start_url (required), mode, max_depth, max_pages, +4 more.
deep_enrich_leadB
Deeply enrich a business lead by crawling its official website with LookaCrawler's token-optimized engine. Extracts verified emails, social media links (Instagram, LinkedIn), decision makers / founders, and generates a concise Markdown summary (<300 tokens) for LLMs.
Inputs: title (required), website (required), mode, phone, +4 more.
discover_local_leadsA
Discover local businesses, establishments, and B2B prospects using the Google Maps engine. Extracts title, category, formatted address, telephone, WhatsApp link, rating, review count, coordinates, and website URL.
Inputs: query (required), lang, limit, location.
extract_contactsA
Extract verified contact information (emails, phone numbers, WhatsApp links, and social media profiles) from a website and its contact subpages. Drastically saves tokens compared to feeding the entire raw DOM to the LLM.
Inputs: url (required), deep_scan.
extract_structured_dataA
Extract structured JSON — page metadata (Open Graph, canonical URL, author, dates) and/or a custom CSS-selector schema — from a website.
What it does Fetches a URL and returns a structured object. With include_metadata=true it reads Open Graph tags (og:title, og:description, og:image), the canonical URL, author, and publication/modified dates. You can also pass a schema (property name -> CSS selector) to pull specific fields (e.g. {"title": "h1", "price": ".price", "rating": "span.rating"}) into a typed object.
How to use
Use for product listings, news/metadata, schema-driven extraction, and any time you need fields instead of prose.
schema keys become the JSON keys; values are CSS selectors. Leave schema empty to get metadata only.
include_metadata (default true) adds the Open Graph / author / date block. Set false for a lean result.
mode/css_selector/headers/cookies/proxy behave exactly as in extract_web_content.
Returns JSON object with the requested fields (and/or metadata). Missing selectors are omitted rather than failing the whole call. Non-http or private URLs fail with isError: true.
Inputs: url (required), mode, proxy, schema, +5 more.
extract_web_contentA
Extract token-optimized, clean Markdown content from a target website for LLM consumption.
What it does Fetches a URL, strips HTML bloat (scripts, styles, inline SVGs, tracking tags, navigation, footers, redundant forms) and returns readable Markdown — cutting token usage by ~73% to 90% versus the raw page.
How to use
Use for any web page, article, or docs you want as LLM-ready text.
mode="fast" (default): native HTTP GET. mode="deep": headless Chrome with stealth fingerprint for JS-heavy or Cloudflare/Turnstile-protected pages.
css_selector scopes to a single element (e.g. "main", "article", "#content").
Pass headers/cookies for gated or geo-restricted content. Providing any of headers/cookies/proxy bypasses the SQLite cache.
No auth required. Respects robots/rate-limit via retry backoff.
Returns Markdown text. Cache hits are prefixed with a NOTE marker. Unsupported URLs (non-http, private IP, SSRF) fail with isError: true and a reason.
Inputs: url (required), mode, proxy, actions, +6 more.
map_websiteA
Discover the architecture and page inventory of a website domain via sitemaps, robots.txt, and link graphs. Returns a list of URLs with source and modification metadata. Useful for exploring a site before targeted crawling.
Inputs: url (required), max_links.
search_placesA
Search for local businesses, places, and establishments on Google Maps, extracting structured entity data (title, address, phone, website, coordinates, rating).
Inputs: query (required), limit.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 9 tools
Most tools target distinct actions, but two near-duplicate pairs stand out: discover_local_leads vs search_places (both pull the same Google Maps entity fields), and extract_contacts vs deep_enrich_lead (both crawl a business site for emails/socials). Descriptions add some differentiation, but an agent could easily misselect within these pairs.
All names are snake_case and follow a verb_noun pattern (discover_local_leads, extract_contacts, crawl_website, map_website, search_places). Minor compounds like deep_enrich_lead and batch_extract_web_content are still readable and consistent with the convention.
Nine tools is well within the sweet spot for a crawling/scraping server, and each tool covers a distinct capability tier (discovery, extraction, enrichment, batch, site mapping). No obvious padding or redundancy-driven bloat.
The surface covers the full lead-gen lifecycle: place discovery, contact extraction, lead enrichment, content/markdown extraction, recursive crawl, batch, structured extraction, and sitemap discovery. The main gap is a lack of a status/polling or pagination/continuation operation for long-running crawls or large result sets.
Maintenance
Related MCP Connectors
Web pages, PDFs and search results as LLM-ready Markdown or chunks. Firecrawl-compatible, EU-hosted.
Any web page -> clean markdown. Firecrawl-lite, 0.001 USDC/extract (x402, Base), free demo.
Keyless Firecrawl/Exa alternative: URL to LLM Markdown, live web search & $4.90 30d pass.
41Firecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.
Related MCP Servers
- AlicenseAqualityCmaintenanceMCP server for web scraping — extract clean markdown, links, and metadata from any URL. Free Firecrawl alternative.5119 npm6MIT
- AlicenseNot gradedqualityDmaintenanceOpen-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.2AGPL 3.0
- AlicenseAqualityBmaintenanceA fault-tolerant, stealth-enabled Model Context Protocol (MCP) server for web searching and content fetching. Built for AI Agents (Cursor, Claude Code, OpenCode), it uses a stealth browser engine to fetch pages, dynamically handles SPAs/React, and converts bloat into token-optimized Markdown.214 npm20 PyPI4MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and LLM pipelines to fetch and crawl websites locally, extract readable content, and convert pages into clean Markdown with preserved links while minimizing token usage.-