LookaCrawler
This server lets LLMs extract clean, token-optimized Markdown and structured metadata from websites through configurable fast/deep crawling modes.
Extract web content: Convert a single URL into clean, LLM-ready Markdown, stripping HTML/script bloat.
Batch extraction: Crawl up to 20 URLs concurrently and get aggregate token statistics.
Extract structured data: Pull Open Graph metadata, canonical URL, author, dates, and custom CSS-selector-based JSON fields.
Choose crawl mode: Use
fast(HTTP fetch) ordeep(headless Playwright with JS execution) per request.Customize requests: Set proxies, cookies, headers, retries, and CSS scoping selectors.
Control concurrency: Adjust parallel workers for batch jobs (1โ10).
Integrate with MCP clients: Exposes tools callable from Claude Desktop, Cursor, and other MCP-compatible assistants.
๐ท๏ธ LookaCrawler
Free, open-source, token-efficient local alternative to Firecrawl with native Model Context Protocol (MCP) Server for LLMs.
๐ Why LookaCrawler?
Web crawling for Large Language Models (LLMs) is broken by default: modern web pages contain massive HTML bloat (scripts, tracking pixels, nested divs, navigation headers, stylesheets), costing thousands of wasted tokens per page.
LookaCrawler is an open-source, token-optimized local crawler that strips >73% to 90% of web bloat, extracts clean Markdown, bypasses anti-bot barriers with stealth Playwright drivers, and exposes a native Model Context Protocol (MCP) Server ready for Claude Desktop, Cursor, and Antigravity.
Related MCP server: Scraper MCP
๐ฅ Comparison: LookaCrawler vs Alternatives
Feature | ๐ท๏ธ LookaCrawler | ๐ฅ Firecrawl (Cloud) | โก Jina Reader |
Pricing / Cost | $0.00 (100% Free Open Source) | $16 to $99+/month | Rate-limited API |
Token Reduction | >73% to 90% pruning + Footnotes | Standard Markdown | Basic Markdown |
Autonomous Crawling | Native | Cloud Crawler | Single-page only |
Pre-Crawl Actions | Native Playwright (click, scroll, fill) | Paid Addon | None |
Link Formatting | Inline, References Footnotes, Strip | Inline only | Inline only |
Data Privacy | 100% Local (Zero Telemetry) | Cloud Provider | Cloud API |
MCP Integration | Native Tools + Resources + Prompts | Community Wrapper | None |
Stealth & Anti-Bot | Real Chrome + Stealth Fingerprint | Cloud Proxies | Basic Headers |
Local SQLite Cache | Built-in (24h TTL cache) | Redis / Paid Addon | None |
JS SPA Support | Playwright + Chrome Pool | Cloud Headless | Headless |
๐ Key Features
Token Economy First: Automatically prunes scripts, styles, inline SVGs, tracking tags, navigations, footers, redundant forms, and boilerplate containers with high link density (>80%).
Advanced Link & Image Formatting:
link_format: Choose betweeninline(standard markdown),references(footnote citations[1], saving ~25% tokens on repetitive URLs), orstrip(pure text).image_mode: Choose betweenignore(zero tokens),alt_only(preserves semantic context without URL bloat), ormarkdown(full).
Autonomous Mapping & Recursive Crawling:
map_website: Inspects/robots.txt, sitemaps, and root anchors to discover all pages in a domain.crawl_website: Breadth-first autonomous crawling with max depth, max pages, route regex filters, and real-time token accounting.
Pre-Crawl Browser Actions: Automate clicks, scrolls, typing, and waits in Playwright before extracting content (dismiss cookie banners, scroll for infinite loading, expand accordions).
Dual Hybrid Crawling Engine:
fast: Ultra-fast native HTTP GET with backoff. Auto-escalates todeepif an anti-bot challenge is encountered.deep: Headless Playwright engine launching real Google Chrome with stealth patches (navigator.webdrivercleared, WebGL spoofed, CDP leaks stripped) to transparently crawl Cloudflare/Turnstile-protected pages.
Native MCP Ecosystem:
Tools:
extract_web_content,crawl_website,map_website,batch_extract_web_content,extract_structured_data.Resources: Live telemetry at
crawler://metricsand cache analytics atcrawler://cache/stats.Prompts: Pre-engineered templates
crawl-and-summarizeandcompare-pages.
Local SQLite Caching: Stores extracted Markdown in
crawler_cache.sqliteto eliminate duplicate network calls.Structured JSON & Metadata Extraction: Extracts Open Graph tags (
og:title,og:description), publication dates, canonical URLs, and custom CSS selectors.
๐ 1-Click MCP Setup (Claude Desktop & Cursor)
Add LookaCrawler to your claude_desktop_config.json or Cursor MCP settings:
{
"mcpServers": {
"lookacrawler": {
"command": "bun",
"args": ["run", "/absolute/path/to/lookacrawler/index.ts"]
}
}
}Now you can prompt Claude or Cursor:
"Crawl https://example.com/docs and extract the API documentation using LookaCrawler."
๐ฆ Quick Start & CLI Usage
1. Installation
Requires Bun 1.1+ (high-performance runtime with native SQLite):
# Clone the repository
git clone https://github.com/lucasmartins-ai/lookacrawler.git
cd lookacrawler
# Install dependencies
bun install2. CLI Commands
# Single URL fast Markdown extraction
bun run cli.ts extract https://news.ycombinator.com --mode fast
# Single URL fast Markdown extraction with reference footnotes
bun run cli.ts extract https://example.com --link-format references --output page.md
# Headless Playwright deep extraction with CSS selector target
bun run cli.ts extract https://example.com --mode deep --selector "main" --json
# Discover all website URLs and sitemaps
bun run cli.ts map https://example.com --max-urls 500
# Recursively crawl documentation with regex filtering and token accounting
bun run cli.ts crawl https://example.com/docs --max-depth 2 --max-pages 15 --link-format references
# Batch concurrent multi-URL crawling
bun run cli.ts batch https://site1.com https://site2.com --concurrency 4
# Structured JSON schema extraction
bun run cli.ts structured https://example.com --schema '{"title":"h1","links":"a"}'
# Start MCP Server via SSE on port 3000
bun run cli.ts serve --transport sse --port 30003. Docker Deployment
# Build and run Docker container
docker build -t lookacrawler .
docker run -p 3000:3000 lookacrawler๐งช Architecture & Testing
Incoming URL โโโบ [Local SQLite Cache Check] โโ(Hit)โโโบ Return Cached Markdown
โ (Miss)
โผ
[Fast HTTP GET Request] โโ(Blocked?)โโโบ [Auto-Escalate to Deep Stealth]
โ โ
โผ โผ
[HTML DOM Tree Parser] โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ
โผ
[Aggressive Token Noise Pruner]
(Strips SVG, Nav, Ads, Tracking, CSS, JS)
โ
โผ
[Mozilla Readability Engine]
โ
โผ
[Turndown Markdown Converter] โโโบ Return Clean LLM MarkdownRun test suite:
bun testโญ Star & Support
If LookaCrawler saves you API fees and token costs:
โญ Star this repository to help other developers find it!
๐ก Open an Issue / PR for new stealth bypasses or crawler features.
Built by LookADev
lookacrawler is built and maintained by LookADev, an engineering studio specializing in AI agents, web architecture, and token optimization.
Start a project โ lookadev.com ยท Email: lucas@lookadev.com
๐ License
Open-source software licensed under the MIT License.
Available Tools
3 toolsbatch_extract_web_contentA
Batch extract token-optimized Markdown content from multiple website URLs concurrently with aggregate token statistics.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | Crawl mode: 'fast' (native HTTP fetch) or 'deep' (headless Playwright browser). | fast |
| urls | Yes | Array of target website URLs to extract. | |
| proxy | No | Optional HTTP/SOCKS5 proxy URL. | |
| cookies | No | Optional custom HTTP cookies key-value dictionary. | |
| headers | No | Optional custom HTTP request headers key-value dictionary. | |
| concurrency | No | Maximum parallel HTTP/browser crawl worker concurrency (default: 3). | |
| max_retries | No | Maximum retry attempts per URL (default: 3). | |
| css_selector | No | Optional CSS selector to filter DOM node across all target URLs. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral disclosure burden. It does disclose concurrency, output format, and token statistics, which are genuinely useful. However, it does not mention failure behavior, retries, partial failures, rate limits, or what happens when a URL cannot be fetched.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence that front-loads the core purpose, then adds concurrency and statistical output details. There is no filler, and every phrase adds distinguishing information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no annotations and no output schema, the description must supply more contextual completeness. It gives the essential purpose and concurrency trait, but it omits output shape, retry/failure semantics, and any usage comparison to sibling tools. For an 8-parameter tool with 100% schema coverage, this is still not fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% parameter description coverage, so the schema already documents all parameters well. The description adds no parameter-specific meaning beyond stating that the tool works on multiple URLs; therefore the baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the action ('batch extract'), resource ('web content from multiple website URLs'), output format ('token-optimized Markdown'), and a distinctive trait ('concurrently with aggregate token statistics'). This distinguishes it from the likely single-URL sibling `extract_web_content` and from structured extraction (`extract_structured_data`).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The word 'batch' and phrase 'multiple website URLs' imply this tool is for multi-URL scenarios, which provides useful context. However, there is no explicit guidance about when to prefer this over `extract_web_content` or `extract_structured_data`, and no stated exclusion conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_structured_dataC
Extract page metadata (OG tags, canonical URL, author, dates) and custom CSS selector JSON schema mapping from a website.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target website URL to extract content and metadata from. | |
| mode | No | Crawl mode: 'fast' (native fetch) or 'deep' (Playwright Chromium). | fast |
| proxy | No | Optional HTTP/SOCKS5 proxy URL. | |
| schema | No | Optional key-value map of property names to CSS selectors (e.g. { title: 'h1', price: '.price' }). | |
| cookies | No | Optional custom HTTP cookies key-value dictionary. | |
| headers | No | Optional custom HTTP request headers key-value dictionary. | |
| max_retries | No | Maximum retry attempts (default: 3). | |
| css_selector | No | Optional CSS selector to scope content before processing. | |
| include_metadata | No | Whether to extract Open Graph tags, canonical URL, author, and date metadata (default: true). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure, but it only states what is extracted and from where. It does not disclose that this likely performs a live network fetch, any side effects, permission or rate-limit considerations, or distinctions between the 'fast' and 'deep' modes in terms of behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single focused sentence with no superfluous words. It leads with the main action and resources, making it efficient and easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 9 parameters, nested objects, no output schema, and sibling tools to disambiguate, this one-sentence description is insufficiently complete. It omits usage guuidelines, behavioral expectations, and return-value structure, leaving meaningful gaps for an agent trying to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers all 9 parameters with detailed descriptions, including examples, so the baseline is 3. The description adds little beyond the schema: it lists example metadata fields (OG tags, canonical URL, author, dates) which clarifies output scope, but does not elaborate on parameter syntax or interplay.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('Extract') and resources ('page metadata', 'custom CSS selector JSON schema mapping'), making the tool's purpose understandable. It does not explicitly differentiate from sibling tools like extract_web_content or batch_extract_web_content, so it misses the top tier by a narrow margin.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided about when to use this tool versus the sibling tools extract_web_content or batch_extract_web_content. The description does not mention any conditions, alternatives, or exclusions, leaving the agent to infer usage entirely from names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_web_contentC
Extract token-optimized clean Markdown content from a target website for LLM consumption.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target website URL to extract content from. | |
| mode | No | Crawl mode: 'fast' (native HTTP fetch) or 'deep' (headless Playwright browser with JS execution). | fast |
| proxy | No | Optional HTTP/SOCKS5 proxy URL (e.g. 'http://proxy.example.com:8080'). | |
| cookies | No | Optional custom HTTP cookies key-value dictionary. | |
| headers | No | Optional custom HTTP request headers key-value dictionary. | |
| max_retries | No | Maximum retry attempts for transient errors or rate limits (default: 3). | |
| css_selector | No | Optional CSS selector to scope content extraction to a specific HTML node. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description carries the full burden of behavioral disclosure. It only states the output is token-optimized clean Markdown; it does not reveal that the tool makes live network requests, that 'deep' mode executes JavaScript via a headless browser, how failures or rate limits are handled, or any side effects. The schema documents the mode options, but the description itself leaves key behavioral traits undisclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that communicates the primary purpose and output format without wasted words. It is appropriately concise, though a little more detail about usage trade-offs would have made it richer without harming structure.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 7 parameters, nested objects, and no output schema or annotations, yet the description only covers the surface purpose. It omits important operational context such as dynamic content handling, proxy/cookie use cases, retry behavior, and how to decide between 'fast' and 'deep' modes. An agent receives the raw schema but not sufficient high-level orientation for a tool of this complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter already has a clear description in the schema. The tool description adds little beyond 'clean Markdown', which indirectly hints at the purpose of css_selector scoping but does not meaningfully extend parameter understanding. Baseline 3 is appropriate because the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description has a clear verb ('Extract'), a specific resource ('content from a target website'), and a defined output format ('clean Markdown content'). It conveys the tool's core purpose well and inherently contrasts with extract_structured_data by promising Markdown. However, it does not explicitly distinguish itself from batch_extract_web_content, leaving the single-URL versus batch distinction to be inferred from the sibling name.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided about when to use this tool versus its siblings. There is no mention of batch_extract_web_content for bulk jobs or extract_structured_data for non-Markdown outputs. The 'for LLM consumption' phrase gives some context but does not help an agent choose among alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
3 tool updates
v0.1.0- First observed
batch_extract_web_content - First observed
extract_structured_data - First observed
extract_web_content
TDQS
The three tools have clear boundaries: single content extraction, batch content extraction, and structured metadata/CSS mapping. The only soft spot is that batch and single share the same core operation, but their singular-vs-batch distinction prevents real ambiguity.
All tools use lowercase snake_case and follow a predictable extract_<object> naming pattern; batch_ is a standard parallelism modifier on the same verb. There are no mixed conventions or vague verbs.
Three tools is appropriate for a narrowly scoped extraction server: one direct, one batched, and one structured metadata. It is not bloated, and slightly minimal but credible for this purpose.
Core extraction workflow is covered: single page, batch pages, and structured metadata. However, the crawler name implies link discovery or site traversal, and no tool enumerates links or sitemaps, so agents need URLs supplied beforehand. This is a notable but not deabilitating gap.
Maintenance
Related MCP Connectors
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Firecrawl MCP โ wraps the Firecrawl API (firecrawl.dev) for web
- fastCRWOAuthio.github.us
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
Related MCP Servers
- AlicenseAqualityDmaintenanceA Model Context Protocol server that enables web scraping, crawling, and content extraction capabilities through integration with Firecrawl.840,1392MIT
- AlicenseNot gradedqualityDmaintenanceA context-optimized web scraping server that converts HTML to markdown/text and applies CSS selectors server-side, reducing token usage by 70-90% while providing AI tools with clean, filtered web content.7MIT
- AlicenseAqualityCmaintenanceMCP server for web scraping โ extract clean markdown, links, and metadata from any URL. Free Firecrawl alternative.51575MIT
- AlicenseAqualityAmaintenanceWeb content extraction for AI agents. 10 tools: scrape, crawl, map, batch, extract, summarize, diff, brand, search, research. Uses TLS fingerprinting to bypass anti-bot without a headless browser. Outputs LLM-optimized markdown with 67% fewer tokens than raw HTML.102,316AGPL 3.0
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/lucasmartins-ai/lookacrawler'
If you have feedback or need assistance with the MCP directory API, please join our Discord server