Skip to main content
Glama

๐Ÿ•ท๏ธ LookaCrawler

Free, open-source, token-efficient local alternative to Firecrawl with native Model Context Protocol (MCP) Server for LLMs.

License: MIT Bun TypeScript MCP Ready GitHub Stars Local-First


๐Ÿ“Œ Why LookaCrawler?

Web crawling for Large Language Models (LLMs) is broken by default: modern web pages contain massive HTML bloat (scripts, tracking pixels, nested divs, navigation headers, stylesheets), costing thousands of wasted tokens per page.

LookaCrawler is an open-source, token-optimized local crawler that cuts a median 90% of tokens on real pages (21โ€“98% across 8 sites, see benchmark-live.ts), extracts clean Markdown, bypasses anti-bot barriers with stealth Playwright drivers, and exposes a native Model Context Protocol (MCP) Server ready for Claude Desktop, Cursor, and Antigravity.


Related MCP server: mcp-server-scraper

๐ŸฅŠ Comparison: LookaCrawler vs Alternatives

Feature

๐Ÿ•ท๏ธ LookaCrawler

๐Ÿ”ฅ Firecrawl (Cloud)

โšก Jina Reader

Pricing / Cost

$0.00 (100% Free Open Source)

$16 to $99+/month

Rate-limited API

Token Reduction

Median 90% fewer tokens (real pages) + Footnotes

Standard Markdown

Basic Markdown

Autonomous Crawling

Native map & crawl (BFS + URL patterns)

Cloud Crawler

Single-page only

Pre-Crawl Actions

Native Playwright (click, scroll, fill)

Paid Addon

None

Link Formatting

Inline, References Footnotes, Strip

Inline only

Inline only

Data Privacy

100% Local (Zero Telemetry)

Cloud Provider

Cloud API

MCP Integration

Native Tools + Resources + Prompts

Community Wrapper

None

Stealth & Anti-Bot

Real Chrome + Stealth Fingerprint

Cloud Proxies

Basic Headers

Local SQLite Cache

Built-in (24h TTL cache)

Redis / Paid Addon

None

JS SPA Support

Playwright + Chrome Pool

Cloud Headless

Headless


๐Ÿš€ Key Features

  • Token Economy First: Automatically prunes scripts, styles, inline SVGs, tracking tags, navigations, footers, redundant forms, and boilerplate containers with high link density (>80%).

  • Advanced Link & Image Formatting:

    • link_format: Choose between inline (standard markdown), references (footnote citations [1], saving ~25% tokens on repetitive URLs), or strip (pure text).

    • image_mode: Choose between ignore (zero tokens), alt_only (preserves semantic context without URL bloat), or markdown (full ![alt](url)).

  • Autonomous Mapping & Recursive Crawling:

    • map_website: Inspects /robots.txt, sitemaps, and root anchors to discover all pages in a domain.

    • crawl_website: Breadth-first autonomous crawling with max depth, max pages, URL pattern filters (substring, * wildcard), and real-time token accounting.

  • Pre-Crawl Browser Actions: Automate clicks, scrolls, typing, and waits in Playwright before extracting content (dismiss cookie banners, scroll for infinite loading, expand accordions).

  • Dual Hybrid Crawling Engine:

    • fast: Ultra-fast native HTTP GET with backoff. Auto-escalates to deep if an anti-bot challenge is encountered.

    • deep: Headless Playwright engine launching real Google Chrome with stealth patches (navigator.webdriver cleared, WebGL spoofed, CDP leaks stripped) to transparently crawl Cloudflare/Turnstile-protected pages.

  • Native MCP Ecosystem:

    • Tools: extract_web_content, crawl_website, map_website, batch_extract_web_content, extract_structured_data, extract_contacts, search_places, discover_local_leads, deep_enrich_lead.

    • Resources: Live telemetry at crawler://metrics and cache analytics at crawler://cache/stats.

    • Prompts: Pre-engineered templates crawl-and-summarize and compare-pages.

  • Entity Extraction: contacts pulls emails, phone numbers, WhatsApp links and social profiles; places returns structured Google Maps business records with formula-injection-safe CSV output.

  • B2B Lead Discovery: maps (Google Maps engine) plus enrich, which crawls a company's own site for verified emails, socials and decision makers.

  • Local SQLite Caching: Stores extracted Markdown in crawler_cache.sqlite to eliminate duplicate network calls.

  • Structured JSON & Metadata Extraction: Extracts Open Graph tags (og:title, og:description), publication dates, canonical URLs, and custom CSS selectors.


๐Ÿ”Œ 1-Click MCP Setup (Claude Desktop & Cursor)

Add LookaCrawler to your claude_desktop_config.json or Cursor MCP settings:

{
  "mcpServers": {
    "lookacrawler": {
      "command": "bun",
      "args": ["run", "/absolute/path/to/lookacrawler/index.ts"]
    }
  }
}

Now you can prompt Claude or Cursor:

"Crawl https://example.com/docs and extract the API documentation using LookaCrawler."


๐Ÿ“ฆ Quick Start & CLI Usage

1. Installation

Requires Bun 1.1+ (high-performance runtime with native SQLite):

# Clone the repository
git clone https://github.com/lucasmartins-ai/lookacrawler.git
cd lookacrawler

# Install dependencies
bun install

2. CLI Commands

# Single URL fast Markdown extraction
bun run cli.ts extract https://news.ycombinator.com --mode fast

# Single URL fast Markdown extraction with reference footnotes
bun run cli.ts extract https://example.com --link-format references --output page.md

# Headless Playwright deep extraction with CSS selector target
bun run cli.ts extract https://example.com --mode deep --selector "main" --json

# Discover all website URLs and sitemaps
bun run cli.ts map https://example.com --max-urls 500

# Recursively crawl documentation with URL pattern filtering and token accounting
bun run cli.ts crawl https://example.com/docs --max-depth 2 --max-pages 15 --include /docs/ --link-format references

# Batch concurrent multi-URL crawling
bun run cli.ts batch https://site1.com https://site2.com --concurrency 4

# Structured JSON schema extraction
bun run cli.ts structured https://example.com --schema '{"title":"h1","links":"a"}'

# Extract emails, phones, WhatsApp links and social profiles from a site
bun run cli.ts contacts https://example.com

# Search local businesses on Google Maps (browser-driven)
bun run cli.ts places "dentists in Bristol" --limit 5 --csv -o places.csv

# B2B lead discovery via the sibling google-maps-scraper engine
# (set MAPS_LEADS_BIN if the binary is not in a standard path)
bun run cli.ts maps "dental clinic" "Bristol" --limit 20

# Enrich a company website: emails, socials, decision makers, short summary
bun run cli.ts enrich https://example.com --title "Example Ltd" --mode fast

# Enrich many leads at once from a JSON array of {title, website}
bun run cli.ts enrich-batch leads.json --concurrency 3 -o enriched.json

# Start MCP Server via SSE on port 3000
bun run cli.ts serve --transport sse --port 3000

Behind a TLS-intercepting proxy? Export LOOKACRAWLER_INSECURE_TLS=true to accept the proxy's forged certificate. Off by default โ€” never enable it for untrusted networks.

Rotating proxies? Export LOOKACRAWLER_PROXIES with a comma or whitespace separated list of URLs. Every request without an explicit --proxy pulls the next entry, so one blocked IP is not fatal for the run.

Local proxy (Tor, mitmproxy, a sidecar)? A proxy passed per request must be a public host. List trusted local proxies in LOOKACRAWLER_ALLOWED_PROXIES (comma separated, exact URLs); entries in LOOKACRAWLER_PROXIES are trusted too.

SSE server security

Variable

Effect

LOOKACRAWLER_SSE_TOKEN

Bearer token required on every request. If the server is bound to a non-loopback address (e.g. --host 0.0.0.0) and this is unset, a random token is generated and printed at startup.

LOOKACRAWLER_SSE_ALLOWED_ORIGINS

Comma-separated web origins allowed to call the server from a browser (CORS). Requests with any other Origin header are refused. CLI/desktop MCP clients are unaffected.

LOOKACRAWLER_SSE_ALLOW_UNAUTHENTICATED

true disables the token requirement on non-loopback binds. Only for isolated networks.

LOOKACRAWLER_SSE_MAX_SESSIONS

Maximum concurrent SSE clients (each gets its own MCP session). Default 16; further connections get 503 until one disconnects.

LOOKACRAWLER_MAX_BROWSERS

Maximum concurrent Chromium processes (one per distinct proxy). Default 4; idle ones are closed after LOOKACRAWLER_BROWSER_IDLE_MS (default 120000).

Every outbound connection goes through a built-in loopback egress guard that resolves each destination once, refuses private/link-local/loopback addresses and connects to exactly the IP it checked. This covers redirects and requests a page makes on its own (fetch, XHR, WebSocket) in deep mode, and prevents DNS rebinding. With a proxy (--proxy, LOOKACRAWLER_PROXIES) the guard sits in front of it: it validates the destination, then asks the proxy to connect to that IP (HTTP CONNECT, SOCKS5 with optional user/password, or SOCKS4), so a proxy โ€” even a trusted local one โ€” can never be steered at an internal host. Side effect: DNS is resolved on this machine, not at the proxy, so geo-DNS/CDN edges follow your location rather than the proxy's, and SOCKS4 cannot reach IPv6-only sites. LOOKACRAWLER_ALLOW_LOCAL=true turns all of this off and talks to the proxy directly (for local development only).

3. Google Maps engine (optional)

The maps command and the discover_local_leads MCP tool delegate to a sibling Google Maps scraper binary. It is a separate Go project (google-maps-scraper, MIT) and is not vendored into this repository.

Place the compiled maps-leads binary in one of these locations, or point MAPS_LEADS_BIN at it:

./google-maps-scraper/maps-leads          # repo root
../google-maps-scraper/maps-leads         # sibling directory
$MAPS_LEADS_BIN                           # anywhere

Without it, maps fails fast with a clear message; extract, crawl, contacts, places and enrich are unaffected. Pass --email to let the scraper extract public emails during discovery.

4. Docker Deployment

# Build and run Docker container
docker build -t lookacrawler .
docker run -p 3000:3000 -e LOOKACRAWLER_SSE_TOKEN="$(openssl rand -hex 24)" lookacrawler

The container listens on 0.0.0.0, so a token is always required. Without LOOKACRAWLER_SSE_TOKEN a random one is generated on each start and printed in the container logs.


๐Ÿงช Architecture & Testing

Incoming URL โ”€โ”€โ–บ [Local SQLite Cache Check] โ”€โ”€(Hit)โ”€โ”€โ–บ Return Cached Markdown
                       โ”‚ (Miss)
                       โ–ผ
            [Fast HTTP GET Request] โ”€โ”€(Blocked?)โ”€โ”€โ–บ [Auto-Escalate to Deep Stealth]
                       โ”‚                                      โ”‚
                       โ–ผ                                      โ–ผ
            [HTML DOM Tree Parser] โ—„โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                       โ”‚
                       โ–ผ
       [Aggressive Token Noise Pruner]
       (Strips SVG, Nav, Ads, Tracking, CSS, JS)
                       โ”‚
                       โ–ผ
         [Mozilla Readability Engine]
                       โ”‚
                       โ–ผ
         [Turndown Markdown Converter] โ”€โ”€โ–บ Return Clean LLM Markdown

Run test suite:

bun test

โญ Star & Support

If LookaCrawler saves you API fees and token costs:

  • โญ Star this repository to help other developers find it!

  • ๐Ÿ’ก Open an Issue / PR for new stealth bypasses or crawler features.


Built by LookADev

lookacrawler is built and maintained by LookADev, an engineering studio specializing in AI agents, web architecture, and token optimization.

Start a project โ†’ lookadev.com ยท Email: lucas@lookadev.com

๐Ÿ“„ License

Open-source software licensed under the MIT License.

Available Tools

9 tools
batch_extract_web_contentA

Batch extract token-optimized Markdown from multiple website URLs concurrently, with aggregate token statistics.

What it does Runs the same fast/deep pipeline as extract_web_content across an array of up to 20 URLs in parallel, then returns per-URL results plus a summary (urls attempted, succeeded, failed, total tokens, and per-page token counts).

How to use

  • Use when you need many pages at once โ€” docs, a list of product pages, a set of sources to compare.

  • mode applies to every URL. concurrency (1-10) caps parallel workers; keep it modest on fast machines / low bandwidth.

  • css_selector is applied to all URLs. headers/cookies/proxy apply globally to every request.

  • Each URL is independent: a single failure does not abort the batch (results carry an "error" field for that URL).

Returns Structured JSON with per-url markdown/tokens/error and aggregate totals. Incomplete or empty batches still return a valid result object.

ParametersJSON Schema
NameRequiredDescriptionDefault
modeNoCrawl mode: 'fast' (native HTTP fetch) or 'deep' (headless Playwright browser).fast
urlsYesArray of target website URLs to extract (1-20).
proxyNoOptional HTTP/SOCKS5 proxy URL.
cookiesNoOptional custom HTTP cookies key-value dictionary.
headersNoOptional custom HTTP request headers key-value dictionary.
concurrencyNoMaximum parallel HTTP/browser crawl worker concurrency (default: 3).
max_retriesNoMaximum retry attempts per URL (default: 3).
css_selectorNoOptional CSS selector to filter DOM node across all target URLs.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden, and it discharges much of it: failures are isolated per URL ('a single failure does not abort the batch'), results carry an 'error' field, and empty/incomplete batches still return a valid object. It omits auth/credential requirements, rate limits, and timeout behavior, which keeps it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded one-line summary followed by labelled 'What it does / How to use / Returns' sections; every bullet adds operational value. The section headings cost a little space but the content is dense and non-repetitive, so it stays efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema AND no annotations, the description must cover both behavior and returns, and it does explain the return shape (per-URL markdown/tokens/error plus aggregate totals). It is complete enough to call correctly, though it leaves prerequisite/auth and rate-limit concerns unstated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3, but the description adds cross-parameter semantics the schema cannot express: mode/css_selector/headers/cookies/proxy apply to every URL in the array, concurrency (1-10) caps parallel workers with advice to keep it modest, and each URL is independent. That is real meaning beyond the per-parameter docs.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource+scope: 'Batch extract token-optimized Markdown from multiple website URLs concurrently, with aggregate token statistics.' It explicitly relates itself to the sibling extract_web_content as the same pipeline applied to an array, so an agent can tell the batch tool from the single-URL tool without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear context: 'Use when you need many pages at once โ€” docs, a list of product pages, a set of sources to compare,' and explains how global options (mode, css_selector, headers/cookies/proxy) apply across all URLs, plus concurrency guidance. It does not state when NOT to use it (e.g., 'for a single URL use extract_web_content'), so routing is implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_website

Recursively crawl an entire website or documentation section with depth limits, URL pattern filtering, and token accounting.

ParametersJSON Schema
NameRequiredDescriptionDefault
modeNoCrawl mode ('fast' native fetch or 'deep' Playwright browser).fast
max_depthNoMaximum link traversal depth (default: 2).
max_pagesNoMaximum total pages to crawl (default: 5, max: 20).
start_urlYesRoot URL to start the recursive crawl from.
image_modeNoFormatting of images.ignore
link_formatNoFormatting of extracted hyperlinks.inline
exclude_patternsNoSubstring patterns (`*` = wildcard) for URLs to ignore (e.g. ['/login', '/tags/']). Max 10.
include_patternsNoSubstring patterns (`*` = wildcard) that candidate URLs must match (e.g. ['/docs/']). Max 10.
deep_enrich_leadB

Deeply enrich a business lead by crawling its official website with LookaCrawler's token-optimized engine. Extracts verified emails, social media links (Instagram, LinkedIn), decision makers / founders, and generates a concise Markdown summary (<300 tokens) for LLMs.

ParametersJSON Schema
NameRequiredDescriptionDefault
modeNoCrawl mode: 'fast' (HTTP) or 'deep' (stealth Playwright)fast
phoneNoKnown phone number
titleYesCompany or establishment name
websiteYesCompany official website URL to crawl
categoryNoCompany category or niche
whatsappNoKnown WhatsApp link
image_modeNoImage mode: alt_only avoids CDN bloatalt_only
link_formatNoLink formatting: references footnotes for token savingsreferences

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses output artifacts (verified emails, social links, founders, a <300-token Markdown summary) and the token-optimized engine, which is useful behavioral context. But it omits cost/latency implications, rate limits, auth requirements, or what happens when the crawl fails.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tightly written sentences that front-load the action and then the outputs. No filler, no restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 8-parameter, no-output-schema tool, the description covers the essential return content (emails, socials, decision makers, markdown summary) so an agent knows what comes back. Missing only failure/edge-case behavior and any alternative-tool routing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% with 8 documented parameters and three enums, so the schema already explains every field. The description adds no parameter-level detail beyond what is there, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (enrich) and resource (business lead) plus the mechanism (crawling the official website) and enumerates the concrete outputs. It is clearly distinct from generic extract_contacts/crawl_website siblings, though it never names an alternative tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use guidance, no prerequisites, and no mention of when to prefer this over crawl_website, extract_contacts, or batch_extract_web_content. The only selection hint is the 'deeply' qualifier and the mode enum, which lives in the schema, not the description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

discover_local_leadsA

Discover local businesses, establishments, and B2B prospects using the Google Maps engine. Extracts title, category, formatted address, telephone, WhatsApp link, rating, review count, coordinates, and website URL.

ParametersJSON Schema
NameRequiredDescriptionDefault
langNoGoogle Maps language code (default: 'pt')pt
limitNoMaximum number of places to retrieve (default: 20)
queryYesTarget business niche or search term (e.g. 'dental clinic', 'vet clinic', 'coworking')
locationNoCity, neighborhood, or state (e.g. 'Bristol', 'Goiania')

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full disclosure burden. It usefully reveals that results come from Google Maps and enumerates the extracted fields, but says nothing about rate limits, quotas, permissions, or whether the operation is idempotent/read-only, and the schema's 1-100 'limit' cap is never mentioned in prose.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences: the first front-loads what the tool does and its engine, the second enumerates return fields. With no output schema present, that second sentence earns its place rather than being filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the field list is a valuable completeness contribution, and the required query parameter is covered by the schema. Gaps remain around sibling differentiation and result-volume expectations, but the core contract is conveyed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all four parameters (query, location, lang, limit) plus defaults and bounds are already documented. The description adds no syntax, format, or interplay guidance (e.g. how location combines with query), so it sits at the baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific verb ('Discover') and resource (local businesses/establishments/B2B prospects) and even names the underlying engine (Google Maps), so the capability is unambiguous. It does not, however, distinguish itself from the very similar sibling 'search_places', which appears to overlap heavily.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'B2B prospects' hints at the lead-generation use case, so usage is implied rather than stated. There is no explicit when-to-use, when-not-to-use, or reference to the near-identical 'search_places' sibling that an agent would need to choose between them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_contactsA

Extract verified contact information (emails, phone numbers, WhatsApp links, and social media profiles) from a website and its contact subpages. Drastically saves tokens compared to feeding the entire raw DOM to the LLM.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesTarget website URL to extract contact details from.
deep_scanNoWhether to scan linked contact and about subpages (/contato, /about) for emails (default: true).

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses meaningful behavior: results are 'verified', extraction follows links into contact/about subpages, and it returns compact structured data rather than raw DOM. It omits failure modes, JS-rendered page handling, timeouts, and whether any auth or rate limits apply.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the verb and extracted resource before any justification. No filler; the second sentence functions as a routing signal rather than padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema and no annotations, so the description does well by naming the returned field categories. It is nearly complete for a two-parameter read tool, with only minor gaps around output shape and behavior on dynamic sites.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both 'url' and 'deep_scan' are already documented, including the subpage examples and the default. The description only restates that subpages are scanned and adds no syntax or format detail beyond the schema, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (extract), a specific resource (contact information), and enumerates the exact output types (emails, phone numbers, WhatsApp links, social profiles). This enumeration clearly separates it from siblings like extract_web_content and extract_structured_data, which target generic or arbitrary content.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The closing sentence implies a use case by contrasting the tool with 'feeding the entire raw DOM to the LLM', which hints at when to prefer it. However, no sibling tools are named, no prerequisites or exclusions are given, and nothing tells the agent when NOT to use it (e.g. when only page text is needed).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_structured_dataA

Extract structured JSON โ€” page metadata (Open Graph, canonical URL, author, dates) and/or a custom CSS-selector schema โ€” from a website.

What it does Fetches a URL and returns a structured object. With include_metadata=true it reads Open Graph tags (og:title, og:description, og:image), the canonical URL, author, and publication/modified dates. You can also pass a schema (property name -> CSS selector) to pull specific fields (e.g. {"title": "h1", "price": ".price", "rating": "span.rating"}) into a typed object.

How to use

  • Use for product listings, news/metadata, schema-driven extraction, and any time you need fields instead of prose.

  • schema keys become the JSON keys; values are CSS selectors. Leave schema empty to get metadata only.

  • include_metadata (default true) adds the Open Graph / author / date block. Set false for a lean result.

  • mode/css_selector/headers/cookies/proxy behave exactly as in extract_web_content.

Returns JSON object with the requested fields (and/or metadata). Missing selectors are omitted rather than failing the whole call. Non-http or private URLs fail with isError: true.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesTarget website URL to extract content and metadata from.
modeNoCrawl mode: 'fast' (native fetch) or 'deep' (Playwright Chromium).fast
proxyNoOptional HTTP/SOCKS5 proxy URL.
schemaNoOptional key-value map of property names to CSS selectors (e.g. { title: 'h1', price: '.price' }).
cookiesNoOptional custom HTTP cookies key-value dictionary.
headersNoOptional custom HTTP request headers key-value dictionary.
max_retriesNoMaximum retry attempts (default: 3).
css_selectorNoOptional CSS selector to scope content before processing.
include_metadataNoWhether to extract Open Graph tags, canonical URL, author, and date metadata (default: true).

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations the description carries the full burden, and it delivers real behavioral detail: missing selectors are omitted rather than failing the call, and non-http/private URLs return isError: true. It adds defaults for include_metadata but does not cover auth, rate limits, or cost/latency differences between fast and deep mode.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the one-line summary, then organizes detail under 'What it does', 'How to use', and 'Returns' headings, which makes scanning easy for a 9-parameter tool. It is slightly verbose with the section headers and parenthetical enumerations, but every line carries actionable content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description explains the return shape, partial-failure behavior, and error signal, and documents the defaults for include_metadata and the schema parameter. An agent has everything needed to invoke and interpret results without consulting other tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3, but the description adds genuine semantic value: schema keys become the JSON keys and values are CSS selectors (with a concrete example), include_metadata=true adds the OG/author/date block and false yields a lean result. mode/css_selector/headers/cookies/proxy are punted to a sibling tool's semantics, which is a mild indirection.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Extract structured JSON โ€” page metadata ... and/or a custom CSS-selector schema โ€” from a website.' It also implicitly distinguishes itself from the prose-returning sibling by framing output as 'fields instead of prose.' An agent can tell what it does and how it differs from extract_web_content without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete use cases ('product listings, news/metadata, schema-driven extraction') and a selection rule ('any time you need fields instead of prose'), plus a clarifying case ('Leave schema empty to get metadata only'). It stops short of explicitly naming extract_web_content as the alternative for prose, so the routing is implied rather than spelled out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_web_content

Extract token-optimized, clean Markdown content from a target website for LLM consumption.

What it does Fetches a URL, strips HTML bloat (scripts, styles, inline SVGs, tracking tags, navigation, footers, redundant forms) and returns readable Markdown โ€” cutting token usage by ~73% to 90% versus the raw page.

How to use

  • Use for any web page, article, or docs you want as LLM-ready text.

  • mode="fast" (default): native HTTP GET. mode="deep": headless Chrome with stealth fingerprint for JS-heavy or Cloudflare/Turnstile-protected pages.

  • css_selector scopes to a single element (e.g. "main", "article", "#content").

  • Pass headers/cookies for gated or geo-restricted content. Providing any of headers/cookies/proxy bypasses the SQLite cache.

  • No auth required. Respects robots/rate-limit via retry backoff.

Returns Markdown text. Cache hits are prefixed with a NOTE marker. Unsupported URLs (non-http, private IP, SSRF) fail with isError: true and a reason.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesTarget website URL to extract content from.
modeNoCrawl mode: 'fast' (native HTTP fetch) or 'deep' (headless Playwright browser with JS execution).fast
proxyNoOptional HTTP/SOCKS5 proxy URL (e.g. 'http://proxy.example.com:8080').
actionsNoOptional pre-crawl automation actions for mode='deep' (e.g. click cookie consent, load more, scrolling).
cookiesNoOptional custom HTTP cookies key-value dictionary.
headersNoOptional custom HTTP request headers key-value dictionary.
image_modeNoImage handling: 'ignore' (omit images), 'alt_only' (> ๐Ÿ–ผ๏ธ *[Imagem: alt]*), or 'markdown' (![alt](src)). Default: ignore.ignore
link_formatNoLink formatting: 'inline' ([text](url)), 'references' ([text][1] footnote citations at end), or 'strip' (plain text). Default: inline.inline
max_retriesNoMaximum retry attempts for transient errors or rate limits (default: 3).
css_selectorNoOptional CSS selector to scope content extraction to a specific HTML node (e.g. 'main', 'article').
map_websiteA

Discover the architecture and page inventory of a website domain via sitemaps, robots.txt, and link graphs. Returns a list of URLs with source and modification metadata. Useful for exploring a site before targeted crawling.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesTarget website URL or domain root (e.g. 'https://example.com').
max_linksNoMaximum number of links to discover (default: 100).

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses the return shape ('list of URLs with source and modification metadata'), which is useful, but says nothing about read-only nature, network/rate-limit behavior, robots.txt compliance, or failure modes for a network-fetching tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences: the purpose and mechanism lead, and the return/usage context follows. Every clause carries information with no padding or restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, and the description does cover the return contents, which is the main gap it needed to fill. Remaining omissions are behavioral (safety/rate/permission context) rather than structural, so it is largely complete for a 2-parameter discovery tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both parameters (url, max_links) are already fully documented in the schema, including the default and 1-500 range. The description adds no parameter-level detail beyond that, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Discover the architecture and page inventory of a website domain') plus the mechanisms used (sitemaps, robots.txt, link graphs). This clearly separates it from extraction siblings, though it does not name the closest sibling (crawl_website) explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Useful for exploring a site before targeted crawling' gives a concrete usage context that positions it ahead of crawl_website/extract_web_content. It stops short of naming an alternative or stating when not to use it, so it is strong context without exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_placesA

Search for local businesses, places, and establishments on Google Maps, extracting structured entity data (title, address, phone, website, coordinates, rating).

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMaximum number of places to retrieve (default: 5, max: 20).
queryYesSearch query (e.g. 'restaurantes em Pinheiros Sao Paulo', 'dentists in Bristol').

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It discloses the data source (Google Maps) and the returned entity fields, which is useful context, but it omits read-only confirmation, auth requirements, rate limits, pagination, and result-limit behavior beyond what the schema states.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, front-loaded sentence with no wasted words. The verb and resource come first, and the extracted field list is compact enough to remain useful without bloating the definition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description appropriately compensates by naming the returned fields (title, address, phone, website, coordinates, rating). For a simple two-parameter search tool, this is nearly complete, though it still leaves usage relative to local-lead siblings unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both the query syntax examples and the limit default/max are already fully documented in the input schema. The description adds no additional parameter meaning, which meets the baseline expectation when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Search) and resource (local businesses, places, establishments on Google Maps) and lists the structured fields returned. It is clear enough to separate from web-content and crawling siblings, but it does not explicitly differentiate itself from the potentially overlapping discover_local_leads or extract_contacts tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the verb 'Search' and the Google Maps/local-business scope, so an agent can infer when it applies. However, there is no explicit when-to-use guidance, no exclusions, and no named alternative among the sibling tools when local lead discovery or contact extraction might be preferred.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv1.1.2
    • Changedcrawl_website6 fields changed
      • changedInput schema / properties / exclude_patterns / description
        Previous value: -"Regex patterns for URLs to ignore (e.g. ['/login', '/tags/'])."New value: +"Substring patterns (`*` = wildcard) for URLs to ignore (e.g. ['/login', '/tags/']). Max 10."
      • addedInput schema / properties / exclude_patterns / items / maxLength
        Added value: +200
      • addedInput schema / properties / exclude_patterns / maxItems
        Added value: +10
      • changedInput schema / properties / include_patterns / description
        Previous value: -"Regex patterns that candidate URLs must match (e.g. ['/docs/'])."New value: +"Substring patterns (`*` = wildcard) that candidate URLs must match (e.g. ['/docs/']). Max 10."
      • addedInput schema / properties / include_patterns / items / maxLength
        Added value: +200
      • addedInput schema / properties / include_patterns / maxItems
        Added value: +10
    • Changedextract_web_content11 fields changed
      • changedInput schema / properties / actions / items / properties / milliseconds / description
        Previous value: -"Milliseconds to wait"New value: +"Milliseconds to wait (max 10000)"
      • addedInput schema / properties / actions / items / properties / milliseconds / maximum
        Added value: +10000
      • addedInput schema / properties / actions / items / properties / milliseconds / minimum
        Added value: +0
      • changedInput schema / properties / actions / items / properties / milliseconds / type
        Previous value: -"number"New value: +"integer"
      • changedInput schema / properties / actions / items / properties / pixels / description
        Previous value: -"Pixels to scroll"New value: +"Pixels to scroll (max 20000)"
      • addedInput schema / properties / actions / items / properties / pixels / maximum
        Added value: +20000
      • addedInput schema / properties / actions / items / properties / pixels / minimum
        Added value: +0
      • changedInput schema / properties / actions / items / properties / pixels / type
        Previous value: -"number"New value: +"integer"
      • addedInput schema / properties / actions / items / properties / selector / maxLength
        Added value: +500
      • addedInput schema / properties / actions / items / properties / value / maxLength
        Added value: +5000
      • addedInput schema / properties / actions / maxItems
        Added value: +20
  2. 8 tool updatesv1.1.1
    • Changedbatch_extract_web_content1 field changed
      • changedInput schema / properties / urls / description
        Previous value: -"Array of target website URLs to extract."New value: +"Array of target website URLs to extract (1-20)."
    • Addedcrawl_website
    • Addeddeep_enrich_lead
    • Addeddiscover_local_leads
    • Addedextract_contacts
    • Changedextract_web_content4 fields changed
      • addedInput schema / properties / actions
        Added value: +{
        +  "description": "Optional pre-crawl automation actions for mode='deep' (e.g. click cookie consent, load more, scrolling).",
        +  "items": {
        +    "additionalProperties": false,
        +    "properties": {
        +      "direction": {
        +        "description": "Scroll direction",
        +        "enum": [
        +          "up",
        +          "down"
        +        ],
        +        "type": "string"
        +      },
        +      "milliseconds": {
        +        "description": "Milliseconds to wait",
        +        "type": "number"
        +      },
        +      "pixels": {
        +        "description": "Pixels to scroll",
        +        "type": "number"
        +      },
        +      "selector": {
        +        "description": "CSS selector for click/fill",
        +        "type": "string"
        +      },
        +      "type": {
        +        "description": "Action type",
        +        "enum": [
        +          "click",
        +          "scroll",
        +          "wait",
        +          "fill"
        +        ],
        +        "type": "string"
        +      },
        +      "value": {
        +        "description": "Text value to fill into input",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "type"
        +    ],
        +    "type": "object"
        +  },
        +  "type": "array"
        +}
      • changedInput schema / properties / css_selector / description
        Previous value: -"Optional CSS selector to scope content extraction to a specific HTML node."New value: +"Optional CSS selector to scope content extraction to a specific HTML node (e.g. 'main', 'article')."
      • addedInput schema / properties / image_mode
        Added value: +{
        +  "default": "ignore",
        +  "description": "Image handling: 'ignore' (omit images), 'alt_only' (> ๐Ÿ–ผ๏ธ *[Imagem: alt]*), or 'markdown' (![alt](src)). Default: ignore.",
        +  "enum": [
        +    "ignore",
        +    "alt_only",
        +    "markdown"
        +  ],
        +  "type": "string"
        +}
      • addedInput schema / properties / link_format
        Added value: +{
        +  "default": "inline",
        +  "description": "Link formatting: 'inline' ([text](url)), 'references' ([text][1] footnote citations at end), or 'strip' (plain text). Default: inline.",
        +  "enum": [
        +    "inline",
        +    "references",
        +    "strip"
        +  ],
        +  "type": "string"
        +}
    • Addedmap_website
    • Addedsearch_places
  3. 3 tool updatesv0.1.0
    • First observedbatch_extract_web_content
    • First observedextract_structured_data
    • First observedextract_web_content

TDQS

A3.6/5.0

Scored across 9 tools

Disambiguation3/5

Most tools have distinct roles, but search_places and discover_local_leads both appear to perform Google Maps business discovery with nearly identical output fields, making selection ambiguous. deep_enrich_lead also overlaps with extract_contacts and crawl_website, though its lead-focused summary provides some differentiation.

Naming Consistency4/5

All names use snake_case and generally follow a verb-first pattern, which is readable and predictable. Minor deviations like batch_extract_web_content and deep_enrich_lead are understandable but slightly less consistent with the core verb_noun convention.

Tool Count5/5

Nine tools is well-scoped for a web crawling, extraction, and lead-generation server. Each tool covers a meaningful capability without excessive fragmentation.

Completeness4/5

The surface covers discovery, single and batch content extraction, structured data, recursive crawling, contact extraction, local search, lead discovery, and enrichment. A few adjacent capabilities such as generic web search or document/PDF extraction are missing, but core crawler and lead workflows are well represented.

Maintenance

ActivityActive
ResponsivenessUnresponsive

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    D
    maintenance
    A Model Context Protocol server that enables web scraping, crawling, and content extraction capabilities through integration with Firecrawl.
    8
    133,864 npm
    2
    MIT
  • A
    license
    A
    quality
    A
    maintenance
    Web content extraction for AI agents. 10 tools: scrape, crawl, map, batch, extract, summarize, diff, brand, search, research. Uses TLS fingerprinting to bypass anti-bot without a headless browser. Outputs LLM-optimized markdown with 67% fewer tokens than raw HTML.
    10
    2,371
    AGPL 3.0
  • A
    license
    Not graded
    quality
    D
    maintenance
    Open-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.
    2
    AGPL 3.0