Skip to main content
Glama

🕷️ LookaCrawler

Alternativa local gratuita, de código abierto y eficiente en tokens a Firecrawl, con servidor nativo de Model Context Protocol (MCP) para LLMs.

License: MIT Bun TypeScript MCP Ready Github Stars Local-First


📌 ¿Por qué LookaCrawler?

El rastreo web para modelos de lenguaje de gran tamaño (LLM) está roto por defecto: las páginas web modernas contienen una enorme cantidad de HTML innecesario (scripts, píxeles de seguimiento, divs anidados, cabeceras de navegación, hojas de estilo), lo que supone miles de tokens desperdiciados por página.

LookaCrawler es un rastreador local de código abierto y optimizado en tokens que elimina más del 73 % y hasta el 90 % del exceso de HTML, extrae Markdown limpio, sortea barreras anti-bot con controladores Playwright sigilosos y expone un servir nativo de Model Context Protocol (MCP) listo para Claude Desktop, Cursor y Antigravity.


Related MCP server: TheCrawler

🥊 Comparación: LookaCrawler vs Alternativas

Característica

🕷️ LookaCrawler

🔥 Firecrawl (Cloud)

⚡ Jina Reader

Precio / Coste

$0.00 (100 % gratuito y de código abierto)

$16 a $99+/mes

API con límite de peticiones

Reducción de tokens

Poda agresiva de >73 % a 90 %

Markdown estándar

Markdown básico

Privacidad de datos

100 % local (cero telemetría)

Proveedor en la nube

API en la nube

Integración MCP

Servidor nativo de 1 clic (stdio y SSE)

Envoltorio de la comunidad

Ninguna

Sigilo y antibots

Chrome real + huella de sigilo

Proxies en la nube

Cabeceras básicas

Caché SQLite local

Integrada (caché TTL de 24 h)

Redis / complemento de pago

Ninguna

Soporte de SPA en JS

Playwright + pool de Chrome

Headless en la nube

Headless


️ Características principales

  • Priridad a la economía de tokens: poda automáticamente scripts, estilos, SVGs en línea, etiquetas de seguimiento, navegaciones, pies de página y formularios redundantes antes de pasar el Markdown a tu LLM.

  • Motor de rastreo híbrido dual:

    • fast: HTTP GET nativo ultrarápido con backoff. Escala automáticamente a deep si se encuentra un desafío anti-bot.

    • deep: motor Playwright headless que lanza Googla Chrome real con parches de sigilo (navigator.webdriver eliminado, WebGL falsificado y fugas de CDP eliminadas) para rastrear de forma transparente páginas protegidas por Cloudflare/Turnstile.

  • Protocolo nativo de servidor MCP: transportes JSON-RPC 2.0 stdio y SSE listos para Claude Desktop, Cursor, Windsurf y Antigravity.

  • Caché SQLite local: almacena el Markdown extraído en crawler_cache.sqlite para eliminar llamadas de red duplicadas.

  • Extracción de JSON estructurado y metadatos: extrae etiquetas Open Graph (og:title, og:description), fechas de publicación, URLs canónicas y selectores CSS personalizados.


🔌 Configuración de MCP en 1 clic (Claude Desktop y Cursor)

Añade LookaCrawler a tu claude_desktop_config.json o a la configuración de MCP de Cursor:

{
  "mcpServers": {
    "lookacrawler": {
      "command": "bun",
      "args": ["run", "/absolute/path/to/lookacrawler/index.ts"]
    }
  }
}

Ahora puedes pedir a Claude o a Cursor:

"Rastrea https://example.com/docs y extrae la documentación de la API con LookaCrawler."


📦 Inicio rápido y uso de la CLI

1. Instalación

Requiere Bun 1.1+ (o Node.js 20+):

# Clone the repository
git clone https://github.com/lucasmartins-ai/lookacrawler.git
cd lookacrawler

# Install dependencies
bun install

2. Comandos CLI

# Single URL fast Markdown extraction
bun run cli.ts extract https://news.ycombinator.com --mode fast

# Headless Playwright deep extraction with CSS selector target
bun run cli.ts extract https://example.com --mode deep --selector "main" --json

# Batch concurrent multi-URL crawling
bun run cli.ts batch https://site1.com https://site2.com --concurrency 4

# Structured JSON schema extraction
bun run cli.ts structured https://example.com --schema '{"title":"h1","links":"a"}'

# Start MCP Server via SSE on port 3000
bun run cli.ts serve --transport sse --port 3000

3. Despliegue con Docker

# Build and run Docker container
docker build -t lookacrawler .
docker run -p 3000:3000 lookacrawler

🧪 Arquitectura y pruebas

Incoming URL ──► [Local SQLite Cache Check] ──(Hit)──► Return Cached Markdown
                       │ (Miss)
                       ▼
            [Fast HTTP GET Request] ──(Blocked?)──► [Auto-Escalate to Deep Stealth]
                       │                                      │
                       ▼                                      ▼
            [HTML DOM Tree Parser] ◄──────────────────────────┘
                       │
                       ▼
       [Aggressive Token Noise Pruner]
       (Strips SVG, Nav, Ads, Tracking, CSS, JS)
                       │
                       ▼
         [Mozilla Readability Engine]
                       │
                       ▼
         [Turndown Markdown Converter] ──► Return Clean LLM Markdown

Ejecuta la suite de pruebas:

bun test

⭐ Deja una estrella y apoya

Si LookaCrawler te ahorra costes de API y tokens:

  • ⭐ Deja una estrella en este repositorio para ayudar a otros desarroladores a encontrarlo.

  • 💡 Abre un Issue / PR para nuevas técnicas de evasión de sigilo o funciones del rastreador.


Desarrollado por LookADev

lookacrawler está desarrollado y mantenido por LookADev, un estudio de ingeniería especializado en agentes de IA, arquitectura web y optimización de tokens.

Inicia un proyecto → lookadev.com · Correo electrónico: lucas@lookadev.com

📄 Licencia

Software de código abierto licenciado bajo la Licencia MIT.

Available Tools

9 tools
batch_extract_web_contentA

Batch extract token-optimized Markdown from multiple website URLs concurrently, with aggregate token statistics.

What it does Runs the same fast/deep pipeline as extract_web_content across an array of up to 20 URLs in parallel, then returns per-URL results plus a summary (urls attempted, succeeded, failed, total tokens, and per-page token counts).

How to use

  • Use when you need many pages at once — docs, a list of product pages, a set of sources to compare.

  • mode applies to every URL. concurrency (1-10) caps parallel workers; keep it modest on fast machines / low bandwidth.

  • css_selector is applied to all URLs. headers/cookies/proxy apply globally to every request.

  • Each URL is independent: a single failure does not abort the batch (results carry an "error" field for that URL).

Returns Structured JSON with per-url markdown/tokens/error and aggregate totals. Incomplete or empty batches still return a valid result object.

NameRequiredDescriptionDefault
modeNoCrawl mode: 'fast' (native HTTP fetch) or 'deep' (headless Playwright browser).fast
urlsYesArray of target website URLs to extract (1-20).
proxyNoOptional HTTP/SOCKS5 proxy URL.
cookiesNoOptional custom HTTP cookies key-value dictionary.
headersNoOptional custom HTTP request headers key-value dictionary.
concurrencyNoMaximum parallel HTTP/browser crawl worker concurrency (default: 3).
max_retriesNoMaximum retry attempts per URL (default: 3).
css_selectorNoOptional CSS selector to filter DOM node across all target URLs.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden, and it discharges much of it: failures are isolated per URL ('a single failure does not abort the batch'), results carry an 'error' field, and empty/incomplete batches still return a valid object. It omits auth/credential requirements, rate limits, and timeout behavior, which keeps it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded one-line summary followed by labelled 'What it does / How to use / Returns' sections; every bullet adds operational value. The section headings cost a little space but the content is dense and non-repetitive, so it stays efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema AND no annotations, the description must cover both behavior and returns, and it does explain the return shape (per-URL markdown/tokens/error plus aggregate totals). It is complete enough to call correctly, though it leaves prerequisite/auth and rate-limit concerns unstated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3, but the description adds cross-parameter semantics the schema cannot express: mode/css_selector/headers/cookies/proxy apply to every URL in the array, concurrency (1-10) caps parallel workers with advice to keep it modest, and each URL is independent. That is real meaning beyond the per-parameter docs.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource+scope: 'Batch extract token-optimized Markdown from multiple website URLs concurrently, with aggregate token statistics.' It explicitly relates itself to the sibling extract_web_content as the same pipeline applied to an array, so an agent can tell the batch tool from the single-URL tool without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear context: 'Use when you need many pages at once — docs, a list of product pages, a set of sources to compare,' and explains how global options (mode, css_selector, headers/cookies/proxy) apply across all URLs, plus concurrency guidance. It does not state when NOT to use it (e.g., 'for a single URL use extract_web_content'), so routing is implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_websiteB

Recursively crawl an entire website or documentation section with depth limits, URL pattern filtering, and token accounting.

NameRequiredDescriptionDefault
modeNoCrawl mode ('fast' native fetch or 'deep' Playwright browser).fast
max_depthNoMaximum link traversal depth (default: 2).
max_pagesNoMaximum total pages to crawl (default: 5, max: 20).
start_urlYesRoot URL to start the recursive crawl from.
image_modeNoFormatting of images.ignore
link_formatNoFormatting of extracted hyperlinks.inline
exclude_patternsNoSubstring patterns (`*` = wildcard) for URLs to ignore (e.g. ['/login', '/tags/']). Max 10.
include_patternsNoSubstring patterns (`*` = wildcard) that candidate URLs must match (e.g. ['/docs/']). Max 10.

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden. It hints at depth control and 'token accounting' but says nothing about read-only safety, robots.txt/rate-limit behavior, timeouts, or whether the crawl is idempotent, which are the traits an agent needs when firing a recursive multi-page fetch.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense sentence with the core action and its three control mechanisms front-loaded, no filler or repetition. It is efficient, though arguably compressed enough that it omits needed detail rather than being maximally informative per word.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 8-parameter recursive crawler with no annotations and no output schema, the description is thin: it does not explain what the crawl returns, how pagination/max_pages interacts with max_depth, or how token accounting surfaces. An agent has enough to start but not enough to predict behavior or cost.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter including mode, max_depth, max_pages, and include/exclude patterns is already documented in the schema. The description's mention of 'depth limits' and 'URL pattern filtering' maps to those params but adds no syntax or interaction detail beyond them, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Recursively crawl an entire website or documentation section') and names the mechanisms (depth limits, URL filtering). It does not differentiate itself from siblings like map_website or extract_web_content, which the agent must distinguish without help.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Scope is implied by 'entire website or documentation section', so an agent can infer this is for broad multi-page harvesting. However, there is no explicit when-to-use or when-not-to-use guidance and no mention of the obvious alternatives (map_website for URL discovery, extract_web_content for a single page).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

deep_enrich_leadB

Deeply enrich a business lead by crawling its official website with LookaCrawler's token-optimized engine. Extracts verified emails, social media links (Instagram, LinkedIn), decision makers / founders, and generates a concise Markdown summary (<300 tokens) for LLMs.

NameRequiredDescriptionDefault
modeNoCrawl mode: 'fast' (HTTP) or 'deep' (stealth Playwright)fast
phoneNoKnown phone number
titleYesCompany or establishment name
websiteYesCompany official website URL to crawl
categoryNoCompany category or niche
whatsappNoKnown WhatsApp link
image_modeNoImage mode: alt_only avoids CDN bloatalt_only
link_formatNoLink formatting: references footnotes for token savingsreferences

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses output artifacts (verified emails, social links, founders, a <300-token Markdown summary) and the token-optimized engine, which is useful behavioral context. But it omits cost/latency implications, rate limits, auth requirements, or what happens when the crawl fails.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tightly written sentences that front-load the action and then the outputs. No filler, no restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 8-parameter, no-output-schema tool, the description covers the essential return content (emails, socials, decision makers, markdown summary) so an agent knows what comes back. Missing only failure/edge-case behavior and any alternative-tool routing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% with 8 documented parameters and three enums, so the schema already explains every field. The description adds no parameter-level detail beyond what is there, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (enrich) and resource (business lead) plus the mechanism (crawling the official website) and enumerates the concrete outputs. It is clearly distinct from generic extract_contacts/crawl_website siblings, though it never names an alternative tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use guidance, no prerequisites, and no mention of when to prefer this over crawl_website, extract_contacts, or batch_extract_web_content. The only selection hint is the 'deeply' qualifier and the mode enum, which lives in the schema, not the description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

discover_local_leadsA

Discover local businesses, establishments, and B2B prospects using the Google Maps engine. Extracts title, category, formatted address, telephone, WhatsApp link, rating, review count, coordinates, and website URL.

NameRequiredDescriptionDefault
langNoGoogle Maps language code (default: 'pt')pt
limitNoMaximum number of places to retrieve (default: 20)
queryYesTarget business niche or search term (e.g. 'dental clinic', 'vet clinic', 'coworking')
locationNoCity, neighborhood, or state (e.g. 'Bristol', 'Goiania')

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full disclosure burden. It usefully reveals that results come from Google Maps and enumerates the extracted fields, but says nothing about rate limits, quotas, permissions, or whether the operation is idempotent/read-only, and the schema's 1-100 'limit' cap is never mentioned in prose.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences: the first front-loads what the tool does and its engine, the second enumerates return fields. With no output schema present, that second sentence earns its place rather than being filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the field list is a valuable completeness contribution, and the required query parameter is covered by the schema. Gaps remain around sibling differentiation and result-volume expectations, but the core contract is conveyed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all four parameters (query, location, lang, limit) plus defaults and bounds are already documented. The description adds no syntax, format, or interplay guidance (e.g. how location combines with query), so it sits at the baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific verb ('Discover') and resource (local businesses/establishments/B2B prospects) and even names the underlying engine (Google Maps), so the capability is unambiguous. It does not, however, distinguish itself from the very similar sibling 'search_places', which appears to overlap heavily.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'B2B prospects' hints at the lead-generation use case, so usage is implied rather than stated. There is no explicit when-to-use, when-not-to-use, or reference to the near-identical 'search_places' sibling that an agent would need to choose between them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_contactsA

Extract verified contact information (emails, phone numbers, WhatsApp links, and social media profiles) from a website and its contact subpages. Drastically saves tokens compared to feeding the entire raw DOM to the LLM.

NameRequiredDescriptionDefault
urlYesTarget website URL to extract contact details from.
deep_scanNoWhether to scan linked contact and about subpages (/contato, /about) for emails (default: true).

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses meaningful behavior: results are 'verified', extraction follows links into contact/about subpages, and it returns compact structured data rather than raw DOM. It omits failure modes, JS-rendered page handling, timeouts, and whether any auth or rate limits apply.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the verb and extracted resource before any justification. No filler; the second sentence functions as a routing signal rather than padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema and no annotations, so the description does well by naming the returned field categories. It is nearly complete for a two-parameter read tool, with only minor gaps around output shape and behavior on dynamic sites.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both 'url' and 'deep_scan' are already documented, including the subpage examples and the default. The description only restates that subpages are scanned and adds no syntax or format detail beyond the schema, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (extract), a specific resource (contact information), and enumerates the exact output types (emails, phone numbers, WhatsApp links, social profiles). This enumeration clearly separates it from siblings like extract_web_content and extract_structured_data, which target generic or arbitrary content.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The closing sentence implies a use case by contrasting the tool with 'feeding the entire raw DOM to the LLM', which hints at when to prefer it. However, no sibling tools are named, no prerequisites or exclusions are given, and nothing tells the agent when NOT to use it (e.g. when only page text is needed).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_structured_dataA

Extract structured JSON — page metadata (Open Graph, canonical URL, author, dates) and/or a custom CSS-selector schema — from a website.

What it does Fetches a URL and returns a structured object. With include_metadata=true it reads Open Graph tags (og:title, og:description, og:image), the canonical URL, author, and publication/modified dates. You can also pass a schema (property name -> CSS selector) to pull specific fields (e.g. {"title": "h1", "price": ".price", "rating": "span.rating"}) into a typed object.

How to use

  • Use for product listings, news/metadata, schema-driven extraction, and any time you need fields instead of prose.

  • schema keys become the JSON keys; values are CSS selectors. Leave schema empty to get metadata only.

  • include_metadata (default true) adds the Open Graph / author / date block. Set false for a lean result.

  • mode/css_selector/headers/cookies/proxy behave exactly as in extract_web_content.

Returns JSON object with the requested fields (and/or metadata). Missing selectors are omitted rather than failing the whole call. Non-http or private URLs fail with isError: true.

NameRequiredDescriptionDefault
urlYesTarget website URL to extract content and metadata from.
modeNoCrawl mode: 'fast' (native fetch) or 'deep' (Playwright Chromium).fast
proxyNoOptional HTTP/SOCKS5 proxy URL.
schemaNoOptional key-value map of property names to CSS selectors (e.g. { title: 'h1', price: '.price' }).
cookiesNoOptional custom HTTP cookies key-value dictionary.
headersNoOptional custom HTTP request headers key-value dictionary.
max_retriesNoMaximum retry attempts (default: 3).
css_selectorNoOptional CSS selector to scope content before processing.
include_metadataNoWhether to extract Open Graph tags, canonical URL, author, and date metadata (default: true).

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations the description carries the full burden, and it delivers real behavioral detail: missing selectors are omitted rather than failing the call, and non-http/private URLs return isError: true. It adds defaults for include_metadata but does not cover auth, rate limits, or cost/latency differences between fast and deep mode.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the one-line summary, then organizes detail under 'What it does', 'How to use', and 'Returns' headings, which makes scanning easy for a 9-parameter tool. It is slightly verbose with the section headers and parenthetical enumerations, but every line carries actionable content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description explains the return shape, partial-failure behavior, and error signal, and documents the defaults for include_metadata and the schema parameter. An agent has everything needed to invoke and interpret results without consulting other tools.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3, but the description adds genuine semantic value: schema keys become the JSON keys and values are CSS selectors (with a concrete example), include_metadata=true adds the OG/author/date block and false yields a lean result. mode/css_selector/headers/cookies/proxy are punted to a sibling tool's semantics, which is a mild indirection.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Extract structured JSON — page metadata ... and/or a custom CSS-selector schema — from a website.' It also implicitly distinguishes itself from the prose-returning sibling by framing output as 'fields instead of prose.' An agent can tell what it does and how it differs from extract_web_content without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete use cases ('product listings, news/metadata, schema-driven extraction') and a selection rule ('any time you need fields instead of prose'), plus a clarifying case ('Leave schema empty to get metadata only'). It stops short of explicitly naming extract_web_content as the alternative for prose, so the routing is implied rather than spelled out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_web_contentA

Extract token-optimized, clean Markdown content from a target website for LLM consumption.

What it does Fetches a URL, strips HTML bloat (scripts, styles, inline SVGs, tracking tags, navigation, footers, redundant forms) and returns readable Markdown — cutting token usage by ~73% to 90% versus the raw page.

How to use

  • Use for any web page, article, or docs you want as LLM-ready text.

  • mode="fast" (default): native HTTP GET. mode="deep": headless Chrome with stealth fingerprint for JS-heavy or Cloudflare/Turnstile-protected pages.

  • css_selector scopes to a single element (e.g. "main", "article", "#content").

  • Pass headers/cookies for gated or geo-restricted content. Providing any of headers/cookies/proxy bypasses the SQLite cache.

  • No auth required. Respects robots/rate-limit via retry backoff.

Returns Markdown text. Cache hits are prefixed with a NOTE marker. Unsupported URLs (non-http, private IP, SSRF) fail with isError: true and a reason.

NameRequiredDescriptionDefault
urlYesTarget website URL to extract content from.
modeNoCrawl mode: 'fast' (native HTTP fetch) or 'deep' (headless Playwright browser with JS execution).fast
proxyNoOptional HTTP/SOCKS5 proxy URL (e.g. 'http://proxy.example.com:8080').
actionsNoOptional pre-crawl automation actions for mode='deep' (e.g. click cookie consent, load more, scrolling).
cookiesNoOptional custom HTTP cookies key-value dictionary.
headersNoOptional custom HTTP request headers key-value dictionary.
image_modeNoImage handling: 'ignore' (omit images), 'alt_only' (> 🖼️ *[Imagem: alt]*), or 'markdown' (![alt](src)). Default: ignore.ignore
link_formatNoLink formatting: 'inline' ([text](url)), 'references' ([text][1] footnote citations at end), or 'strip' (plain text). Default: inline.inline
max_retriesNoMaximum retry attempts for transient errors or rate limits (default: 3).
css_selectorNoOptional CSS selector to scope content extraction to a specific HTML node (e.g. 'main', 'article').

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden, and it delivers: caching behavior (header/cookie/proxy bypasses SQLite cache), auth-free access, robots/rate-limit respect via retry backoff, and failure semantics (isError: true for unsupported/SSRF URLs). It omits any note on rate limits' magnitude, output size caps, or timeout behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded one-line purpose, then clearly labeled What it does / How to use / Returns sections. Every sentence adds actionable information with zero filler, and it is well sized for a 10-param tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 10-parameter tool with nested actions, 100% schema coverage, and no output schema, the description covers purpose, mode selection, caching, auth, and error behavior adequately. A brief mention of actions usage or pagination/output limits would make it fully complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, setting a baseline of 3, but the description adds meaning beyond the schema: it explains mode tradeoffs (native HTTP vs stealth headless Chrome), that css_selector scopes to a single element, and that supplying headers/cookies/proxy changes caching. The nested actions array is only lightly touched via mode='deep' context.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Extract) and resource (web content), and qualifies the output precisely as token-optimized clean Markdown for LLM consumption. This distinguishes it from siblings like crawl_website and extract_structured_data by naming the output format rather than a crawl or structured-data framing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete when-to-use (any page, article, or docs) and per-parameter routing advice like mode=fast vs mode=deep for JS-heavy or Cloudflare-protected pages. It does not explicitly name sibling alternatives such as crawl_website or extract_structured_data to say when not to use this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

map_websiteA

Discover the architecture and page inventory of a website domain via sitemaps, robots.txt, and link graphs. Returns a list of URLs with source and modification metadata. Useful for exploring a site before targeted crawling.

NameRequiredDescriptionDefault
urlYesTarget website URL or domain root (e.g. 'https://example.com').
max_linksNoMaximum number of links to discover (default: 100).

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses the return shape ('list of URLs with source and modification metadata'), which is useful, but says nothing about read-only nature, network/rate-limit behavior, robots.txt compliance, or failure modes for a network-fetching tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences: the purpose and mechanism lead, and the return/usage context follows. Every clause carries information with no padding or restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, and the description does cover the return contents, which is the main gap it needed to fill. Remaining omissions are behavioral (safety/rate/permission context) rather than structural, so it is largely complete for a 2-parameter discovery tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both parameters (url, max_links) are already fully documented in the schema, including the default and 1-500 range. The description adds no parameter-level detail beyond that, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Discover the architecture and page inventory of a website domain') plus the mechanisms used (sitemaps, robots.txt, link graphs). This clearly separates it from extraction siblings, though it does not name the closest sibling (crawl_website) explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Useful for exploring a site before targeted crawling' gives a concrete usage context that positions it ahead of crawl_website/extract_web_content. It stops short of naming an alternative or stating when not to use it, so it is strong context without exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_placesA

Search for local businesses, places, and establishments on Google Maps, extracting structured entity data (title, address, phone, website, coordinates, rating).

NameRequiredDescriptionDefault
limitNoMaximum number of places to retrieve (default: 5, max: 20).
queryYesSearch query (e.g. 'restaurantes em Pinheiros Sao Paulo', 'dentists in Bristol').

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It discloses the data source (Google Maps) and the returned entity fields, which is useful context, but it omits read-only confirmation, auth requirements, rate limits, pagination, and result-limit behavior beyond what the schema states.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, front-loaded sentence with no wasted words. The verb and resource come first, and the extracted field list is compact enough to remain useful without bloating the definition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description appropriately compensates by naming the returned fields (title, address, phone, website, coordinates, rating). For a simple two-parameter search tool, this is nearly complete, though it still leaves usage relative to local-lead siblings unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both the query syntax examples and the limit default/max are already fully documented in the input schema. The description adds no additional parameter meaning, which meets the baseline expectation when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Search) and resource (local businesses, places, establishments on Google Maps) and lists the structured fields returned. It is clear enough to separate from web-content and crawling siblings, but it does not explicitly differentiate itself from the potentially overlapping discover_local_leads or extract_contacts tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the verb 'Search' and the Google Maps/local-business scope, so an agent can infer when it applies. However, there is no explicit when-to-use guidance, no exclusions, and no named alternative among the sibling tools when local lead discovery or contact extraction might be preferred.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv1.1.2
    • Changedcrawl_website6 fields changed
      • changedInput schema / properties / exclude_patterns / description
        Previous value: -"Regex patterns for URLs to ignore (e.g. ['/login', '/tags/'])."New value: +"Substring patterns (`*` = wildcard) for URLs to ignore (e.g. ['/login', '/tags/']). Max 10."
      • addedInput schema / properties / exclude_patterns / items / maxLength
        Added value: +200
      • addedInput schema / properties / exclude_patterns / maxItems
        Added value: +10
      • changedInput schema / properties / include_patterns / description
        Previous value: -"Regex patterns that candidate URLs must match (e.g. ['/docs/'])."New value: +"Substring patterns (`*` = wildcard) that candidate URLs must match (e.g. ['/docs/']). Max 10."
      • addedInput schema / properties / include_patterns / items / maxLength
        Added value: +200
      • addedInput schema / properties / include_patterns / maxItems
        Added value: +10
    • Changedextract_web_content11 fields changed
      • changedInput schema / properties / actions / items / properties / milliseconds / description
        Previous value: -"Milliseconds to wait"New value: +"Milliseconds to wait (max 10000)"
      • addedInput schema / properties / actions / items / properties / milliseconds / maximum
        Added value: +10000
      • addedInput schema / properties / actions / items / properties / milliseconds / minimum
        Added value: +0
      • changedInput schema / properties / actions / items / properties / milliseconds / type
        Previous value: -"number"New value: +"integer"
      • changedInput schema / properties / actions / items / properties / pixels / description
        Previous value: -"Pixels to scroll"New value: +"Pixels to scroll (max 20000)"
      • addedInput schema / properties / actions / items / properties / pixels / maximum
        Added value: +20000
      • addedInput schema / properties / actions / items / properties / pixels / minimum
        Added value: +0
      • changedInput schema / properties / actions / items / properties / pixels / type
        Previous value: -"number"New value: +"integer"
      • addedInput schema / properties / actions / items / properties / selector / maxLength
        Added value: +500
      • addedInput schema / properties / actions / items / properties / value / maxLength
        Added value: +5000
      • addedInput schema / properties / actions / maxItems
        Added value: +20
  2. 8 tool updatesv1.1.1
    • Changedbatch_extract_web_content1 field changed
      • changedInput schema / properties / urls / description
        Previous value: -"Array of target website URLs to extract."New value: +"Array of target website URLs to extract (1-20)."
    • Addedcrawl_website
    • Addeddeep_enrich_lead
    • Addeddiscover_local_leads
    • Addedextract_contacts
    • Changedextract_web_content4 fields changed
      • addedInput schema / properties / actions
        Added value: +{
        +  "description": "Optional pre-crawl automation actions for mode='deep' (e.g. click cookie consent, load more, scrolling).",
        +  "items": {
        +    "additionalProperties": false,
        +    "properties": {
        +      "direction": {
        +        "description": "Scroll direction",
        +        "enum": [
        +          "up",
        +          "down"
        +        ],
        +        "type": "string"
        +      },
        +      "milliseconds": {
        +        "description": "Milliseconds to wait",
        +        "type": "number"
        +      },
        +      "pixels": {
        +        "description": "Pixels to scroll",
        +        "type": "number"
        +      },
        +      "selector": {
        +        "description": "CSS selector for click/fill",
        +        "type": "string"
        +      },
        +      "type": {
        +        "description": "Action type",
        +        "enum": [
        +          "click",
        +          "scroll",
        +          "wait",
        +          "fill"
        +        ],
        +        "type": "string"
        +      },
        +      "value": {
        +        "description": "Text value to fill into input",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "type"
        +    ],
        +    "type": "object"
        +  },
        +  "type": "array"
        +}
      • changedInput schema / properties / css_selector / description
        Previous value: -"Optional CSS selector to scope content extraction to a specific HTML node."New value: +"Optional CSS selector to scope content extraction to a specific HTML node (e.g. 'main', 'article')."
      • addedInput schema / properties / image_mode
        Added value: +{
        +  "default": "ignore",
        +  "description": "Image handling: 'ignore' (omit images), 'alt_only' (> 🖼️ *[Imagem: alt]*), or 'markdown' (![alt](src)). Default: ignore.",
        +  "enum": [
        +    "ignore",
        +    "alt_only",
        +    "markdown"
        +  ],
        +  "type": "string"
        +}
      • addedInput schema / properties / link_format
        Added value: +{
        +  "default": "inline",
        +  "description": "Link formatting: 'inline' ([text](url)), 'references' ([text][1] footnote citations at end), or 'strip' (plain text). Default: inline.",
        +  "enum": [
        +    "inline",
        +    "references",
        +    "strip"
        +  ],
        +  "type": "string"
        +}
    • Addedmap_website
    • Addedsearch_places
  3. 3 tool updatesv0.1.0
    • First observedbatch_extract_web_content
    • First observedextract_structured_data
    • First observedextract_web_content

TDQS

A3.7/5.0

Scored across 9 tools

Disambiguation3/5

Most tools target distinct actions, but two near-duplicate pairs stand out: discover_local_leads vs search_places (both pull the same Google Maps entity fields), and extract_contacts vs deep_enrich_lead (both crawl a business site for emails/socials). Descriptions add some differentiation, but an agent could easily misselect within these pairs.

Naming Consistency5/5

All names are snake_case and follow a verb_noun pattern (discover_local_leads, extract_contacts, crawl_website, map_website, search_places). Minor compounds like deep_enrich_lead and batch_extract_web_content are still readable and consistent with the convention.

Tool Count5/5

Nine tools is well within the sweet spot for a crawling/scraping server, and each tool covers a distinct capability tier (discovery, extraction, enrichment, batch, site mapping). No obvious padding or redundancy-driven bloat.

Completeness4/5

The surface covers the full lead-gen lifecycle: place discovery, contact extraction, lead enrichment, content/markdown extraction, recursive crawl, batch, structured extraction, and sitemap discovery. The main gap is a lack of a status/polling or pagination/continuation operation for long-running crawls or large result sets.

Maintenance

ActivityActive
ResponsivenessSyncing

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    Open-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.
    2
    AGPL 3.0
  • A
    license
    A
    quality
    B
    maintenance
    A fault-tolerant, stealth-enabled Model Context Protocol (MCP) server for web searching and content fetching. Built for AI Agents (Cursor, Claude Code, OpenCode), it uses a stealth browser engine to fetch pages, dynamically handles SPAs/React, and converts bloat into token-optimized Markdown.
    2
    29 npm
    51 PyPI
    4
    MIT

Appeared in Searches