LookaCrawler
🕷️ LookaCrawler
Бесплатная, с открытым исходным кодом, эффективная по токенам локальная альтернатива Firecrawl с нативным сервером Model Context Protocol (MCP) для LLM.
📌 Почему LookaCrawler?
Веб-краулинг для больших языковых моделей (LLM) по умолчанию неэффективен: современные веб-страницы содержат огромный HTML-мусор (скрипты, пиксели отслеживания, вложенные div, навигационные шапки, таблицы стилей), что приводит к тысячам потраченных впустую токенов на страницу.
LookaCrawler — это локальный краулер с открытым исходным кодом, оптимизированный по токенам: он удаляет от >73% до 90% веб-мусора, извлекает чистый Markdown, обходит анти-бот барьеры с помощью стелс-драйверов Playwright и предоставляет нативный сервер Model Context Protocol (MCP), готовый для Claude Desktop, Cursor и Antigravity.
Related MCP server: TheCrawler
🥊 Сравнение: LookaCrawler и альтернативы
Возможность | 🕷️ LookaCrawler | 🔥 Firecrawl (Cloud) | ⚡ Jina Reader |
Цена / Стоимость | $0.00 (100% бесплатно, открытый исходный код) | $16–$99+ в месяц | API с ограничением запросов |
Сокращение токенов | агрессивное сокращение от >73% до 90% | Стандартный Markdown | Базовый Markdown |
Конфиденциальность данных | 100% локально (ноль телеметрии) | Облачный провайдер | Облачное API |
Интеграция MCP | Нативный сервер в 1 клик (stdio и SSE) | Обёртка от сообщества | Нет |
Стелс и анти-бот | Реальный Chrome + стелс-отпечаток | Облачные прокси | Базовые заголовки |
Локальный кэш SQLite | Встроенный (кэш с TTL 24 часа) | Redis / платный аддон | Нет |
Поддержка JS SPA | Playwright + пул Chrome | Облачный headless | Headless |
🚀 Ключевые возможности
Экономия токенов в приоритете: автоматически удаляет скрипты, стили, встроенные SVG, трекеры, навигацию, футеры и лишние формы перед передачей Markdown вашей LLM.
Гибридный движок сканирования двух режимов:
fast: сверхбыстрый нативный HTTP GET с backoff. Автоматически повышает режим доdeepпри обнаружении анти-бот проверки.deep: headless-движок Playwright, запускающий реальный Google Chrome со стелс-патчами (navigator.webdriverочищен, WebGL подменён, утечки CDP устранены) для незаметного обхода страниц, защищённых Cloudflare/Turnstile.
Нативный протокол MCP-сервера: транспорты JSON-RPC 2.0 stdio и SSE, готовые для Claude Desktop, Cursor, Windsurf и Antigravity.
Локальное кэширование SQLite: сохраняет извлечённый Markdown в
crawler_cache.sqlite, чтобы исключить повторные сетевые запросы.Извлечение структурированного JSON и метаданных: извлекает теги Open Graph (
og:title,og:description), даты публикации, канонические URL и пользовательские CSS-селекторы.
🔌 Настройка MCP в 1 клик (Claude Desktop и Cursor)
Добавьте LookaCrawler в ваш claude_desktop_config.json или в настройки MCP в Cursor:
{
"mcpServers": {
"lookacrawler": {
"command": "bun",
"args": ["run", "/absolute/path/to/lookacrawler/index.ts"]
}
}
}Теперь вы можете попросить Claude или Cursor:
«Скраулируйте https://example.com/docs и извлеките документацию API с помощью LookaCrawler.»
📦 Быстрый старт и использование CLI
1. Установка
Требуется Bun 1.1+ (или Node.js 20+):
# Clone the repository
git clone https://github.com/lucasmartins-ai/lookacrawler.git
cd lookacrawler
# Install dependencies
bun install2. Команды CLI
# Single URL fast Markdown extraction
bun run cli.ts extract https://news.ycombinator.com --mode fast
# Headless Playwright deep extraction with CSS selector target
bun run cli.ts extract https://example.com --mode deep --selector "main" --json
# Batch concurrent multi-URL crawling
bun run cli.ts batch https://site1.com https://site2.com --concurrency 4
# Structured JSON schema extraction
bun run cli.ts structured https://example.com --schema '{"title":"h1","links":"a"}'
# Start MCP Server via SSE on port 3000
bun run cli.ts serve --transport sse --port 30003. Развёртывание в Docker
# Build and run Docker container
docker build -t lookacrawler .
docker run -p 3000:3000 lookacrawler🧪 Архитектура и тестирование
Incoming URL ──► [Local SQLite Cache Check] ──(Hit)──► Return Cached Markdown
│ (Miss)
▼
[Fast HTTP GET Request] ──(Blocked?)──► [Auto-Escalate to Deep Stealth]
│ │
▼ ▼
[HTML DOM Tree Parser] ◄──────────────────────────┘
│
▼
[Aggressive Token Noise Pruner]
(Strips SVG, Nav, Ads, Tracking, CSS, JS)
│
▼
[Mozilla Readability Engine]
│
▼
[Turndown Markdown Converter] ──► Return Clean LLM MarkdownЗапустите набор тестов:
bun test⭐ Звезда и поддержка
Если LookaCrawler экономит ваши расходы на API и токены:
⭐ Поставьте звезду этому репозиторию, чтобы помочь другим разработчикам найти его!
💡 Откройте Issue / PR с новыми стелс-обходами или функциями краулера.
Создано LookADev
lookacrawler создан и поддерживается LookADev, инженерной студией, специализирующейся на AI-агентах, веб-архитектуре и оптимизации токенов.
Начните проект → lookadev.com · Эл. почта: lucas@lookadev.com
📄 Лицензия
Программное обеспечение с открытым исходным кодом, распространяемое по лицензии MIT.
Available Tools
9 toolsbatch_extract_web_contentA
Batch extract token-optimized Markdown from multiple website URLs concurrently, with aggregate token statistics.
What it does Runs the same fast/deep pipeline as extract_web_content across an array of up to 20 URLs in parallel, then returns per-URL results plus a summary (urls attempted, succeeded, failed, total tokens, and per-page token counts).
How to use
Use when you need many pages at once — docs, a list of product pages, a set of sources to compare.
mode applies to every URL. concurrency (1-10) caps parallel workers; keep it modest on fast machines / low bandwidth.
css_selector is applied to all URLs. headers/cookies/proxy apply globally to every request.
Each URL is independent: a single failure does not abort the batch (results carry an "error" field for that URL).
Returns Structured JSON with per-url markdown/tokens/error and aggregate totals. Incomplete or empty batches still return a valid result object.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | Crawl mode: 'fast' (native HTTP fetch) or 'deep' (headless Playwright browser). | fast |
| urls | Yes | Array of target website URLs to extract (1-20). | |
| proxy | No | Optional HTTP/SOCKS5 proxy URL. | |
| cookies | No | Optional custom HTTP cookies key-value dictionary. | |
| headers | No | Optional custom HTTP request headers key-value dictionary. | |
| concurrency | No | Maximum parallel HTTP/browser crawl worker concurrency (default: 3). | |
| max_retries | No | Maximum retry attempts per URL (default: 3). | |
| css_selector | No | Optional CSS selector to filter DOM node across all target URLs. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden, and it discharges much of it: failures are isolated per URL ('a single failure does not abort the batch'), results carry an 'error' field, and empty/incomplete batches still return a valid object. It omits auth/credential requirements, rate limits, and timeout behavior, which keeps it short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded one-line summary followed by labelled 'What it does / How to use / Returns' sections; every bullet adds operational value. The section headings cost a little space but the content is dense and non-repetitive, so it stays efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema AND no annotations, the description must cover both behavior and returns, and it does explain the return shape (per-URL markdown/tokens/error plus aggregate totals). It is complete enough to call correctly, though it leaves prerequisite/auth and rate-limit concerns unstated.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3, but the description adds cross-parameter semantics the schema cannot express: mode/css_selector/headers/cookies/proxy apply to every URL in the array, concurrency (1-10) caps parallel workers with advice to keep it modest, and each URL is independent. That is real meaning beyond the per-parameter docs.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource+scope: 'Batch extract token-optimized Markdown from multiple website URLs concurrently, with aggregate token statistics.' It explicitly relates itself to the sibling extract_web_content as the same pipeline applied to an array, so an agent can tell the batch tool from the single-URL tool without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear context: 'Use when you need many pages at once — docs, a list of product pages, a set of sources to compare,' and explains how global options (mode, css_selector, headers/cookies/proxy) apply across all URLs, plus concurrency guidance. It does not state when NOT to use it (e.g., 'for a single URL use extract_web_content'), so routing is implied rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_websiteB
Recursively crawl an entire website or documentation section with depth limits, URL pattern filtering, and token accounting.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | Crawl mode ('fast' native fetch or 'deep' Playwright browser). | fast |
| max_depth | No | Maximum link traversal depth (default: 2). | |
| max_pages | No | Maximum total pages to crawl (default: 5, max: 20). | |
| start_url | Yes | Root URL to start the recursive crawl from. | |
| image_mode | No | Formatting of images. | ignore |
| link_format | No | Formatting of extracted hyperlinks. | inline |
| exclude_patterns | No | Substring patterns (`*` = wildcard) for URLs to ignore (e.g. ['/login', '/tags/']). Max 10. | |
| include_patterns | No | Substring patterns (`*` = wildcard) that candidate URLs must match (e.g. ['/docs/']). Max 10. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It hints at depth control and 'token accounting' but says nothing about read-only safety, robots.txt/rate-limit behavior, timeouts, or whether the crawl is idempotent, which are the traits an agent needs when firing a recursive multi-page fetch.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single dense sentence with the core action and its three control mechanisms front-loaded, no filler or repetition. It is efficient, though arguably compressed enough that it omits needed detail rather than being maximally informative per word.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 8-parameter recursive crawler with no annotations and no output schema, the description is thin: it does not explain what the crawl returns, how pagination/max_pages interacts with max_depth, or how token accounting surfaces. An agent has enough to start but not enough to predict behavior or cost.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter including mode, max_depth, max_pages, and include/exclude patterns is already documented in the schema. The description's mention of 'depth limits' and 'URL pattern filtering' maps to those params but adds no syntax or interaction detail beyond them, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Recursively crawl an entire website or documentation section') and names the mechanisms (depth limits, URL filtering). It does not differentiate itself from siblings like map_website or extract_web_content, which the agent must distinguish without help.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Scope is implied by 'entire website or documentation section', so an agent can infer this is for broad multi-page harvesting. However, there is no explicit when-to-use or when-not-to-use guidance and no mention of the obvious alternatives (map_website for URL discovery, extract_web_content for a single page).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
deep_enrich_leadB
Deeply enrich a business lead by crawling its official website with LookaCrawler's token-optimized engine. Extracts verified emails, social media links (Instagram, LinkedIn), decision makers / founders, and generates a concise Markdown summary (<300 tokens) for LLMs.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | Crawl mode: 'fast' (HTTP) or 'deep' (stealth Playwright) | fast |
| phone | No | Known phone number | |
| title | Yes | Company or establishment name | |
| website | Yes | Company official website URL to crawl | |
| category | No | Company category or niche | |
| No | Known WhatsApp link | ||
| image_mode | No | Image mode: alt_only avoids CDN bloat | alt_only |
| link_format | No | Link formatting: references footnotes for token savings | references |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses output artifacts (verified emails, social links, founders, a <300-token Markdown summary) and the token-optimized engine, which is useful behavioral context. But it omits cost/latency implications, rate limits, auth requirements, or what happens when the crawl fails.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tightly written sentences that front-load the action and then the outputs. No filler, no restatement of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 8-parameter, no-output-schema tool, the description covers the essential return content (emails, socials, decision makers, markdown summary) so an agent knows what comes back. Missing only failure/edge-case behavior and any alternative-tool routing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with 8 documented parameters and three enums, so the schema already explains every field. The description adds no parameter-level detail beyond what is there, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (enrich) and resource (business lead) plus the mechanism (crawling the official website) and enumerates the concrete outputs. It is clearly distinct from generic extract_contacts/crawl_website siblings, though it never names an alternative tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No when-to-use guidance, no prerequisites, and no mention of when to prefer this over crawl_website, extract_contacts, or batch_extract_web_content. The only selection hint is the 'deeply' qualifier and the mode enum, which lives in the schema, not the description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
discover_local_leadsA
Discover local businesses, establishments, and B2B prospects using the Google Maps engine. Extracts title, category, formatted address, telephone, WhatsApp link, rating, review count, coordinates, and website URL.
| Name | Required | Description | Default |
|---|---|---|---|
| lang | No | Google Maps language code (default: 'pt') | pt |
| limit | No | Maximum number of places to retrieve (default: 20) | |
| query | Yes | Target business niche or search term (e.g. 'dental clinic', 'vet clinic', 'coworking') | |
| location | No | City, neighborhood, or state (e.g. 'Bristol', 'Goiania') |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full disclosure burden. It usefully reveals that results come from Google Maps and enumerates the extracted fields, but says nothing about rate limits, quotas, permissions, or whether the operation is idempotent/read-only, and the schema's 1-100 'limit' cap is never mentioned in prose.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences: the first front-loads what the tool does and its engine, the second enumerates return fields. With no output schema present, that second sentence earns its place rather than being filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the field list is a valuable completeness contribution, and the required query parameter is covered by the schema. Gaps remain around sibling differentiation and result-volume expectations, but the core contract is conveyed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all four parameters (query, location, lang, limit) plus defaults and bounds are already documented. The description adds no syntax, format, or interplay guidance (e.g. how location combines with query), so it sits at the baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Names a specific verb ('Discover') and resource (local businesses/establishments/B2B prospects) and even names the underlying engine (Google Maps), so the capability is unambiguous. It does not, however, distinguish itself from the very similar sibling 'search_places', which appears to overlap heavily.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'B2B prospects' hints at the lead-generation use case, so usage is implied rather than stated. There is no explicit when-to-use, when-not-to-use, or reference to the near-identical 'search_places' sibling that an agent would need to choose between them.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_contactsA
Extract verified contact information (emails, phone numbers, WhatsApp links, and social media profiles) from a website and its contact subpages. Drastically saves tokens compared to feeding the entire raw DOM to the LLM.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target website URL to extract contact details from. | |
| deep_scan | No | Whether to scan linked contact and about subpages (/contato, /about) for emails (default: true). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses meaningful behavior: results are 'verified', extraction follows links into contact/about subpages, and it returns compact structured data rather than raw DOM. It omits failure modes, JS-rendered page handling, timeouts, and whether any auth or rate limits apply.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the verb and extracted resource before any justification. No filler; the second sentence functions as a routing signal rather than padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema and no annotations, so the description does well by naming the returned field categories. It is nearly complete for a two-parameter read tool, with only minor gaps around output shape and behavior on dynamic sites.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so both 'url' and 'deep_scan' are already documented, including the subpage examples and the default. The description only restates that subpages are scanned and adds no syntax or format detail beyond the schema, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (extract), a specific resource (contact information), and enumerates the exact output types (emails, phone numbers, WhatsApp links, social profiles). This enumeration clearly separates it from siblings like extract_web_content and extract_structured_data, which target generic or arbitrary content.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The closing sentence implies a use case by contrasting the tool with 'feeding the entire raw DOM to the LLM', which hints at when to prefer it. However, no sibling tools are named, no prerequisites or exclusions are given, and nothing tells the agent when NOT to use it (e.g. when only page text is needed).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_structured_dataA
Extract structured JSON — page metadata (Open Graph, canonical URL, author, dates) and/or a custom CSS-selector schema — from a website.
What it does Fetches a URL and returns a structured object. With include_metadata=true it reads Open Graph tags (og:title, og:description, og:image), the canonical URL, author, and publication/modified dates. You can also pass a schema (property name -> CSS selector) to pull specific fields (e.g. {"title": "h1", "price": ".price", "rating": "span.rating"}) into a typed object.
How to use
Use for product listings, news/metadata, schema-driven extraction, and any time you need fields instead of prose.
schema keys become the JSON keys; values are CSS selectors. Leave schema empty to get metadata only.
include_metadata (default true) adds the Open Graph / author / date block. Set false for a lean result.
mode/css_selector/headers/cookies/proxy behave exactly as in extract_web_content.
Returns JSON object with the requested fields (and/or metadata). Missing selectors are omitted rather than failing the whole call. Non-http or private URLs fail with isError: true.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target website URL to extract content and metadata from. | |
| mode | No | Crawl mode: 'fast' (native fetch) or 'deep' (Playwright Chromium). | fast |
| proxy | No | Optional HTTP/SOCKS5 proxy URL. | |
| schema | No | Optional key-value map of property names to CSS selectors (e.g. { title: 'h1', price: '.price' }). | |
| cookies | No | Optional custom HTTP cookies key-value dictionary. | |
| headers | No | Optional custom HTTP request headers key-value dictionary. | |
| max_retries | No | Maximum retry attempts (default: 3). | |
| css_selector | No | Optional CSS selector to scope content before processing. | |
| include_metadata | No | Whether to extract Open Graph tags, canonical URL, author, and date metadata (default: true). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations the description carries the full burden, and it delivers real behavioral detail: missing selectors are omitted rather than failing the call, and non-http/private URLs return isError: true. It adds defaults for include_metadata but does not cover auth, rate limits, or cost/latency differences between fast and deep mode.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the one-line summary, then organizes detail under 'What it does', 'How to use', and 'Returns' headings, which makes scanning easy for a 9-parameter tool. It is slightly verbose with the section headers and parenthetical enumerations, but every line carries actionable content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description explains the return shape, partial-failure behavior, and error signal, and documents the defaults for include_metadata and the schema parameter. An agent has everything needed to invoke and interpret results without consulting other tools.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3, but the description adds genuine semantic value: schema keys become the JSON keys and values are CSS selectors (with a concrete example), include_metadata=true adds the OG/author/date block and false yields a lean result. mode/css_selector/headers/cookies/proxy are punted to a sibling tool's semantics, which is a mild indirection.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: 'Extract structured JSON — page metadata ... and/or a custom CSS-selector schema — from a website.' It also implicitly distinguishes itself from the prose-returning sibling by framing output as 'fields instead of prose.' An agent can tell what it does and how it differs from extract_web_content without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives concrete use cases ('product listings, news/metadata, schema-driven extraction') and a selection rule ('any time you need fields instead of prose'), plus a clarifying case ('Leave schema empty to get metadata only'). It stops short of explicitly naming extract_web_content as the alternative for prose, so the routing is implied rather than spelled out.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_web_contentA
Extract token-optimized, clean Markdown content from a target website for LLM consumption.
What it does Fetches a URL, strips HTML bloat (scripts, styles, inline SVGs, tracking tags, navigation, footers, redundant forms) and returns readable Markdown — cutting token usage by ~73% to 90% versus the raw page.
How to use
Use for any web page, article, or docs you want as LLM-ready text.
mode="fast" (default): native HTTP GET. mode="deep": headless Chrome with stealth fingerprint for JS-heavy or Cloudflare/Turnstile-protected pages.
css_selector scopes to a single element (e.g. "main", "article", "#content").
Pass headers/cookies for gated or geo-restricted content. Providing any of headers/cookies/proxy bypasses the SQLite cache.
No auth required. Respects robots/rate-limit via retry backoff.
Returns Markdown text. Cache hits are prefixed with a NOTE marker. Unsupported URLs (non-http, private IP, SSRF) fail with isError: true and a reason.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target website URL to extract content from. | |
| mode | No | Crawl mode: 'fast' (native HTTP fetch) or 'deep' (headless Playwright browser with JS execution). | fast |
| proxy | No | Optional HTTP/SOCKS5 proxy URL (e.g. 'http://proxy.example.com:8080'). | |
| actions | No | Optional pre-crawl automation actions for mode='deep' (e.g. click cookie consent, load more, scrolling). | |
| cookies | No | Optional custom HTTP cookies key-value dictionary. | |
| headers | No | Optional custom HTTP request headers key-value dictionary. | |
| image_mode | No | Image handling: 'ignore' (omit images), 'alt_only' (> 🖼️ *[Imagem: alt]*), or 'markdown' (). Default: ignore. | ignore |
| link_format | No | Link formatting: 'inline' ([text](url)), 'references' ([text][1] footnote citations at end), or 'strip' (plain text). Default: inline. | inline |
| max_retries | No | Maximum retry attempts for transient errors or rate limits (default: 3). | |
| css_selector | No | Optional CSS selector to scope content extraction to a specific HTML node (e.g. 'main', 'article'). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden, and it delivers: caching behavior (header/cookie/proxy bypasses SQLite cache), auth-free access, robots/rate-limit respect via retry backoff, and failure semantics (isError: true for unsupported/SSRF URLs). It omits any note on rate limits' magnitude, output size caps, or timeout behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded one-line purpose, then clearly labeled What it does / How to use / Returns sections. Every sentence adds actionable information with zero filler, and it is well sized for a 10-param tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 10-parameter tool with nested actions, 100% schema coverage, and no output schema, the description covers purpose, mode selection, caching, auth, and error behavior adequately. A brief mention of actions usage or pagination/output limits would make it fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, setting a baseline of 3, but the description adds meaning beyond the schema: it explains mode tradeoffs (native HTTP vs stealth headless Chrome), that css_selector scopes to a single element, and that supplying headers/cookies/proxy changes caching. The nested actions array is only lightly touched via mode='deep' context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Extract) and resource (web content), and qualifies the output precisely as token-optimized clean Markdown for LLM consumption. This distinguishes it from siblings like crawl_website and extract_structured_data by naming the output format rather than a crawl or structured-data framing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives concrete when-to-use (any page, article, or docs) and per-parameter routing advice like mode=fast vs mode=deep for JS-heavy or Cloudflare-protected pages. It does not explicitly name sibling alternatives such as crawl_website or extract_structured_data to say when not to use this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
map_websiteA
Discover the architecture and page inventory of a website domain via sitemaps, robots.txt, and link graphs. Returns a list of URLs with source and modification metadata. Useful for exploring a site before targeted crawling.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target website URL or domain root (e.g. 'https://example.com'). | |
| max_links | No | Maximum number of links to discover (default: 100). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses the return shape ('list of URLs with source and modification metadata'), which is useful, but says nothing about read-only nature, network/rate-limit behavior, robots.txt compliance, or failure modes for a network-fetching tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences: the purpose and mechanism lead, and the return/usage context follows. Every clause carries information with no padding or restatement of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, and the description does cover the return contents, which is the main gap it needed to fill. Remaining omissions are behavioral (safety/rate/permission context) rather than structural, so it is largely complete for a 2-parameter discovery tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so both parameters (url, max_links) are already fully documented in the schema, including the default and 1-500 range. The description adds no parameter-level detail beyond that, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Discover the architecture and page inventory of a website domain') plus the mechanisms used (sitemaps, robots.txt, link graphs). This clearly separates it from extraction siblings, though it does not name the closest sibling (crawl_website) explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Useful for exploring a site before targeted crawling' gives a concrete usage context that positions it ahead of crawl_website/extract_web_content. It stops short of naming an alternative or stating when not to use it, so it is strong context without exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_placesA
Search for local businesses, places, and establishments on Google Maps, extracting structured entity data (title, address, phone, website, coordinates, rating).
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Maximum number of places to retrieve (default: 5, max: 20). | |
| query | Yes | Search query (e.g. 'restaurantes em Pinheiros Sao Paulo', 'dentists in Bristol'). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It discloses the data source (Google Maps) and the returned entity fields, which is useful context, but it omits read-only confirmation, auth requirements, rate limits, pagination, and result-limit behavior beyond what the schema states.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, front-loaded sentence with no wasted words. The verb and resource come first, and the extracted field list is compact enough to remain useful without bloating the definition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description appropriately compensates by naming the returned fields (title, address, phone, website, coordinates, rating). For a simple two-parameter search tool, this is nearly complete, though it still leaves usage relative to local-lead siblings unaddressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so both the query syntax examples and the limit default/max are already fully documented in the input schema. The description adds no additional parameter meaning, which meets the baseline expectation when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Search) and resource (local businesses, places, establishments on Google Maps) and lists the structured fields returned. It is clear enough to separate from web-content and crawling siblings, but it does not explicitly differentiate itself from the potentially overlapping discover_local_leads or extract_contacts tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the verb 'Search' and the Google Maps/local-business scope, so an agent can infer when it applies. However, there is no explicit when-to-use guidance, no exclusions, and no named alternative among the sibling tools when local lead discovery or contact extraction might be preferred.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
v1.1.2- Changed
crawl_website6 fields changed- changed
Input schema / properties / exclude_patterns / descriptionPrevious value: -"Regex patterns for URLs to ignore (e.g. ['/login', '/tags/'])."New value: +"Substring patterns (`*` = wildcard) for URLs to ignore (e.g. ['/login', '/tags/']). Max 10." - added
Input schema / properties / exclude_patterns / items / maxLengthAdded value: +200 - added
Input schema / properties / exclude_patterns / maxItemsAdded value: +10 - changed
Input schema / properties / include_patterns / descriptionPrevious value: -"Regex patterns that candidate URLs must match (e.g. ['/docs/'])."New value: +"Substring patterns (`*` = wildcard) that candidate URLs must match (e.g. ['/docs/']). Max 10." - added
Input schema / properties / include_patterns / items / maxLengthAdded value: +200 - added
Input schema / properties / include_patterns / maxItemsAdded value: +10
- Changed
extract_web_content11 fields changed- changed
Input schema / properties / actions / items / properties / milliseconds / descriptionPrevious value: -"Milliseconds to wait"New value: +"Milliseconds to wait (max 10000)" - added
Input schema / properties / actions / items / properties / milliseconds / maximumAdded value: +10000 - added
Input schema / properties / actions / items / properties / milliseconds / minimumAdded value: +0 - changed
Input schema / properties / actions / items / properties / milliseconds / typePrevious value: -"number"New value: +"integer" - changed
Input schema / properties / actions / items / properties / pixels / descriptionPrevious value: -"Pixels to scroll"New value: +"Pixels to scroll (max 20000)" - added
Input schema / properties / actions / items / properties / pixels / maximumAdded value: +20000 - added
Input schema / properties / actions / items / properties / pixels / minimumAdded value: +0 - changed
Input schema / properties / actions / items / properties / pixels / typePrevious value: -"number"New value: +"integer" - added
Input schema / properties / actions / items / properties / selector / maxLengthAdded value: +500 - added
Input schema / properties / actions / items / properties / value / maxLengthAdded value: +5000 - added
Input schema / properties / actions / maxItemsAdded value: +20
8 tool updates
v1.1.1- Changed
batch_extract_web_content1 field changed- changed
Input schema / properties / urls / descriptionPrevious value: -"Array of target website URLs to extract."New value: +"Array of target website URLs to extract (1-20)."
- Added
crawl_website - Added
deep_enrich_lead - Added
discover_local_leads - Added
extract_contacts - Changed
extract_web_content4 fields changed- added
Input schema / properties / actionsAdded value: +{ + "description": "Optional pre-crawl automation actions for mode='deep' (e.g. click cookie consent, load more, scrolling).", + "items": { + "additionalProperties": false, + "properties": { + "direction": { + "description": "Scroll direction", + "enum": [ + "up", + "down" + ], + "type": "string" + }, + "milliseconds": { + "description": "Milliseconds to wait", + "type": "number" + }, + "pixels": { + "description": "Pixels to scroll", + "type": "number" + }, + "selector": { + "description": "CSS selector for click/fill", + "type": "string" + }, + "type": { + "description": "Action type", + "enum": [ + "click", + "scroll", + "wait", + "fill" + ], + "type": "string" + }, + "value": { + "description": "Text value to fill into input", + "type": "string" + } + }, + "required": [ + "type" + ], + "type": "object" + }, + "type": "array" +} - changed
Input schema / properties / css_selector / descriptionPrevious value: -"Optional CSS selector to scope content extraction to a specific HTML node."New value: +"Optional CSS selector to scope content extraction to a specific HTML node (e.g. 'main', 'article')." - added
Input schema / properties / image_modeAdded value: +{ + "default": "ignore", + "description": "Image handling: 'ignore' (omit images), 'alt_only' (> 🖼️ *[Imagem: alt]*), or 'markdown' (). Default: ignore.", + "enum": [ + "ignore", + "alt_only", + "markdown" + ], + "type": "string" +} - added
Input schema / properties / link_formatAdded value: +{ + "default": "inline", + "description": "Link formatting: 'inline' ([text](url)), 'references' ([text][1] footnote citations at end), or 'strip' (plain text). Default: inline.", + "enum": [ + "inline", + "references", + "strip" + ], + "type": "string" +}
- Added
map_website - Added
search_places
3 tool updates
v0.1.0- First observed
batch_extract_web_content - First observed
extract_structured_data - First observed
extract_web_content
TDQS
Scored across 9 tools
Most tools target distinct actions, but two near-duplicate pairs stand out: discover_local_leads vs search_places (both pull the same Google Maps entity fields), and extract_contacts vs deep_enrich_lead (both crawl a business site for emails/socials). Descriptions add some differentiation, but an agent could easily misselect within these pairs.
All names are snake_case and follow a verb_noun pattern (discover_local_leads, extract_contacts, crawl_website, map_website, search_places). Minor compounds like deep_enrich_lead and batch_extract_web_content are still readable and consistent with the convention.
Nine tools is well within the sweet spot for a crawling/scraping server, and each tool covers a distinct capability tier (discovery, extraction, enrichment, batch, site mapping). No obvious padding or redundancy-driven bloat.
The surface covers the full lead-gen lifecycle: place discovery, contact extraction, lead enrichment, content/markdown extraction, recursive crawl, batch, structured extraction, and sitemap discovery. The main gap is a lack of a status/polling or pagination/continuation operation for long-running crawls or large result sets.
Maintenance
Related MCP Connectors
Web pages, PDFs and search results as LLM-ready Markdown or chunks. Firecrawl-compatible, EU-hosted.
Any web page -> clean markdown. Firecrawl-lite, 0.001 USDC/extract (x402, Base), free demo.
Firecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Related MCP Servers
- AlicenseAqualityCmaintenanceMCP server for web scraping — extract clean markdown, links, and metadata from any URL. Free Firecrawl alternative.562 npm6MIT
- AlicenseNot gradedqualityDmaintenanceOpen-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.2AGPL 3.0
- AlicenseAqualityBmaintenanceA fault-tolerant, stealth-enabled Model Context Protocol (MCP) server for web searching and content fetching. Built for AI Agents (Cursor, Claude Code, OpenCode), it uses a stealth browser engine to fetch pages, dynamically handles SPAs/React, and converts bloat into token-optimized Markdown.229 npm51 PyPI4MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and LLM pipelines to fetch and crawl websites locally, extract readable content, and convert pages into clean Markdown with preserved links while minimizing token usage.-