# Headless Lead Scraping MCP Server v0.4.0

A UI-free MCP server for real-time public-web lead research with **no search API provider, no Tavily key, and no hosted scraping service**.

## What it does

- Searches **Google Maps first** (browser-rendered) for local places — returns name, address, phone, and official website.
- Then searches **web search engines** (Yahoo, DuckDuckGo, Bing) directly at request time, with no search API key.
- Then searches **related search engines** (Startpage, browser-rendered) as an extra fallback source.
- Fetches target websites directly over HTTP.
- Falls back to headless Chromium/Playwright for JavaScript-rendered pages.
- Crawls relevant internal contact/about/team/social pages.
- Uses the configured LLM through the Vercel AI SDK for structured extraction and verification.
- Deduplicates results and filters by required fields.
- Stores leads locally in `data/leads.jsonl`.
- Filters stored leads and exports them as CSV or JSON files.
- Exposes everything as MCP tools over stdio.

## Data-source priority

Sources are tried in order until enough results are found:

1. **Google Maps** (`googlemaps`) — browser-rendered `google.com/maps/search/`; best for local businesses/product companies (name, address, phone, website, rating).
2. **Web search engines** — Yahoo, DuckDuckGo, Bing (`yahoo`, `duckduckgo`, `bing`).
3. **Related search engines** — Startpage (`startpage`, browser-rendered) for additional coverage.

The order is configurable via `SEARCH_ENGINES` (comma-separated). Maps-related settings: `MAPS_SEARCH_URL`, `MAPS_MAX_RESULTS`.

## No external data API

There is **no Tavily, Bing Search API, Google Search API, SerpAPI, Apify, Bright Data, or hosted scraper** in this version.

The server performs search by requesting normal public search-result webpages and Google Maps search pages directly. Engines that respond with a CAPTCHA or bot-challenge page are skipped automatically. This means there is no API account or API key for search, but a public search website is still necessarily involved: the open Internet has no universal built-in index that a standalone program can query without using some indexed search source or already-known URLs.

If a search website blocks automated requests, the server cannot and does not bypass CAPTCHA, login, paywalls, robots directives, rate limits, or access controls.

## LLM

The only required AI dependency is your LLM endpoint. The server supports OpenAI-compatible endpoints, including:

- Ollama
- LM Studio
- OpenAI
- other OpenAI-compatible local/self-hosted endpoints

For maximum independence, use Ollama or LM Studio locally.

## Playwright

Playwright is the browser engine, not a lead-data provider. It is used only when a normal HTTP fetch cannot obtain the rendered page content or when browser rendering is explicitly requested.

Install Chromium:

```bash
npm install
npx playwright install chromium
```

## Install

```bash
cp .env.example .env
npm install
npx playwright install chromium
npm run build
npm start
```

For development:

```bash
npm run dev
```

## Ollama

```bash
ollama pull qwen3
```

`.env`:

```env
LLM_PROVIDER=ollama
LLM_MODEL=qwen3
LLM_BASE_URL=http://127.0.0.1:11434/v1
LLM_API_KEY=ollama
```

## LM Studio

Start a local OpenAI-compatible server in LM Studio and set:

```env
LLM_PROVIDER=lmstudio
LLM_MODEL=YOUR_LOADED_MODEL_ID
LLM_BASE_URL=http://127.0.0.1:1234/v1
LLM_API_KEY=lm-studio
```

## MCP tools

- `search_web` — live web search in priority order (Google Maps, then web engines, then related engines); optional `engines` param.
- `scrape_page` — HTTP fetch + Playwright fallback.
- `extract_leads` — LLM extraction from supplied page content.
- `find_leads` — Google Maps place discovery first, then web + related engine search; crawls, extracts, verifies, deduplicates, filters by `requiredFields`, stores.
- `list_stored_leads` — read locally stored records.
- `filter_leads` — filter stored leads by keyword, field presence, and minimum confidence.
- `export_leads` — write stored leads (optionally filtered) to a CSV or JSON file on disk.
- `clear_stored_leads` — clear local records.

## Example

```json
{
  "query": "software development companies",
  "location": "California, USA",
  "limit": 25,
  "requiredFields": ["name", "website", "email"],
  "maxSearchQueries": 8
}
```

`requiredFields` filters out any lead missing all of the listed fields. Default is `["name"]`.

Export example:

```json
{
  "format": "csv",
  "hasEmail": true,
  "minConfidence": "medium",
  "filename": "california-leads"
}
```

This writes `data/california-leads.csv` (or `.json` for `format: "json"`) and returns the file path and row count.

## Stored data

Default:

```text
data/leads.jsonl
```

No hosted database is required. The file is local to the MCP server process.

## Important limitation

A truly provider-free program cannot discover arbitrary Internet resources from nothing: search requires an index or known URLs. This project removes **search APIs and hosted scraping providers** and performs live search-page retrieval itself. It does not pretend that the Internet has a universal local index.

Only publicly accessible information is collected. The server does not bypass authentication, CAPTCHA, paywalls, robots/access controls, or other security mechanisms.