# Web Search MCP Server

A Model Context Protocol (MCP) server that enables free web searching with anti-detection, using Google primarily and falling back to Bing then DuckDuckGo (no API keys required).

## Features

- Search the web with stealth browser and orchestrated failover: **Google (Playwright, stealth, UA rotation, consent cookies, rate limit, block cooldown) -> Bing (Playwright) -> DuckDuckGo (https+cheerio lightweight)**
- DuckDuckGo fallback is lightweight `node:https` + `cheerio` (no browser) against `html.duckduckgo.com`
- No API keys or authentication required
- Returns structured results with titles, URLs, and descriptions
- Configurable number of results per search (1-10) and language (`lang`, default `en`)

## Architecture - Fallback Order

Orchestrator (`src/search-orchestrator.ts`) tries engines sequentially and returns the first success (`isSuccessResults` filters synthetic error titles):

| Order | Engine | Transport | Notes |
|-------|--------|-----------|-------|
| 1 | **Google** | Playwright (`chromium`) | Stealth + UA rotation + consent cookies; rate limit + block cooldown (see details below) |
| 2 | **Bing** | Playwright (`chromium`) | Same stealth context + UA rotation; decodes `bing.com/ck/a?u=a1<base64url>` redirects; `networkidle` 30s may time out - falls through to DDG |
| 3 | **DuckDuckGo** | `node:https` + `cheerio` (no browser) | Hits `html.duckduckgo.com/html/?q=...`, parses `.result`; handles `uddg` param and `//` prefix; HTTP 202 treated as engine failure; 20s request timeout |

**Google antibot details** (`src/antibot-config.ts` / `src/playwright-helpers.ts`):
- **Stealth JS** (`STEALTH_JS`): hides `navigator.webdriver`, `window.chrome`, `navigator.languages`, `navigator.plugins`, `navigator.permissions.query` (notifications)
- **UA rotation** (`USER_AGENTS`): 21 desktop UAs (Windows Chrome 120-131, macOS, Linux, Edge, Firefox) - random per context
- **Consent cookies** (`GOOGLE_CONSENT_COOKIES`): `CONSENT` + `SOCS` for `.google.com`
- **Rate limit** (`RATE_LIMIT_MS` = 2000ms via `enforceRateLimit()` with 1.5-3.5s jitter)
- **Block cooldown** (`GOOGLE_BLOCK_COOLDOWN_MS` = 900s; `/sorry/` or `GoogleBlocked` marks blocked and skips Google and the orchestrator falls back to Bing -> DuckDuckGo for 15m - it does NOT wait/sleep, just bypasses Google until cooldown expires)

```
query -> enforceRateLimit() -> Google (skip if isGoogleBlocked()) -> Bing -> DuckDuckGo -> Error: AllEnginesFailed
```

Source: `src/google-search.ts`, `src/bing-search.ts`, `src/duckduckgo-search.ts`, `src/playwright-helpers.ts`, `src/antibot-config.ts`, `src/search-orchestrator.ts`.

## Installation

1. Clone or download this repository (requires Node >=18 for `type: module` + Playwright 1.62)
2. Install dependencies:
```bash
npm install
```
3. Install Playwright browser (needed for Google/Bing stealth):
```bash
npx playwright install chromium
# or (Linux only, installs system deps): npx playwright install --with-deps chromium
```
> **Prerequisite:** Without this, Google/Bing engines fail to launch and the orchestrator falls through to DuckDuckGo only. The `playwright` package alone does not download the browser binary.

4. Build the server:
```bash
npm run build
```
> `build/` is **gitignored** (see `.gitignore`). The MCP entry `build/index.js` is not committed - you **must** run `npm run build` locally after clone/pull or after any `src/` edit. The build script is `tsc && node -e "require('fs').chmodSync('build/index.js', '755')"` (uses Node fs, not shell `chmod` - Windows compatible; no `selenium-search` artifact anymore).

5. Add the server to your MCP configuration:

For VSCode (Claude Dev Extension):
```json
{
  "mcpServers": {
    "web-search": {
      "command": "node",
      "args": ["/path/to/web-search/build/index.js"]
    }
  }
}
```

For Claude Desktop:
```json
{
  "mcpServers": {
    "web-search": {
      "command": "node",
      "args": ["/path/to/web-search/build/index.js"]
    }
  }
}
```

## Windows ESM Gotcha - `type: module` + `.js` Extension

This project is **ESM-only**:

- `package.json` has `"type": "module"` - Node treats `.js` as ESM.
- `tsconfig.json` uses `"module": "Node16"` + `"moduleResolution": "Node16"`.
- All local imports **must** include the `.js` extension even though source is `.ts`:

```ts
// correct
import { googleSearch } from './google-search.js';
import { duckDuckGoSearch } from './duckduckgo-search.js';

// wrong - will throw ERR_MODULE_NOT_FOUND on Windows/Linux
import { googleSearch } from './google-search';
import { googleSearch } from './google-search.ts';
```

On **Windows**, `type: module` also makes path casing matter (`forceConsistentCasingInFileNames: true`). If you rename a file, update every import's casing. After changing `tsconfig` module settings, delete `build/` and rebuild.

## Usage

The server provides a single tool named `search` that accepts the following parameters:

```typescript
{
  "query": string,    // The search query
  "limit": number,    // Optional: Number of results to return (default: 5, max: 10)
  "lang": string      // Optional: Language code (default: "en")
}
```

Example usage:
```typescript
use_mcp_tool({
  server_name: "web-search",
  tool_name: "search",
  arguments: {
    query: "your search query",
    limit: 3  // optional
  }
})
```

Example response:
```json
[
  {
    "title": "Example Search Result",
    "url": "https://example.com",
    "description": "Description of the search result..."
  }
]
```

## Limitations

Since this tool scrapes Google/Bing/DuckDuckGo, there are some important limitations to be aware of:

1. **Anti-bot & Rate limiting**: Google may block (`/sorry/` or `GoogleBlocked`); orchestrator enforces `RATE_LIMIT_MS` (2000ms) + `GOOGLE_BLOCK_COOLDOWN_MS` (900s) and falls back to Bing then DDG. Rapid DDG calls may return HTTP 202 (treated as engine failure).
2. **Network stability**: Bing `networkidle` may time out (30s) intermittently - orchestrator falls through to DDG.
3. **Legal Considerations**:
   - This tool is intended for personal use
   - Respect Google/Bing/DuckDuckGo terms of service

## Migration (Phase 6.1 - Dependency Hygiene)

- `src/selenium-search.js` (CJS) **removed**. Use ESM import instead:
  ```ts
  import { duckDuckGoSearch } from './duckduckgo-search.js'
  // CLI: node build/duckduckgo-search.js "query" 3
  ```
  Previous `build/selenium-search.cjs` artifact no longer produced; `npm run build` is now `tsc && node -e "require('fs').chmodSync(...)"`.
- Dead deps removed: `selenium-webdriver`, `chromedriver`, `axios` (+ `@types/axios`). Production DDG now uses native `node:https`. If you relied on `test-search.js` with axios, run `npm i -D axios` or migrate to the ESM CLI above.
- `@types/cheerio` moved to `devDependencies` (types-only).
- Dependency hygiene reduced install from ~115 to ~40 packages and cleared 4 audit vulns (remaining 1 high is `@modelcontextprotocol/sdk <1.24.0` GHSA-w48q-cv73-mx4w - `undici` itself patched to 6.28.0 via `npm audit fix`; update SDK to `>=1.24.0` separately for breaking fix).

## Development - Rebuild & Restart

After editing any file under `src/`:

```bash
npm run build
```

Then **restart opencode** (or reload the `web-search` MCP server) to reload `build/index.js`.

> **Global lesson (see `global lessons.md` #11 - MCP server edits need an opencode restart):** The MCP process (`build/index.js` via stdio) is loaded once by opencode and kept alive. Editing source or rebuilding does NOT take effect until the process is restarted. A stale process will still serve the old code (and may return stale errors). This applies to `mcp-web-search` the same as `messages-board-mcp/server.py` - always restart opencode after changing this server.

Verify the build:

```bash
# smoke test (DDG, no browser needed):
node build/duckduckgo-search.js "test query" 2
# inspector (validates MCP stdio):
npm run inspector  # opens @modelcontextprotocol/inspector
# verify artifact exists (Windows):
dir build
# or: npx tsc --noEmit  (typecheck without emit)
```

```bash
git status  # build/ should stay untracked (gitignored); only src/, README.md, package.json are versioned
```

## Contributing

Feel free to submit issues and enhancement requests!