# Web Search MCP

Multi-source web search MCP server with RRF fusion, 4-layer URL extraction, and provider health tracking.

## Status

**Phase 2**: 5 providers (SearXNG/ddgr/Tavily/Exa/opencli-zh), RRF fusion, snippet cleaning (rules + optional LLM compression), `search`/`extract`/`doctor` tools, provider health tracking with auto-degradation.

## Install

Add to Claude Desktop 3P config (`~/Library/Application Support/Claude-3p/configLibrary/<uuid>.json`):

```jsonc
{
  "name": "web-search",
  "source": "user",
  "transport": "stdio",
  "command": "node",
  "args": ["/Users/hades/projects/web-search-mcp/bin/cli.js"],
  "env": {
    "SEARXNG_URL": "http://localhost:18443",
    "TAVILY_API_KEY": "...",
    "EXA_API_KEY": "...",
    "FIRECRAWL_API_KEY": "...",
    "WSM_LLM_PROVIDER": "haiku",
    "ANTHROPIC_API_KEY": "..."
  },
  "toolPolicy": {
    "search": "allow",
    "extract": "allow",
    "doctor": "allow"
  }
}
```

Once published to npm, replace `command: "node"` + absolute path with `command: "npx"` + `args: ["-y", "@thehappyboy/web-search-mcp@latest"]`.

## Configuration

Environment variables (or `.env` in cwd or `~/.agents/skills/web-search/.env`):

| Var | Default | Required | Description |
|---|---|---|---|
| `SEARXNG_URL` | `http://localhost:18443` | no | SearXNG base URL |
| `TAVILY_API_KEY` | — | no | Enables Tavily provider |
| `EXA_API_KEY` | — | no | Enables Exa provider |
| `FIRECRAWL_API_KEY` | — | no | Enables Firecrawl in extract pipeline (layer 3) |
| `WSM_LLM_PROVIDER` | `none` | no | `haiku` for Anthropic, `local` for Ollama, `none` to disable |
| `ANTHROPIC_API_KEY` | — | no | Required when `WSM_LLM_PROVIDER=haiku` |
| `WSM_LOCAL_LLM_URL` | — | no | Required when `WSM_LLM_PROVIDER=local` |
| `WSM_MAX_RESULTS` | `10` | no | Default max results |
| `WSM_TIMEOUT_MS` | `25000` | no | Global search timeout |

Providers without keys are automatically disabled. The search tool still works with whatever is available. `ddgr`, `opencli`, `defuddle` are auto-detected via `which`.

## Tools

### `search`

```js
{
  query: string,              // required
  max_results?: number,       // 1-30, default 10
  category?: 'general'|'news'|'science',
  mode?: 'fast'|'quality',    // quality = LLM compression (top-5 results)
  providers?: string[],       // override default list (replace semantics)
}
```

Returns:
```js
{
  count: number,
  took_ms: number,
  providers_used: string[],     // providers that returned ≥1 result
  providers_failed: string[],   // providers that errored OR returned []
  results: [{
    title, url, snippet,
    providers: string[],        // which sources surfaced this URL (multi-source corroboration)
    rrf_score: number,          // reciprocal rank fusion score
  }]
}
```

**opencli-zh auto-routing**: When query contains keywords like `知乎`, `小红书`, `雪球`, `B站`, `微博`, or patterns like `600519.SH`, the opencli-zh provider routes to the corresponding adapter. Force a specific site with `opencli_sites` (passed via providers extension, see orchestrator).

### `extract`

```js
{
  url: string,                 // required
  timeout?: number,            // 5-60 seconds, default 20
}
```

Tries 4 layers in order until one succeeds:
1. **defuddle** — fastest, pure-text sites
2. **opencli adapter** — for known sites (zhihu/xiaohongshu/xueqiu/bilibili/weibo)
3. **Firecrawl** — JS-rendered pages (requires `FIRECRAWL_API_KEY`)
4. **opencli browser** — last resort, drives real Chrome

Returns:
```js
{
  content: string,             // markdown
  source: 'defuddle'|'opencli'|'firecrawl'|'opencli-browser'|'none',
  took_ms: number,
  error?: string,              // only when source === 'none'
}
```

### `doctor`

```js
{} // no arguments
```

Returns provider availability + runtime health stats:
```js
{
  providers: {
    searxng:    { available: true, url: "http://localhost:18443" },
    ddgr:       { available: true },
    tavily:     { available: true },
    exa:        { available: true },
    opencli_zh: { available: true },
    firecrawl:  { available: true },
    llm:        { available: true, provider: "haiku" },
  },
  health: {
    tavily: {
      total: 24,
      success: 15,
      fail: 9,
      success_rate: 0.625,
      last_failure: "2026-07-06T...",
      last_failure_reason: "timeout",
      recommendation: "healthy" | "insufficient" | "degraded",
    },
    // ...
  },
  version: "0.2.0",
}
```

When a provider's `success_rate` drops below 0.5 with at least 20 samples, it's automatically skipped (recommendation: `degraded`). Health stats are in-memory and reset on restart.

## Development

```bash
npm test                  # all tests (unit + integration, 158 total)
npm run test:unit         # unit only
npm run test:integration  # integration only (needs searxng at SEARXNG_URL)
npm start                 # start stdio server
```

## Architecture

```
MCP client (Claude/Cowork)
  │ JSON-RPC over stdio
  ▼
server.js ───┬── search tool ──→ search.js (orchestrator)
             ├── extract tool ─→ extract/index.js (4-layer pipeline)
             └── doctor tool ──→ health tracker snapshot

search.js:
  Provider layer (parallel, isolated failure)
    searxng | ddgr | tavily | exa | opencli-zh
  ▼
  Merger layer: normalize → dedup(URL) → RRF(k=60)
  ▼
  Cleaner layer: rules (default) | LLM compression (quality mode, top-5)
  ▼
  Health tracker records success/failure per provider

extract/index.js:
  defuddle → opencli adapter → Firecrawl → opencli browser
  (each layer returns null on failure, falls through to next)
```

## Phase history

- **Phase 1** (15 tasks): 4 providers (searxng/ddgr/tavily/exa), RRF fusion, rule-based cleaning, stdio MCP, `search` tool only
- **Phase 2** (8 tasks): opencli-zh provider, LLM quality mode, 4-layer `extract`, `doctor` + health auto-degradation

## License

MIT