Web Search MCP
This server provides tools for multi-source web searching, content extraction from URLs, and diagnostics for provider availability and health. You can:
Search the web: Perform general, news, or science-related searches using multiple providers with features like RRF fusion, snippet cleaning, and optional LLM compression.
Perform vertical searches: Conduct structured searches in specific domains like finance, academic, security, and more, retrieving field-level data.
Discover vertical sub-domains: Find out available sub-domains and required parameters for specific vertical domains before performing a vertical search.
Batch vertical queries: Execute up to 5 vertical search queries in parallel.
Extract content from URLs: Get clean markdown content from a given URL using a multi-layered extraction process (defuddle, opencli adapter, Firecrawl, opencli browser).
Diagnose provider health: Check the availability and runtime health statistics of search and extraction providers, including success rates and any degradation recommendations.
Supports search routing and content extraction for Bilibili (Chinese video platform) via the opencli-zh provider.
Provides optional LLM-driven snippet compression using a local Ollama instance for quality search results.
Provides web search results from a SearXNG instance as one of multiple search providers.
Supports search routing and content extraction for Xiaohongshu (Chinese social media platform) via the opencli-zh provider.
Supports search routing and content extraction for Zhihu (Chinese Q&A platform) via the opencli-zh provider.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Web Search MCP search for latest AI news"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Web Search MCP
Multi-source web search MCP server with RRF fusion, 4-layer URL extraction, and provider health tracking.
Status
Phase 2: 5 providers (SearXNG/ddgr/Tavily/Exa/opencli-zh), RRF fusion, snippet cleaning (rules + optional LLM compression), search/extract/doctor tools, provider health tracking with auto-degradation.
Related MCP server: web-mcp
Install
Add to Claude Desktop 3P config (~/Library/Application Support/Claude-3p/configLibrary/<uuid>.json):
{
"name": "web-search",
"source": "user",
"transport": "stdio",
"command": "node",
"args": ["/Users/hades/projects/web-search-mcp/bin/cli.js"],
"env": {
"SEARXNG_URL": "http://localhost:18443",
"TAVILY_API_KEY": "...",
"EXA_API_KEY": "...",
"FIRECRAWL_API_KEY": "...",
"WSM_LLM_PROVIDER": "haiku",
"ANTHROPIC_API_KEY": "..."
},
"toolPolicy": {
"search": "allow",
"extract": "allow",
"doctor": "allow"
}
}Once published to npm, replace command: "node" + absolute path with command: "npx" + args: ["-y", "@thehappyboy/web-search-mcp@latest"].
Configuration
Environment variables (or .env in cwd or ~/.agents/skills/web-search/.env):
Var | Default | Required | Description |
|
| no | SearXNG base URL |
| — | no | Enables Tavily provider |
| — | no | Enables Exa provider |
| — | no | Enables Firecrawl in extract pipeline (layer 3) |
|
| no |
|
| — | no | Required when |
| — | no | Required when |
|
| no | Default max results |
|
| no | Global search timeout |
Providers without keys are automatically disabled. The search tool still works with whatever is available. ddgr, opencli, defuddle are auto-detected via which.
Tools
search
{
query: string, // required
max_results?: number, // 1-30, default 10
category?: 'general'|'news'|'science',
mode?: 'fast'|'quality', // quality = LLM compression (top-5 results)
providers?: string[], // override default list (replace semantics)
}Returns:
{
count: number,
took_ms: number,
providers_used: string[], // providers that returned ≥1 result
providers_failed: string[], // providers that errored OR returned []
results: [{
title, url, snippet,
providers: string[], // which sources surfaced this URL (multi-source corroboration)
rrf_score: number, // reciprocal rank fusion score
}]
}opencli-zh auto-routing: When query contains keywords like 知乎, 小红书, 雪球, B站, 微博, or patterns like 600519.SH, the opencli-zh provider routes to the corresponding adapter. Force a specific site with opencli_sites (passed via providers extension, see orchestrator).
extract
{
url: string, // required
timeout?: number, // 5-60 seconds, default 20
}Tries 4 layers in order until one succeeds:
defuddle — fastest, pure-text sites
opencli adapter — for known sites (zhihu/xiaohongshu/xueqiu/bilibili/weibo)
Firecrawl — JS-rendered pages (requires
FIRECRAWL_API_KEY)opencli browser — last resort, drives real Chrome
Returns:
{
content: string, // markdown
source: 'defuddle'|'opencli'|'firecrawl'|'opencli-browser'|'none',
took_ms: number,
error?: string, // only when source === 'none'
}doctor
{} // no argumentsReturns provider availability + runtime health stats:
{
providers: {
searxng: { available: true, url: "http://localhost:18443" },
ddgr: { available: true },
tavily: { available: true },
exa: { available: true },
opencli_zh: { available: true },
firecrawl: { available: true },
llm: { available: true, provider: "haiku" },
},
health: {
tavily: {
total: 24,
success: 15,
fail: 9,
success_rate: 0.625,
last_failure: "2026-07-06T...",
last_failure_reason: "timeout",
recommendation: "healthy" | "insufficient" | "degraded",
},
// ...
},
version: "0.2.0",
}When a provider's success_rate drops below 0.5 with at least 20 samples, it's automatically skipped (recommendation: degraded). Health stats are in-memory and reset on restart.
Development
npm test # all tests (unit + integration, 158 total)
npm run test:unit # unit only
npm run test:integration # integration only (needs searxng at SEARXNG_URL)
npm start # start stdio serverArchitecture
MCP client (Claude/Cowork)
│ JSON-RPC over stdio
▼
server.js ───┬── search tool ──→ search.js (orchestrator)
├── extract tool ─→ extract/index.js (4-layer pipeline)
└── doctor tool ──→ health tracker snapshot
search.js:
Provider layer (parallel, isolated failure)
searxng | ddgr | tavily | exa | opencli-zh
▼
Merger layer: normalize → dedup(URL) → RRF(k=60)
▼
Cleaner layer: rules (default) | LLM compression (quality mode, top-5)
▼
Health tracker records success/failure per provider
extract/index.js:
defuddle → opencli adapter → Firecrawl → opencli browser
(each layer returns null on failure, falls through to next)Phase history
Phase 1 (15 tasks): 4 providers (searxng/ddgr/tavily/exa), RRF fusion, rule-based cleaning, stdio MCP,
searchtool onlyPhase 2 (8 tasks): opencli-zh provider, LLM quality mode, 4-layer
extract,doctor+ health auto-degradation
License
MIT
Available Tools
6 toolsdoctorC
Diagnose provider availability and health.
Inputs: no named parameters.
extractA
Extract clean markdown from a URL. Tries defuddle → opencli adapter → Firecrawl → opencli browser in order.
Inputs: url (required), timeout.
searchA
Multi-source web search with RRF fusion. Providers tried in parallel: SearXNG/ddgr/Tavily/Exa/opencli-zh. Results deduplicated by URL and reranked by reciprocal rank fusion. Snippets cleaned.
Inputs: query (required), mode, category, providers, +1 more.
vertical_batchC
Up to 5 vertical queries in parallel via AnySearch.
Inputs: queries (required).
vertical_searchA
Vertical-domain structured search via AnySearch. Use for finance (real-time quotes, fundamentals), academic (papers), security (CVE), ip (patents), legal, health. Returns structured payload (field-level data, not flattened).
Inputs: query (required), domain, sub_domain, max_results, +1 more.
vertical_sub_domainsA
Discover sub-domains and required parameters for a given vertical domain. Call before vertical_search when sub_domain schema is unknown.
Inputs: domain, domains.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 6 tools
Most tools are clearly distinct: 'doctor' for health checks, 'extract' for URL content, 'search' for general search, and 'vertical_*' for domain-specific search. However, 'vertical_search' and 'vertical_batch' both perform vertical searches, differing only in batch size, which could cause slight confusion.
Naming is inconsistent: 'doctor', 'extract', 'search' are single verbs without a clear pattern, while 'vertical_batch', 'vertical_search', and 'vertical_sub_domains' use a snake_case prefix. This mix of conventions reduces predictability.
With 6 tools, the set is well-scoped for a web search and extraction server. Each tool serves a distinct purpose without being excessive or insufficient.
The tool surface covers core functionality: health checks, general search, URL extraction, and vertical domain search with discovery. Minor gaps exist, such as no tool for image or news search, but the provided set is functionally complete for the stated domain.
Maintenance
Related MCP Connectors
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Unified 15-tool SEO, SERP, clean Markdown & GEO (LLM citation Share-of-Voice) MCP server.
Web search API and MCP server for developers building AI agents.
MCP server for Google search results via SERP API
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceMulti-engine aggregated search MCP server that combines results from 7 search engines with deduplication, relevance ranking, and web page content extraction.1MIT
- AlicenseNot gradedqualityCmaintenanceMCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.5MIT
- FlicenseNot gradedqualityDmaintenanceMCP server that provides web search via LLM API gateways with server-side search, returning cited answers and multi-provider fallback.-
- AlicenseAqualityCmaintenanceA self-hosted MCP server for web search with multi-engine support, combining search, URL fetching, and deep research along with proxy routing and deduplication.3MIT