Skip to main content
Glama

Web Search MCP

Multi-source web search MCP server with RRF fusion, 4-layer URL extraction, and provider health tracking.

Status

Phase 2: 5 providers (SearXNG/ddgr/Tavily/Exa/opencli-zh), RRF fusion, snippet cleaning (rules + optional LLM compression), search/extract/doctor tools, provider health tracking with auto-degradation.

Related MCP server: Prism

Install

Add to Claude Desktop 3P config (~/Library/Application Support/Claude-3p/configLibrary/<uuid>.json):

{
  "name": "web-search",
  "source": "user",
  "transport": "stdio",
  "command": "node",
  "args": ["/Users/hades/projects/web-search-mcp/bin/cli.js"],
  "env": {
    "SEARXNG_URL": "http://localhost:18443",
    "TAVILY_API_KEY": "...",
    "EXA_API_KEY": "...",
    "FIRECRAWL_API_KEY": "...",
    "WSM_LLM_PROVIDER": "haiku",
    "ANTHROPIC_API_KEY": "..."
  },
  "toolPolicy": {
    "search": "allow",
    "extract": "allow",
    "doctor": "allow"
  }
}

Once published to npm, replace command: "node" + absolute path with command: "npx" + args: ["-y", "@thehappyboy/web-search-mcp@latest"].

Configuration

Environment variables (or .env in cwd or ~/.agents/skills/web-search/.env):

Var

Default

Required

Description

SEARXNG_URL

http://localhost:18443

no

SearXNG base URL

TAVILY_API_KEY

no

Enables Tavily provider

EXA_API_KEY

no

Enables Exa provider

FIRECRAWL_API_KEY

no

Enables Firecrawl in extract pipeline (layer 3)

WSM_LLM_PROVIDER

none

no

haiku for Anthropic, local for Ollama, none to disable

ANTHROPIC_API_KEY

no

Required when WSM_LLM_PROVIDER=haiku

WSM_LOCAL_LLM_URL

no

Required when WSM_LLM_PROVIDER=local

WSM_MAX_RESULTS

10

no

Default max results

WSM_TIMEOUT_MS

25000

no

Global search timeout

Providers without keys are automatically disabled. The search tool still works with whatever is available. ddgr, opencli, defuddle are auto-detected via which.

Tools

{
  query: string,              // required
  max_results?: number,       // 1-30, default 10
  category?: 'general'|'news'|'science',
  mode?: 'fast'|'quality',    // quality = LLM compression (top-5 results)
  providers?: string[],       // override default list (replace semantics)
}

Returns:

{
  count: number,
  took_ms: number,
  providers_used: string[],     // providers that returned ≥1 result
  providers_failed: string[],   // providers that errored OR returned []
  results: [{
    title, url, snippet,
    providers: string[],        // which sources surfaced this URL (multi-source corroboration)
    rrf_score: number,          // reciprocal rank fusion score
  }]
}

opencli-zh auto-routing: When query contains keywords like 知乎, 小红书, 雪球, B站, 微博, or patterns like 600519.SH, the opencli-zh provider routes to the corresponding adapter. Force a specific site with opencli_sites (passed via providers extension, see orchestrator).

extract

{
  url: string,                 // required
  timeout?: number,            // 5-60 seconds, default 20
}

Tries 4 layers in order until one succeeds:

  1. defuddle — fastest, pure-text sites

  2. opencli adapter — for known sites (zhihu/xiaohongshu/xueqiu/bilibili/weibo)

  3. Firecrawl — JS-rendered pages (requires FIRECRAWL_API_KEY)

  4. opencli browser — last resort, drives real Chrome

Returns:

{
  content: string,             // markdown
  source: 'defuddle'|'opencli'|'firecrawl'|'opencli-browser'|'none',
  took_ms: number,
  error?: string,              // only when source === 'none'
}

doctor

{} // no arguments

Returns provider availability + runtime health stats:

{
  providers: {
    searxng:    { available: true, url: "http://localhost:18443" },
    ddgr:       { available: true },
    tavily:     { available: true },
    exa:        { available: true },
    opencli_zh: { available: true },
    firecrawl:  { available: true },
    llm:        { available: true, provider: "haiku" },
  },
  health: {
    tavily: {
      total: 24,
      success: 15,
      fail: 9,
      success_rate: 0.625,
      last_failure: "2026-07-06T...",
      last_failure_reason: "timeout",
      recommendation: "healthy" | "insufficient" | "degraded",
    },
    // ...
  },
  version: "0.2.0",
}

When a provider's success_rate drops below 0.5 with at least 20 samples, it's automatically skipped (recommendation: degraded). Health stats are in-memory and reset on restart.

Development

npm test                  # all tests (unit + integration, 158 total)
npm run test:unit         # unit only
npm run test:integration  # integration only (needs searxng at SEARXNG_URL)
npm start                 # start stdio server

Architecture

MCP client (Claude/Cowork)
  │ JSON-RPC over stdio
  ▼
server.js ───┬── search tool ──→ search.js (orchestrator)
             ├── extract tool ─→ extract/index.js (4-layer pipeline)
             └── doctor tool ──→ health tracker snapshot

search.js:
  Provider layer (parallel, isolated failure)
    searxng | ddgr | tavily | exa | opencli-zh
  ▼
  Merger layer: normalize → dedup(URL) → RRF(k=60)
  ▼
  Cleaner layer: rules (default) | LLM compression (quality mode, top-5)
  ▼
  Health tracker records success/failure per provider

extract/index.js:
  defuddle → opencli adapter → Firecrawl → opencli browser
  (each layer returns null on failure, falls through to next)

Phase history

  • Phase 1 (15 tasks): 4 providers (searxng/ddgr/tavily/exa), RRF fusion, rule-based cleaning, stdio MCP, search tool only

  • Phase 2 (8 tasks): opencli-zh provider, LLM quality mode, 4-layer extract, doctor + health auto-degradation

License

MIT

Available Tools

6 tools
doctorC

Diagnose provider availability and health.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description must carry the full burden. It suggests a read-only operation ('diagnose') but does not confirm idempotency, side effects, or authorization needs. Important behavioral traits are missing.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, concise sentence. It is front-loaded, but the content is too minimal to earn a 5. There is no fluff, making it appropriately sized for a simple tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the lack of annotations, output schema, and parameter details, the description is the sole source of context. It does not explain what the tool returns, how to interpret results, or any operational constraints. A healthy provider check would benefit from at least indicating output format or typical usage.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There are no parameters, so the schema coverage is effectively 100%. According to guidelines, baseline is 4 for zero params. The description does not need to add param details, and it correctly omits them. No contradiction.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a verb 'diagnose' and references 'provider availability and health,' but the meaning of 'diagnose' is vague. It is unclear whether this tool checks status, runs tests, or returns reports. It distinguishes from sibling tools like 'search' and 'extract' only implicitly, but not explicitly enough.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives. The description does not mention prerequisites, context, or exclusions. The agent is left to infer usage from the vague verb 'diagnose'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extractA

Extract clean markdown from a URL. Tries defuddle → opencli adapter → Firecrawl → opencli browser in order.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesURL to extract
timeoutNo

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and effectively discloses the fallback order of extraction methods, which is useful for understanding reliability and behavior. However, it does not mention side effects or error handling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that efficiently conveys the main purpose and the fallback chain. Every part earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the basic function but lacks details about return value format, handling of failures, or edge cases. Without an output schema, more context about what 'clean markdown' entails would improve completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only 50% of schema parameters have descriptions (url has one, timeout does not). The tool description does not compensate by explaining timeout or adding meaning beyond the schema. It repeats that extraction is from a URL but adds no parameter-specific details.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool extracts clean markdown from a URL, with a specific verb and resource. It distinguishes from sibling tools like search and doctor by focusing on content extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for extracting markdown from URLs but does not explicitly say when or when not to use this tool versus alternatives. No comparison to sibling tools is provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

vertical_batchC

Up to 5 vertical queries in parallel via AnySearch.

ParametersJSON Schema
NameRequiredDescriptionDefault
queriesYesUp to 5 independent queries. Each item: {query, domain?, sub_domain?, max_results?}

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses parallel execution and a limit of 5, but no annotations exist. Missing information on safety (read-only vs destructive), authentication, error behavior, or result format. The description carries the full burden and is insufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence, no waste, and front-loads the key constraint. Could be slightly expanded to include return behavior without losing conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has one complex parameter, no output schema, and no annotations. The description does not explain return values, error conditions, or what 'vertical queries' are. It is adequate for a simple tool but incomplete for practical use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a clear description of each item's structure. The description adds 'in parallel' but that is behavioral, not about parameter semantics. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states it runs 'vertical queries in parallel' up to 5, clearly indicating the action and resource. However, it does not use a specific verb like 'execute' or 'run', and assumes familiarity with the term 'vertical queries'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus siblings like vertical_search, search, or extract. The phrase 'via AnySearch' provides context about the backend but not about selection criteria.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

vertical_sub_domainsA

Discover sub-domains and required parameters for a given vertical domain. Call before vertical_search when sub_domain schema is unknown.

ParametersJSON Schema
NameRequiredDescriptionDefault
domainNoSingle domain to inspect (e.g. "finance")
domainsNoComma-separated list of domains

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description must cover behavior. It implies a read operation ('Discover'), but does not explicitly state it's non-destructive, mention return format, or disclose any side effects or authorization needs.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the action verb and resource. No unnecessary words; each sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description does not clarify return structure (e.g., list of subdomains). It also does not mention handling of multiple domains in 'domains' parameter or relation to other siblings like vertical_batch.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with parameter descriptions. The description adds value by explaining the purpose of the parameters (discovering sub-domains/params), giving context beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool discovers sub-domains and required parameters for a vertical domain, and distinguishes it from the sibling 'vertical_search' by specifying when to call it.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says to call before vertical_search when sub_domain schema is unknown, providing good contextual guidance. However, it does not address when not to use the tool or mention other siblings like vertical_batch.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 6 tool updatesv0.1.0
    • First observeddoctor
    • First observedextract
    • First observedsearch
    • First observedvertical_batch
    • First observedvertical_search
    • First observedvertical_sub_domains

TDQS

A3.5/5.0

Scored across 6 tools

Disambiguation4/5

Most tools are clearly distinct: 'doctor' for health checks, 'extract' for URL content, 'search' for general search, and 'vertical_*' for domain-specific search. However, 'vertical_search' and 'vertical_batch' both perform vertical searches, differing only in batch size, which could cause slight confusion.

Naming Consistency3/5

Naming is inconsistent: 'doctor', 'extract', 'search' are single verbs without a clear pattern, while 'vertical_batch', 'vertical_search', and 'vertical_sub_domains' use a snake_case prefix. This mix of conventions reduces predictability.

Tool Count5/5

With 6 tools, the set is well-scoped for a web search and extraction server. Each tool serves a distinct purpose without being excessive or insufficient.

Completeness4/5

The tool surface covers core functionality: health checks, general search, URL extraction, and vertical domain search with discovery. Minor gaps exist, such as no tool for image or news search, but the provided set is functionally complete for the stated domain.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Multi-level web search MCP server that wraps Claude, Gemini, Perplexity, and Tavily behind a unified interface, enabling multi-depth searches with session management and provider selection.
    MIT
  • A
    license
    Not graded
    quality
    A
    maintenance
    Multi-engine aggregated search MCP server that combines results from 7 search engines with deduplication, relevance ranking, and web page content extraction.
    1
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    MCP server for multi-engine web search and web page fetching, supporting parallel search, content extraction, and optional LLM-powered search summarization and deep search.
    4
    MIT