Skip to main content
Glama
aggregate-search

aggregate-search

by TZH-user

aggregate-search โ€” Multi-Engine Search MCP Server

Overview

aggregate-search turns multiple search backends โ€” Bocha / Zhipu (China-friendly), Tavily / Serper (overseas), keyless DuckDuckGo and an optional self-hosted SearXNG โ€” into a single MCP server (web_search / read_url / engine_status) served over stdio with FastMCP. Chinese queries automatically boost domestic engines (weight 1.3) and English queries boost overseas ones; when an overseas engine is unreachable it lands in a 300 s health blacklist and the request falls back to domestic sources instead of timing out.

On top of the fusion layer sits a lexical relevance filter (drops off-topic results, with a full audit trail), a near-duplicate merger for same-site URL-parameter variants, and an optional embedding/LLM re-ranker that promotes authoritative pages. Every iteration is tracked by a frozen 14-query gold set plus an 8-query hold-out set in eval/ โ€” gold Hit@5 11/11, junk results 0/70, hold-out generalization confirmed with zero false kills.

๐Ÿ‡จ๐Ÿ‡ณ ไธญๆ–‡็‰ˆ่ฏดๆ˜Ž่ง README_CN.md

Related MCP server: evo-scry

Highlights

  • ๐Ÿ”€ Weighted RRF fusion โ€” score = ฮฃ w/(60+rank) with a cross-engine hit bonus (+0.25), so results confirmed by multiple engines rank higher; each result carries an engines field listing its sources

  • ๐Ÿงน Relevance filter + near-duplicate merge โ€” lexical coverage scoring (AGG_RELEVANCE=1 default) drops off-topic results with a dropped_items / downweighted_items audit trail, and merges same-site URL-parameter variants (title similarity โ‰ฅ 0.9)

  • ๐ŸŽฏ Optional re-ranking โ€” AGG_RERANK=embedding (Zhipu embedding-3 cosine) or llm (glm-4-flash intent ranking, RankGPT-style with graceful degrade), lifting authoritative pages that lexical layers cannot see

  • ๐Ÿ“„ read_url full-text tool โ€” fetch a page and get clean markdown (trafilatura), direct-first with proxy retry; closes the search โ†’ read loop for agents

  • ๐Ÿฉบ Health blacklist & failure taxonomy โ€” failing engines are benched for 300 s; every failure is classified (quota_exhausted / auth_failed / network_unreachable / timeout / rate_limited / server_error) with an actionable hint in engine_errors

  • โšก Query cache + soft deadline โ€” TTL cache (300 s, LRU 128, never caches failures) and AGG_SOFT_DEADLINE_S early return from completed engines on tail-latency spikes

  • ๐ŸŒ Proxy auto-negotiation โ€” overseas engines go through AGG_PROXY (env โ†’ common Clash ports โ†’ Windows registry proxy), validated by a real CONNECT + TLS handshake before use, with automatic direct-retry fallback; domestic engines always connect directly

  • ๐Ÿงช 90 offline tests + stdio E2E + real-quota check โ€” pytest suite (fusion / engines / orchestrator / relevance / rerank / cache) runs without network or quota; verify_e2e.py drives the server through a real MCP stdio client; real_check.py exercises live keys

  • ๐Ÿงฉ Pluggable engines โ€” add one adapter file + one config.SPECS registration; per-engine kill switch via AGG_ENABLE_<NAME>=0

Tech Stack

Architecture

             query (zh/en auto-detected)
                      โ”‚
        โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ orchestrator โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
        โ”‚ parallel dispatch ยท timeoutsโ”‚
        โ”‚ soft deadline ยท query cache โ”‚
        โ–ผ                              โ–ผ
  domestic engines              overseas engines
  Bocha ยท Zhipu                 Tavily ยท Serper  (+ keyless DuckDuckGo,
  always direct                 via validated proxy   optional SearXNG)
        โ”‚                              โ”‚
        โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ fusion โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
   URL normalization โ†’ dedup โ†’ weighted RRF (k=60)
   + cross-engine bonus โ†’ per-host cap (2)
   โ†’ near-duplicate merge (same-site variants)
                      โ”‚
                      โ–ผ
             relevance filter          AGG_RELEVANCE=1
   coverage scoring ยท auth-domain boost ยท audit trail
                      โ”‚
                      โ–ผ
             optional re-rank           AGG_RERANK
   embedding cosine / LLM intent order, graceful degrade
                      โ”‚
                      โ–ผ
  MCP tools: web_search / read_url / engine_status (stdio)

Getting Started

bash setup.sh                                   # uv venv (Python 3.12) + deps
cp .env.example .env                            # fill at least one engine key
.venv/Scripts/python.exe -m pytest -q           # 90 offline tests (no network)
.venv/Scripts/python.exe verify_e2e.py          # stdio end-to-end (mocked)
.venv/Scripts/python.exe real_check.py          # live quota check

Register with any MCP client (ZCode / Claude Code etc.):

"aggregate-search": {
  "type": "stdio",
  "command": "<abs-path>/.venv/Scripts/python.exe",
  "args": ["<abs-path>/server.py"],
  "cwd": "<abs-path>"
}

No key ships with this repository โ€” obtain your own: Bocha https://open.bochaai.com (CN, 1000 free), Zhipu https://open.bigmodel.cn, Tavily https://app.tavily.com (1000/month free), Serper https://serper.dev, or point SEARXNG_BASE_URL at a self-hosted instance (no key). See .env.example.

Configuration

  • BOCHA_API_KEY / ZHIPU_API_KEY / TAVILY_API_KEY / SERPER_API_KEY / SEARXNG_BASE_URL โ€” all optional; whichever is set gets used. AGG_ENABLE_DUCKDUCKGO=1 enables the keyless DuckDuckGo fallback engine

  • AGG_RELEVANCE (1) โ€” lexical relevance filter; AGG_REL_MIN_COV (0.4) / AGG_REL_PENALTY (0.3) / AGG_AUTH_BOOST (1.3) tune it; 0 = pass-through

  • AGG_RERANK (off) โ€” embedding (Zhipu embedding-3, AGG_RERANK_MODEL) / llm (glm-4-flash, AGG_RERANK_LLM_MODEL) or stacked "embedding,llm"; AGG_RERANK_ALPHA (0.4) blends similarity into the score, AGG_RERANK_POOL (20) caps candidates

  • AGG_CACHE_TTL_S (300, 0 = off) โ€” LRU query cache (128 entries, failures never cached); AGG_SOFT_DEADLINE_S (10, 0 = off) โ€” return completed engines' results early on tail-latency spikes

  • AGG_MAX_PER_HOST (2) โ€” per-site result cap; AGG_NEAR_DUP (1) / AGG_NEAR_DUP_THRESHOLD (0.9) โ€” same-site parameter-variant merge; AGG_READ_MAX_CHARS (8000) โ€” read_url output cap

  • AGG_DOMAIN_BOOST / AGG_DOMAIN_BLOCK โ€” per-domain boost/block rules (AGG_DOMAIN_BOOST_FACTOR 1.5)

  • AGG_PROXY โ€” egress proxy for overseas engines only; empty = env โ†’ probe ports 7897/7890/10809 โ†’ Windows registry proxy

  • AGG_ENABLE_<NAME>=0 โ€” force-disable an engine; AGG_TIMEOUT_S (20) / AGG_CONNECT_TIMEOUT_S (10) / AGG_UNHEALTHY_TTL_S (300)

  • ZHIPU_SEARCH_ENGINE โ€” search_std (default) / search_pro / search_pro_sogou / search_pro_quark

Evaluation

A frozen 14-query gold set across 5 dimensions (CN/EN freshness, authoritative-source locating, off-topic resistance, precise facts), each result scored 0/1/2 against the built-in WebSearch baseline; plus an 8-query hold-out set (never used for tuning) to check filter over-fitting. Methodology, per-item scoring evidence and raw engine outputs are all kept in eval/.

Metric (v6)

aggregate-search

built-in WebSearch

Gold Hit@5 (11 applicable)

11/11

11/11

MRR

0.848

1.000

Junk results (score=0)

0/70

0/62

Result coverage

70/70

62/70

Latency

mean 2.7 s / median 2.6 s

not measured

  • v1โ†’v6 evolution (coverage filter โ†’ per-engine cap & dedup fixes โ†’ semantic re-rank โ†’ near-duplicate merge): junk 5/70 โ†’ 0/70, Hit@5 10/11 โ†’ 11/11 โ€” eval/report.md

  • Hold-out A/B: with the filter on, junk 5/40 โ†’ 0/40 and zero false kills on normal queries โ€” eval/holdout_report.md

Repository Layout

server.py              MCP entry (stdio): web_search / read_url / engine_status
aggregate_search/
  config.py            engine registry + zh/en weights + key detection
  models.py            unified SearchResult model
  fusion.py            URL normalization, dedup, weighted RRF, cross-source bonus, per-host cap, near-duplicate merge
  relevance.py         lexical relevance filter + authoritative-domain boost + audit trail
  rerank.py            embedding / LLM re-ranking with graceful degradation
  reader.py            read_url full-text extraction (trafilatura) + proxy retry
  orchestrator.py      parallel dispatch, health blacklist, failure taxonomy, proxy probing, cache, soft deadline
  engines/             per-engine adapters (add file + register in config.SPECS)
tests/                 90 mocked tests (fusion / engines / orchestrator / relevance / rerank / cache)
eval/                  frozen gold set, per-version scores, hold-out report, raw outputs
verify_e2e.py          stdio E2E via a real MCP client
real_check.py          live-key verification (spends quota, prints failure kinds)
CHANGELOG.md           v1โ†’v6 iteration log
docker-compose.yml     optional self-hosted SearXNG (+ searxng-settings/)
HANDOFF_ๆŽฅ็ฎก่ฏŠๆ–ญ.md     engineering diagnosis report (CN)
ๅ€Ÿ้‰ด่ฐƒ็ ”.md             survey of comparable open-source search MCPs (CN)
.env.example           key template

Known Limitations

  • Engine adapters follow official docs; after API changes only the adapter file needs updating

  • Re-ranking can only reorder what engines return โ€” an authoritative site absent from the result pool stays absent (recall gap, see eval/holdout_report.md H8); per-engine depth or engine-side fixes are the way out

  • Overseas engines need a working egress proxy; on machines without one, disable them via AGG_ENABLE_<NAME>=0 to skip cold-start timeouts

License

MIT ยฉ 2026 TZH-user โ€” see LICENSE.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelogโ€ฆ

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that enables web search capabilities by integrating with a SearXNG instance to aggregate results from over 130 engines. It allows users to perform filtered searches across categories like news, science, and social media while supporting advanced parameters for language and time range.
      14
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      MCP server for internet search via direct Google and DuckDuckGo HTML scraping with AI-powered result normalization and optional summarization, requiring no API keys for search.
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that aggregates web search results from multiple engines and optionally renders pages to Markdown, providing a unified search interface.
      6 npm
      3
      ISC