How do I use Universal Web Retrieval MCP?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Universal Web Retrieval MCP search for the latest AI news" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

Universal Web Retrieval MCP

by withgardener

Overview Schema Source Related Servers Discussions Admin

Python

Remote

Universal Web Retrieval MCP

A universal MCP server providing resilient web search and content retrieval through automatic provider failover across AnySearch, Tavily, and DDGS.

Overview

Give any MCP-capable agent reliable web access through a single server. Universal Web Retrieval routes each request through an ordered provider chain, picks keyed or keyless authentication automatically based on which API keys are present, and falls over to the next provider on rate limits, outages, or upstream errors.

Agent (any MCP client)
│
├── web_search ──▶ AnySearch ──fail──▶ Tavily ──fail──▶ DDGS
└── web_fetch  ──▶ AnySearch ──fail──▶ Tavily ──fail──▶ DDGS.extract

Related MCP server: scrapesearch-mcp

Features

Two standard tools — web_search and web_fetch, clean JSON in/out.
Automatic provider failover — one provider failing never fails the tool.
Auto keyed/keyless — key configured → official keyed API; key absent → official keyless access. No mode switches to configure.
Strict official APIs — every provider is called through its official documented endpoint and auth mechanism. No scraping hacks, no undocumented endpoints.
Error classification — rate limits / outages fall over; invalid credentials fail loudly (ERROR log) and move to the next provider; malformed input never triggers fallback.
Bounded latency — per-provider timeouts plus an overall chain deadline.
Zero agent coupling — standard MCP stdio; works with Hermes, Claude Code, OpenCode, or any MCP client.

Architecture

src/universal_web_retrieval/
├── server.py            # MCP tool surface (web_search / web_fetch)
├── config.py            # env-driven settings, call-time key reads
├── errors.py            # ProviderError classification, result models, Provider ABC
└── providers/
    ├── router.py        # ProviderRouter: ordered chains + error classification
    ├── anysearch.py     # AnySearch (keyed Bearer / keyless no-auth-header)
    ├── tavily.py        # Tavily (keyed Bearer / keyless X-Tavily-Access-Mode)
    └── ddgs.py          # DDGS (official package; text + extract)

Installation

# from PyPI (planned distribution name):
pip install uwr
# or from source:
pip install git+https://github.com/withgardener/universal-web-retrieval-mcp.git

# run (stdio MCP):
uwr
# or:
PYTHONPATH=src python -m universal_web_retrieval
# version:
uwr --version

ddgs is a runtime dependency (the fixed final fallback of both chains), not an optional extra. Dependencies are pinned to compatible ranges (mcp>=2,<3, httpx>=0.27,<1, ddgs>=9.16,<10) so a major-version breaking change never ships to a running server via a routine pip update.

Configuration

Env var	Required	Effect
`ANYSEARCH_API_KEY`	no	AnySearch keyed mode; absent → official keyless
`TAVILY_API_KEY`	no	Tavily keyed mode; absent → official keyless (`X-Tavily-Access-Mode: keyless`)
`WEB_RETRIEVAL_TIMEOUT`	no	Overall chain deadline in seconds (default 45)
`WEB_RETRIEVAL_LOG_LEVEL`	no	`WARNING` default; logs go to stderr (stdout is MCP JSON-RPC)
`WEB_RETRIEVAL_MAX_RESULTS`	no	Cap for `web_search.limit` (default 20)
`WEB_RETRIEVAL_EXTRACT_CHAR_LIMIT`	no	Max fetched-content chars (default 15000)
`WEB_RETRIEVAL_ANYSEARCH_CONNECT_TIMEOUT` / `_TIMEOUT`	no	AnySearch connect/read timeouts (5s / 20s)
`WEB_RETRIEVAL_TAVILY_CONNECT_TIMEOUT` / `_TIMEOUT`	no	Tavily connect/read timeouts (5s / 20s)
`WEB_RETRIEVAL_DDGS_TIMEOUT`	no	DDGS overall timeout (15s)

Tools

`web_search(query, limit=5)`

{
  "success": true,
  "provider": "anysearch",
  "mode": "keyless",
  "latency_ms": 1555,
  "results": [{"title": "...", "url": "...", "snippet": "..."}]
}

provider / mode / latency_ms are informational metadata — consumers only need results.

`web_fetch(url | urls)`

Accepts a single url string or a urls array (max 5). Returns a JSON array:

[{
  "url": "https://example.com",
  "content": "# Example Domain\n\n...",
  "content_type": "text_markdown",
  "title": "Example Domain",
  "provider": "anysearch",
  "mode": "keyless",
  "latency_ms": 191
}]

Failed URLs carry an error field instead of content. Content is Markdown/clean text — never raw HTML. JS-rendered, login-walled, or CAPTCHA-protected pages are not guaranteed (plain HTTP retrieval only; no browser automation).

Network scope & SSRF note

web_fetch retrieves any URL the host environment can reach — this deliberately includes private-network and localhost targets, which is a legitimate capability for a local agent tool. Redirects may land on private or link-local destinations. When deploying against untrusted agent input, apply network-level isolation (container/netns/firewall) as appropriate for your threat model. A future WEB_RETRIEVAL_BLOCK_PRIVATE=1 opt-in may add in-process filtering; v0.1.0 intentionally does not restrict private fetches.

Batch behavior: urls is capped at 5 per call (larger batches are rejected with an explicit error, not silently truncated), duplicates are de-duplicated preserving first-seen order, and the whole batch shares one deadline — URLs whose turn arrives after the deadline return a timeout error without any network attempt. Error text containing URLs has credential-looking query parameters (token=, api_key=, ...) redacted.

Routing & fallback

Capability	Chain
`web_search`	AnySearch → Tavily → DDGS
`web_fetch`	AnySearch → Tavily → DDGS.extract

A link is skipped when: rate-limited (429), timed out, upstream 5xx, or any provider error. Invalid credentials fail loudly (ERROR log AUTH FAILURE on <provider>) and the chain moves to the next provider — a wrong key is never silently retried as keyless on the same provider. Malformed input (empty query, bad URL scheme) returns an immediate input error with no fallback.

Provider behavior

See docs/providers.md for each provider's official endpoint, auth mechanism, keyless behavior, and last-verified date.

Examples

Generic MCP client (stdio)

{
  "mcpServers": {
    "universal-web-retrieval": {
      "command": "python",
      "args": ["-m", "universal_web_retrieval"],
      "env": {
        "PYTHONPATH": "/path/to/universal-web-retrieval-mcp/src",
        "ANYSEARCH_API_KEY": "as_sk_...",
        "TAVILY_API_KEY": "tvly-..."
      }
    }
  }
}

Hermes Agent

In ~/.hermes/config.yaml:

mcp_servers:
  universal-web-retrieval:
    command: /path/to/venv/bin/python
    args: ["-m", "universal_web_retrieval"]
    transport: stdio
    enabled: true
    env:
      PYTHONPATH: /path/to/universal-web-retrieval-mcp/src
      TAVILY_API_KEY: tvly-...

Claude Code

claude mcp add universal-web-retrieval \
  -- python -m universal_web_retrieval
# with PYTHONPATH set, or after `pip install -e .`

Development

pip install pytest
pytest tests/unit -v            # router/auth/deadline unit tests (no network)

# Live provider smoke tests (hit real APIs; opt-in):
RUN_LIVE_TESTS=1 pytest tests/integration -v

Live tests cover all three providers in both keyless and keyed modes; keyed tests SKIP (not fail) when the corresponding key is not configured.

Testing

Unit tests cover the router (fallback order, auth-failure semantics, invalid input), and integration smoke tests exercise all three providers live. See tests/.

License

MIT

Available Tools

2 tools

web_fetchA

Fetch one URL's page content as Markdown with automatic provider failover (AnySearch -> Tavily -> DDGS.extract). Auth is automatic (keyed when a key is configured, else official keyless). Accepts a single url string, or urls array (max 5) for batch. Returns JSON array of {url, content, title, content_type, provider, mode, latency_ms} — failed URLs carry an error field instead. Note: JS-rendered, login-walled or CAPTCHA-protected pages are not guaranteed.

ParametersJSON Schema

Name	Required	Description	Default
`url`	No
`urls`	No

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

A4.4/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so well: it discloses the provider failover chain (AnySearch -> Tavily -> DDGS.extract), automatic/keyless auth behavior, batch limits, the return field set, the error-field failure mode, and JS/ CAPTCHA limitations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the core action, then layers failover, auth, parameter modes, and limitations in tight sentences. No filler and no repetition of the schema beyond useful clarification.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Even though an output schema exists (so return values need not be described), the description still names the response shape and error path, and covers auth, failover, and page-type limits. Nothing needed to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate, and it does: it explains url as a single URL string and urls as a batch array capped at 5, clarifying the two parameters' roles. It does not explain defaults or the interaction when both are supplied, keeping it out of 5.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Fetch one URL's page content as Markdown'), which is clearly separable from the web_search sibling in practice. It never names web_search explicitly, so the sibling differentiation is implicit rather than stated, holding it below a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear usage context: single url string or a urls array for batch up to 5, plus a note on what kinds of pages are not guaranteed. There is no explicit 'use web_search instead when...' exclusion, so it stops short of full routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_searchB

Web search with automatic provider failover (AnySearch -> Tavily -> DDGS). Auth is automatic: a configured API key enables keyed mode, otherwise the provider's official keyless access is used. Returns JSON: {success, provider, mode, latency_ms, results: [{title, url, snippet}]}.

ParametersJSON Schema

Name	Required	Description	Default
`limit`	No
`query`	Yes

Output Schema

ParametersJSON Schema

Name	Required	Description
`result`	Yes

TDQS

B3.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does substantial work: it names the failover order (AnySearch -> Tavily -> DDGS), explains that auth is automatic, and distinguishes keyed from keyless mode. It stops short of covering failure behavior when all providers fail, rate limits, or read-only/write semantics.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences with the purpose front-loaded and no filler. One sentence is spent restating the return JSON shape, which is partly redundant given an output schema exists, so it is efficient but not maximally so.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter search tool with an output schema, the description covers purpose, auth, and failover adequately. It is incomplete on parameter behavior (limit default/ceiling) and on what the agent should expect when every provider fails, leaving real gaps despite the output schema reducing the need to explain results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, yet the description never mentions 'query' or 'limit' semantics — no defaults, ranges, or format guidance. With two undocumented parameters and zero schema descriptions, the description fails to compensate; only the trivially self-evident name 'query' saves it from a 1.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear verb+resource ('Web search') and adds meaningful scope detail through the provider failover chain. It does not explicitly contrast itself with the sibling web_fetch, so sibling differentiation is left to inference, but an agent can still tell what the tool does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no statement of when to use this tool versus web_fetch or any other alternative, and no exclusion conditions. The only context given is internal operating behavior (failover, auth), not selection guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

2 tool updatesv0.1.0Sep 14, 2026
- First observedweb_fetch
- First observedweb_search

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.8/5.0

Scored 2026-09-14 14:50 across 2 tools

Disambiguation5/5

web_fetch retrieves content from a known URL, while web_search discovers URLs via a query. Their purposes are clearly distinct with no overlapping behavior.

Naming Consistency5/5

Both tools follow the same snake_case 'web_<verb>' pattern, making the set predictable and easy to scan.

Tool Count3/5

Only two tools are provided; while they are core, the set feels thin for a server named 'Universal Web Retrieval' and lacks depth beyond basic search and fetch.

Completeness4/5

The core read path (search + fetch) is covered, including batch fetch up to 5 URLs. Minor gaps remain around pagination, crawling, or advanced search filters, but agents can work around them.

Maintenance

ActivityNo data

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

Keenable Web Search A
ai.keenable.api
Docs: https://docs.keenable.ai/mcp-server Keenable is a free, remote MCP server that gives agents access to the web index. Search the web with ranked results and date/site filters, then fetch any indexed page as clean markdown. Works out of the box with no account or API key.
Research & Data Search Web Scraping
23
MESSORA A
dev.messora
Scrape, crawl and search the web for AI agents via MCP.
Web Scraping Browser Automation Search
51
AIsa Web Search & Research AOAuth
one.aisa
Your agent needs the open web — searched by more than one engine, and read as clean markdown rather than raw HTML. **What you can ask for** • "Search this question with two providers and tell me where they disagree." • "Scrape these 40 URLs into markdown, in one batch." • "Crawl this documentation site and give me every page." • "Do deep research on this topic and cite the sources." • "Find the academic papers behind this claim." **How to use it** Point any MCP client at https://mcp.aisa.one/search/mcp and sign in with OAuth — there is no key to create or paste. 30 tools across several independent providers: Tavily and Exa search, answers, contents and agent runs; Firecrawl scrape, batch scrape, crawl, map and search; Perplexity Sonar, Sonar Pro, reasoning and deep research; Oxylabs AI search and LLM jobs; OpenAI and Anthropic web search; and scholarly search. **Why this rather than the source** Several independent indexes behind one account, because one engine's blind spot is not visible from inside it. **It is also a door to the rest** The same login reaches 26 sources and 580+ operations. Find the page here, then ask the same agent who links to it or how much traffic it gets — without adding a second server. **What it costs** Finding and inspecting an operation is free. Running one is billed per call at API prices, with no seat and no monthly minimum, and every call takes max_price_usd so an agent cannot overspend by accident. **Where else it reaches** https://mcp.aisa.one/seo-serp/mcp for the Google results page itself, https://mcp.aisa.one/seo-serp-other-engines/mcp for Bing, Baidu and Naver.
Browser Automation Search Web Scraping
351
pixserp A
com.pixserp
Live AI-native web search with citations. One tool for every MCP client. Flat per-request pricing.
Search Browser Automation
12

Related MCP Servers

websearch-skill
Search Web Scraping Browser Automation
hec-ovi
A
license
A
quality
B
maintenance
Enables AI agents to perform multi-engine web search, fetch web pages, and extract clean Markdown content via MCP, with no API keys required.
Updated a month ago (2026-08-13 14:14 UTC)
3
92 PyPI
8
MIT
scrapesearch-mcp
tonmoy-dhroubo
F
license
Not graded
quality
D
maintenance
MCP server that provides web search scraping from DuckDuckGo (with Mojeek fallback) and URL content fetching as markdown/text or raw HTML.
Updated 8 months ago (2026-01-25 21:53 UTC)
1
-
Search Toolkit
Search Web Scraping Agent Orchestration
TheWiseWolfHolo
A
license
Not graded
quality
B
maintenance
Enables AI agents to perform web search, extraction, and research across multiple provider backends with persistent multi-key rotation and auditable routing. Supports MCP, CLI, and Agent Skill entry points.
Updated 21 days ago (2026-08-30 06:54 UTC)
MIT
ddgs-mcp
Search Web Scraping
marcinkuk
A
license
A
quality
C
maintenance
Enables keyless multi-engine web, news, image, and video metasearch plus full-page markdown extraction for AI agents over MCP, with resilient fallback across DuckDuckGo, Bing, Brave, Google, and other backends.
Updated 20 days ago (2026-08-30 12:27 UTC)
6
MIT

Universal Web Retrieval MCP

Overview

Features

Architecture

Installation

Configuration

Tools

web_search(query, limit=5)

web_fetch(url | urls)

Network scope & SSRF note

Routing & fallback

Provider behavior

Examples

Generic MCP client (stdio)

Hermes Agent

Claude Code

Development

Testing

License

Available Tools

Output Schema

TDQS

Output Schema

TDQS

Tool Schema Changelog

TDQS

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

websearch-skill

scrapesearch-mcp

Search Toolkit

ddgs-mcp

`web_search(query, limit=5)`

`web_fetch(url | urls)`