Web Research MCP

A high-quality, multi-source web research MCP server for AI agents. Plug it into Claude Desktop, Hermes, Cursor, or any MCP-compatible client and get production-grade search + page-fetching across Wikipedia, arXiv, Hacker News, Stack Exchange, Crossref, Brave, Tavily, and any URL on the web.

# One-line install (anywhere on disk)
git clone https://github.com/infinit3labs/web-research-mcp.git
hermes mcp add web-research --command "$(pwd)/web-research-mcp/bin/web-research-mcp"
# 9 of 10 tools work with zero API keys. Add Brave or Tavily to unlock general web search.

Why this exists

Most "web search" MCP servers try to scrape Google through a headless browser with randomized fingerprints. That approach is a losing arms race — search engines detect and ban scrapers within days, and even when it works, you get DOM soup that your LLM has to clean up.

This server takes a different approach — it talks to APIs that are built for agents:

The six vertical providers work without any API keys; Brave and Tavily keys unlock real-time general web search. That's the highest-quality approach — you get better results than scraping because real web-index APIs use signals (click models, freshness, link analysis) that no scraper can replicate.

Related MCP server: sofya-mcp

Quick start

Option A — pip install (when published)

pip install deep-web-research-mcp
hermes mcp add web-research --command "$(which web-research-mcp)"

Note on naming. The PyPI distribution name is deep-web-research-mcp (so pip install deep-web-research-mcp), but the binary on your PATH after install is web-research-mcp (defined by [project.scripts] in pyproject.toml). That's intentional — the binary matches the local launcher bin/web-research-mcp and the MCP registration name web-research. Same package, two names.

Option B — Clone from source

git clone https://github.com/infinit3labs/web-research-mcp.git
hermes mcp add web-research \
  --command "$(pwd)/bin/web-research-mcp"

When prompted, accept all 10 tools. Done.

Option C — Install with Claude Desktop

Edit ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "web-research": {
      "command": "/Users/code/mcp-servers/web-research/bin/web-research-mcp"
    }
  }
}

Option D — Install with Cursor / any stdio MCP client

{
  "mcpServers": {
    "web-research": {
      "command": "/absolute/path/to/web-research-mcp/bin/web-research-mcp"
    }
  }
}

The launcher script auto-creates a venv on first run, installs dependencies from pyproject.toml, and sources web-research.env for any API keys you've configured.

2. (Optional) Add API keys for real web search

cp web-research.env.example web-research.env
$EDITOR web-research.env

The launcher picks up keys from web-research.env on every invocation — no restart of your MCP client needed.

3. Use it

Ask your agent things like:

"Search Hacker News and Stack Overflow for the best MCP servers released in 2026"

"Use pro_mode to research the current state of small language models"

"Fetch https://arxiv.org/abs/2506.06962 and summarize the methodology"

"Cross-reference this claim against Wikipedia and arXiv"

Upgrading & rollback

pip install:

pip install --upgrade deep-web-research-mcp   # upgrade to latest
pip install deep-web-research-mcp==0.1.0      # roll back to a specific version

Restart your MCP client after changing versions so it re-spawns the server process.

Source checkout (bin/web-research-mcp launcher):

git pull                              # upgrade to latest main
git checkout v0.1.0                   # roll back to a tagged release

The launcher re-syncs .venv from pyproject.toml on every invocation, so no manual venv rebuild is needed either way. See CHANGELOG.md for what changed between versions.

Tools

All 10 tools registered in tools/list. Tools fall into two layers:

• Search & fetch (7 tools) — single-shot lookups. One tool, one API, one result.

• Deep research (3 tools) — multi-step pipelines that plan, gather, and structure evidence. Use these when a single search isn't enough.

Search & fetch

search_web — multi-source general web search

search_web(
    query: str,                  # search query
    max_results: int = 10,       # per source, before dedup (1–30)
    pro_mode: bool = False,      # also fetch top 3 URLs and append excerpts
) -> str

Backed by Brave + Tavily with URL-canonicalization dedup and cross-source score boosting. Requires BRAVE_API_KEY and/or TAVILY_API_KEY. With no keys, returns a clear message telling you how to enable it. Tavily runs at advanced search depth with chunks_per_source=3 (multiple relevant excerpts per source) and supports up to 20 results per query.

pro_mode: true is the research-killer feature — it runs a normal search, fetches the top 3 results via Jina, and appends the content as the snippet. One call does what would otherwise be search_web + 3 × fetch_url.

fetch_url — clean markdown of any page

fetch_url(url: str) -> str

Goes through Jina Reader, which:

• renders JS-heavy pages (SPAs, React apps)

• bypasses most bot-detection (Jina is whitelisted)

• returns clean markdown with metadata block (Title:, URL Source:, Published Time:)

• truncates to ~20k chars to protect your context window

Tavily Extract fallback: with TAVILY_API_KEY configured, pages that fail via Jina (hard bot walls, paywalls) are automatically retried through Tavily Extract at advanced depth with markdown formatting. If both paths fail, you get a combined error naming both providers.

search_wikipedia — encyclopedic grounding

search_wikipedia(query: str, max_results: int = 5) -> str

Wikipedia MediaWiki API. Keyless. Fast. Best for definitions and historical context.

search_academic — arXiv preprints

search_academic(query: str, max_results: int = 5) -> str

Returns title, authors, abstract snippet, published date, PDF URL. Keyless. Best for CS, physics, math, bio.

search_news — Hacker News + mainstream news signal

search_news(query: str, max_results: int = 10) -> str

Searches Hacker News (keyless) and — when TAVILY_API_KEY is configured — blends in Tavily's news-topic results from the last week, merged and deduplicated with HN. Best for what's trending in tech right now plus its mainstream coverage.

search_stackexchange — Q&A from 180+ sites

search_stackexchange(query: str, max_results: int = 5, site: str = "stackoverflow") -> str

Set site to any SE community: serverfault, superuser, askubuntu, math, tex, datascience, ai, etc. Keyless.

search_scholar_meta — peer-reviewed papers via Crossref

search_scholar_meta(query: str, max_results: int = 5) -> str

Returns title, DOI, citation count, publisher, publication date, abstract. Covers papers arXiv doesn't (Elsevier, Springer, Wiley, IEEE, ACM). Keyless.

Deep research

These three tools compose the search/fetch primitives above into multi-step research workflows. They never call an LLM themselves — the calling model stays in charge of writing the final narrative; the server's job is to plan, gather, and structure evidence with verifiable citations.

plan_research — structured plan only (no fetches)

plan_research(question: str, depth: str = "standard") -> str  # JSON

Returns a JSON research plan: sub-questions, recommended sources per sub-question, rationale, queries to run, and estimated searches + fetches. Use this when you want to inspect or modify the plan before committing to the full pipeline.

• depth: "quick" (2-3 sub-questions), "standard" (4-6), "deep" (6-8)

extract_evidence — targeted quotes from one URL

extract_evidence(
    url: str,
    question: str,
    max_passages: int = 5,
) -> str  # JSON

Fetches the URL via Jina, splits into paragraphs, scores each for relevance to your question, and returns the top passages. Each passage includes before / quote / after context, a relevance score (0-1), and a offset (character position in the source) so citations are independently verifiable.

Use this when you already have a specific source and want to drill into it for evidence on a narrow claim.

research — full deep-research pipeline

research(question: str, depth: str = "standard") -> str  # markdown + JSON

End-to-end research workflow:

1. Plan — builds the sub-question plan

2. Fan out — searches across the recommended sources for each sub-question in parallel

3. Rank — deduplicates URLs across the whole plan, ranks them with source-aware composite scoring (Wikipedia/arXiv/Crossref 2.0×, Stack Exchange 1.7×, web search 1.5×, Hacker News 1.0×)

4. Fetch — pulls the top URLs via Jina Reader

5. Extract — scores paragraphs for relevance with a quality floor (filters out nav menus, link-only paragraphs, footer cruft)

6. Return — emits a structured ResearchReport:

{
  "question": "What is retrieval augmented generation?",
  "depth": "quick",
  "plan": { "sub_questions": [...], "estimated_searches": 4, ... },
  "citations": [
    { "id": 1, "url": "...", "title": "...", "source": "wikipedia", "quotes": 2 }
  ],
  "evidence": {
    "sq_def": [
      { "citation_id": 1, "relevance": 0.78, "offset": 1234,
        "before": "...", "quote": "...", "after": "..." }
    ]
  },
  "synthesis_template": "# Research Report: ..."
}

The synthesis_template is a Markdown skeleton with one section per sub-question plus a Sources table. You (the model) fill in the narrative, citing each [n] marker against the corresponding entry in citations. Every quoted passage carries a character offset so a reader can verify the citation against the original page.

depth controls breadth:

• "quick" — 2-3 sub-questions, ~6 fetches, ~2 minutes

• "standard" — 4-6 sub-questions, ~20 fetches, ~3 minutes

• "deep" — 6-8 sub-questions, ~32 fetches, ~5 minutes

Architecture

┌─────────────────────────────────────────────────────────┐
│                    MCP Client                            │
│  (Claude Desktop, Hermes, Cursor, custom agent)          │
└────────────────────┬────────────────────────────────────┘
                     │ JSON-RPC over stdio
                     ▼
┌─────────────────────────────────────────────────────────┐
│              bin/web-research-mcp                         │
│  • Boots venv (or reuses cached one)                     │
│  • Sources web-research.env for API keys                 │
│  • Execs python -m web_research.server                   │
└────────────────────┬────────────────────────────────────┘
                     ▼
┌─────────────────────────────────────────────────────────┐
│           web_research.server (MCPServer)                 │
│  10 tool functions registered via @app.tool() decorator   │
│  • Pydantic-driven JSON schemas from type hints           │
│  • Single shared httpx.AsyncClient per call              │
│  • Graceful degradation: one bad source ≠ failed call    │
└────────────────────┬────────────────────────────────────┘
                     │ asyncio.gather for parallel fan-out
                     ▼
┌─────────────────────────────────────────────────────────┐
│          web_research.providers (7 backends)              │
│  ┌──────────┐ ┌──────────┐ ┌─────────────┐               │
│  │ brave    │ │ tavily   │ │ jina_fetch  │  ← general web│
│  └──────────┘ └──────────┘ └─────────────┘               │
│  ┌──────────┐ ┌──────────┐ ┌─────────────┐               │
│  │ wikipedia│ │ arxiv    │ │ crossref    │  ← academic   │
│  └──────────┘ └──────────┘ └─────────────┘               │
│  ┌──────────┐ ┌──────────┐                                │
│  │ hn_algolia│ │stackex   │  ← tech signal               │
│  └──────────┘ └──────────┘                                │
│  + merge_results() with URL-canonical dedup               │
└─────────────────────────────────────────────────────────┘

Key design decisions

API-first, not scrape-first. This is the core thesis. Every source is an official API designed for programmatic access. You get clean structured data, no IP bans, no maintenance burden when sites redesign.

Per-source error isolation. Each provider wraps its HTTP call in try/except. A 429 from one source never sinks the whole search — you get partial results plus a clear message about which source failed.

URL canonicalization. merge_results() strips tracking params (utm_*, fbclid, gclid, ref) before dedup, normalizes case on host, drops fragments. When Brave and Tavily return the same article, you see it once with also_found_in: [brave, tavily] and a boosted score.

Shared HTTP client per call. httpx.AsyncClient with connection pooling (max_connections=20), sane timeouts (30s default, 45s for fetch_url), and automatic redirect following. New client per call because stdio MCP servers process one request at a time and we want clean state.

Bounded fan-out + result caching. Every search/fetch call — plain tool calls and the deep-research pipeline alike — goes through providers.cached_search/cached_fetch. A per-provider semaphore caps concurrent in-flight requests (WEB_RESEARCH_MAX_CONCURRENCY, default 4), so a depth="deep" research run can't fan out into dozens of simultaneous requests against one upstream. A short-TTL in-memory cache (WEB_RESEARCH_CACHE_TTL_SECONDS, default 300s) avoids repeating identical search/fetch calls within one run; failures and empty results are never cached, so a rate-limited provider still gets retried on the next call instead of being "stuck empty" for the TTL window.

No headless browsers. Zero Playwright, Selenium, Puppeteer, or proxy rotation. Smaller attack surface, smaller dependencies, no JVM/Chrome footprint. Jina does the heavy lifting on the few sites that need JS rendering.

Comparison with alternatives

Testing

.venv/bin/python tests/e2e_protocol.py

This launches the actual server, performs a real MCP initialize + tools/list handshake, then makes live JSON-RPC calls against every tool and verifies that:

• Real APIs return real data (not stubs)

• Each tool's response has the expected shape

• Error states are handled gracefully

• search_web without keys returns a clear "set API keys" message

The live subprocess check exercises all 10 registered tools. The deterministic matrix runs separately and covers provider fixtures, failure degradation, generated schemas, and MCP tool-call content contracts without network access:

.venv/bin/python -m unittest discover -s tests -p 'test_*.py'

Troubleshooting

Server starts but tools don't show in my MCP client

Check hermes mcp list (or equivalent). The server is registered with --command, which means Hermes will exec the launcher directly. Make sure the launcher is executable:

chmod +x bin/web-research-mcp

fetch_url returns truncated content

By design — 20k char cap protects your context window. For longer reads, fetch the page yourself and pass excerpts to search_web for follow-up questions, or split into sections via multiple calls.

search_web returns "No web results. This is likely because no API key is configured"

You need at least one of BRAVE_API_KEY or TAVILY_API_KEY set in web-research.env for general web search. The other 9 tools (including the six vertical providers, fetch_url, and the three deep-research tools) work without those keys; JINA_API_KEY is optional for higher fetch limits.

Stack Exchange returns 400 Bad Request

If you've configured a custom filter parameter, the API rejects unknown filter IDs. Use the default filter (omit the param) — it returns more fields than you need but everything works. This server uses the default.

Server crashes on first launch

Check stderr for the actual traceback. Common cause: Python <3.10. Check with python3 --version.

Rate limits

Provider calls use bounded retries for transient network errors, HTTP 5xx responses, and HTTP 429 responses. A provider that remains unavailable is isolated and returns no results, so other sources and deep-research phases can still complete. Configure the behavior with WEB_RESEARCH_TIMEOUT_SECONDS, provider-specific overrides such as WEB_RESEARCH_TIMEOUT_WIKIPEDIA_SECONDS or WEB_RESEARCH_TIMEOUT_JINA_SECONDS, WEB_RESEARCH_MAX_RETRIES, WEB_RESEARCH_RETRY_BACKOFF_SECONDS, and WEB_RESEARCH_MAX_BACKOFF_SECONDS. Exhausted 429s are reported as rate-limited; multi-source responses identify unavailable providers when partial results remain.

Concurrent fan-out per provider is capped by WEB_RESEARCH_MAX_CONCURRENCY (default 4), with per-provider overrides such as WEB_RESEARCH_MAX_CONCURRENCY_JINA. Search/fetch results are cached in-memory for WEB_RESEARCH_CACHE_TTL_SECONDS (default 300, set to 0 to disable) up to WEB_RESEARCH_CACHE_MAX_ENTRIES (default 256) entries; set either to 0 to turn caching off entirely.

Each keyless API has its own limits. If you hit them:

• Wikipedia: ~200 req/min, identify yourself with a real User-Agent (this server sends one)

• arXiv: ~1 req/3s for unauthenticated, please back off

• Hacker News Algolia: 10k req/hour with API key, 5k without

• Stack Exchange: 300 req/day without key (plenty for research sessions)

• Crossref: please add mailto in User-Agent (this server does), then it's polite-pool unlimited

Development

Project layout

web-research-mcp/
├── bin/
│   └── web-research-mcp          # Launcher: venv bootstrap + exec
├── src/web_research/
│   ├── __init__.py
│   ├── server.py                  # MCPServer + 10 @app.tool functions
│   └── providers.py               # 7 search backends + Result dataclass
├── tests/
│   └── e2e_protocol.py            # Real subprocess JSON-RPC test
├── web-research.env.example       # API key template
├── pyproject.toml                 # PEP 621, uv-installable
├── README.md
├── CHANGELOG.md
├── LICENSE
└── .gitignore

Adding a new tool

1. Add an async function to providers.py:

   async def search_my_source(query: str, max_results: int, client: httpx.AsyncClient) -> list[Result]:
       try:
           # ... your HTTP call ...
       except Exception as e:
           print(f"[my_source] error: {e}", flush=True)
           return []
       return [Result(title=..., url=..., snippet=..., source="my_source")]

2. Register it in server.py:

   @app.tool(name="search_my_source", description="...", annotations=ToolAnnotations(readOnlyHint=True, openWorldHint=True))
   async def search_my_source(query: Annotated[str, Field(description="Search query")], max_results: Annotated[int, Field(ge=1, le=10, default=5)] = 5) -> str:
       async with await _new_client() as client:
           res = await providers.search_my_source(query, max_results, client)
       return _format_results(query, res, "my_source") if res else f"No my_source results for: {query}"

3. Add a live test case in tests/e2e_protocol.py.

4. Update the README's Tools section.

Coding style

• Python 3.10+, async-first

• Type hints everywhere; let Pydantic derive the MCP JSON schema

• Every provider wraps its network call in try/except and degrades to []

• Per-call HTTP client (_new_client()) — don't share across calls in stdio mode

Contributing

PRs welcome. Before opening one:

1. Run the e2e test against a live install: .venv/bin/python tests/e2e_protocol.py

2. Add a test case for any new tool

3. Keep providers.py independent of MCP-specific types — it should be reusable as a plain Python module

4. Don't add dependencies on headless browsers or proxy rotation — that violates the project's thesis

For major changes, open an issue first.

License

MIT — see LICENSE.

Credits

• Built on the Model Context Protocol by Anthropic

• Uses Jina Reader for clean page fetching

• Search APIs: Brave, Tavily, Wikipedia, arXiv, Crossref, Hacker News Algolia, Stack Exchange