Web Tools MCP

Multi-engine web search and content extraction MCP server for AI agents. Drop-in replacement for native web_search / web_fetch tools, compatible with any LLM connected to Grok Build, Claude Code, Cursor, or other MCP hosts.

Features

• 10 classified data sources — Bing, 360 Search, Wikipedia, arXiv, Semantic Scholar, StackOverflow, Hacker News, Brave (optional), comprehensive hybrid, code search

• 4-way parallel multi-angle search — each query auto-expands into 4 complementary search angles, executed concurrently via asyncio.gather

• Content-first extraction — Jina Reader returns clean Markdown from any URL; BeautifulSoup fallback for html pages

• Smart TTL caching — search results cached for 5 min, fetched pages for 30 min; ~3000x speedup on cache hits

• Zero API keys required — all sources work out of the box; optional Brave Search API key for independent search index

• Drop-in compatible — same tool names and parameter schema as native Grok web_search/web_fetch

Related MCP server: Deep Search MCP Server

Quick Start

Prerequisites

• Python 3.11+

• Grok Build / Claude Code / Cursor / any MCP-compatible host

Install

git clone https://github.com/hellowind777/web-tools-mcp.git
cd web-tools-mcp
pip install -r requirements.txt

Configure

Add to ~/.grok/config.toml (or equivalent MCP host config):

[mcp_servers.web-tools]
command = "python"
args = ["/path/to/web-tools-mcp/mcp_server/server.py"]
enabled = true
startup_timeout_sec = 30
tool_timeout_sec = 30

Restart your MCP host. The tools web-tools__web_search and web-tools__web_fetch become available to your model.

Optional: Brave Search API

For higher-quality results from an independent search index:

export BRAVE_API_KEY="your-key-here"

Get a free key at brave.com/search/api (2000 queries/month free).

Tools

web_search

Classified search with 10 source options. Automatically runs 4-angle parallel queries for web/comprehensive/news sources.

Available sources:

web_fetch

Fetch and extract readable content from any URL.

Uses Jina Reader as primary backend (clean Markdown, handles JS pages), falls back to BeautifulSoup HTML extraction.

Architecture

web_search(query, source)         web_fetch(url)
         │                              │
    ┌────▼────┐              ┌──────────▼──────────┐
    │  Cache   │              │   Jina Reader (1st)  │
    │ TTL 300s │              │  BeautifulSoup (2nd) │
    └────┬────┘              └──────────┬──────────┘
         │                              │
    ┌────▼──────────────────────────────▼──┐
    │         Data Source Layer             │
    │                                       │
    │  Bing ─── 360 ─── Wikipedia ─── arXiv │
    │  Scholar ─── StackOverflow ─── HN     │
    │  Brave (optional) ─── Comprehensive   │
    └───────────────────────────────────────┘

For web/comprehensive/news sources, each query is expanded into 4 parallel searches:

1. Original query

2. Official documentation

3. Community sources (GitHub/CSDN/Zhihu)

4. Latest (year-aware freshness)

Comparison

Development

# Install dev dependencies
pip install -r requirements.txt

# Run tests (manual — MCP protocol)
python mcp_server/server.py

License

Apache-2.0