# Web Tools MCP

Multi-engine web search and content extraction MCP server for AI agents. Drop-in replacement for native `web_search` / `web_fetch` tools, compatible with any LLM connected to Grok Build, Claude Code, Cursor, or other MCP hosts.

![banner](readme_images/01-hero-banner.svg)

## Features

- **10 classified data sources** — Bing, 360 Search, Wikipedia, arXiv, Semantic Scholar, StackOverflow, Hacker News, Brave (optional), comprehensive hybrid, code search
- **4-way parallel multi-angle search** — each query auto-expands into 4 complementary search angles, executed concurrently via `asyncio.gather`
- **Content-first extraction** — Jina Reader returns clean Markdown from any URL; BeautifulSoup fallback for html pages
- **Smart TTL caching** — search results cached for 5 min, fetched pages for 30 min; ~3000x speedup on cache hits
- **Zero API keys required** — all sources work out of the box; optional Brave Search API key for independent search index
- **Drop-in compatible** — same tool names and parameter schema as native Grok `web_search`/`web_fetch`

## Quick Start

### Prerequisites

- Python 3.11+
- Grok Build / Claude Code / Cursor / any MCP-compatible host

### Install

```bash
git clone https://github.com/hellowind777/web-tools-mcp.git
cd web-tools-mcp
pip install -r requirements.txt
```

### Configure

Add to `~/.grok/config.toml` (or equivalent MCP host config):

```toml
[mcp_servers.web-tools]
command = "python"
args = ["/path/to/web-tools-mcp/mcp_server/server.py"]
enabled = true
startup_timeout_sec = 30
tool_timeout_sec = 30
```

Restart your MCP host. The tools `web-tools__web_search` and `web-tools__web_fetch` become available to your model.

### Optional: Brave Search API

For higher-quality results from an independent search index:

```bash
export BRAVE_API_KEY="your-key-here"
```

Get a free key at [brave.com/search/api](https://brave.com/search/api/) (2000 queries/month free).

## Tools

### web_search

Classified search with 10 source options. Automatically runs 4-angle parallel queries for web/comprehensive/news sources.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `query` | string | required | Search query |
| `source` | string | `"web"` | Data source (see below) |
| `num_results` | integer | 10 | Results to return |

**Available sources:**

| Source | Backend | Best for |
|--------|---------|----------|
| `web` | Bing (or Brave if key set) | General search |
| `comprehensive` | Bing + Wikipedia + arXiv | Research topics |
| `wikipedia` | Wikipedia API | Definitions, facts |
| `scholar` | Semantic Scholar → arXiv fallback | CS papers, citations |
| `academic` | arXiv API | Preprints, papers |
| `code` | GitHub via Bing | Repos, issues |
| `stackoverflow` | StackExchange API | Q&A with scores |
| `hackernews` | HN Algolia API | Tech news, discussions |
| `chinese` | 360 Search (so.com) | Chinese-language content |
| `news` | Bing (semantic alias) | News queries |

### web_fetch

Fetch and extract readable content from any URL.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `url` | string | required | URL to fetch |
| `max_length` | integer | 8000 | Max characters returned |

Uses Jina Reader as primary backend (clean Markdown, handles JS pages), falls back to BeautifulSoup HTML extraction.

## Architecture

```
web_search(query, source)         web_fetch(url)
         │                              │
    ┌────▼────┐              ┌──────────▼──────────┐
    │  Cache   │              │   Jina Reader (1st)  │
    │ TTL 300s │              │  BeautifulSoup (2nd) │
    └────┬────┘              └──────────┬──────────┘
         │                              │
    ┌────▼──────────────────────────────▼──┐
    │         Data Source Layer             │
    │                                       │
    │  Bing ─── 360 ─── Wikipedia ─── arXiv │
    │  Scholar ─── StackOverflow ─── HN     │
    │  Brave (optional) ─── Comprehensive   │
    └───────────────────────────────────────┘
```

For `web`/`comprehensive`/`news` sources, each query is expanded into 4 parallel searches:
1. Original query
2. Official documentation
3. Community sources (GitHub/CSDN/Zhihu)
4. Latest (year-aware freshness)

## Comparison

| | Web Tools MCP | Tavily MCP | Brave MCP | Exa MCP |
|---|---|---|---|---|
| Free tier | ✅ Unlimited | 1000/mo | 2000/mo | Trial only |
| Sources | 10 | 1 | 1 | 1 |
| Parallel multi-angle | ✅ 4-way | `search_depth` | Single query | Single query |
| Content extraction | ✅ Jina+BS4 | `extract` | None | None |
| API key required | ❌ | ✅ | ✅ | ✅ |
| Chinese support | ✅ 360 search | ❌ | ❌ | ❌ |
| Academic | ✅ arXiv+Scholar | ❌ | ❌ | ❌ |

## Development

```bash
# Install dev dependencies
pip install -r requirements.txt

# Run tests (manual — MCP protocol)
python mcp_server/server.py
```

## License

Apache-2.0