Web Search MCP

11 tools that give your LLM real-time access to the web — from simple Google-style searches to deep AI-powered research. Search the web, read articles, check Reddit, browse Hacker News, track GitHub issues, follow X/Twitter discussions, explore prediction markets, and more.

No API keys required for most tools. Works out of the box.

What This Does

When you ask your LLM "what's the latest on X?", it can only answer from its training data. That data is months or years old. Web Search MCP fixes this by connecting your LLM to live web sources through the Model Context Protocol.

Here's what you get:

• Search the web — text and news search via DuckDuckGo

• Read any article — extract clean text from URLs, stripping ads and clutter

• Search Reddit — find real discussions and community sentiment

• Search Hacker News — developer opinions and tech discourse

• Search GitHub — find issues, PRs, and bug reports across repos

• Search X/Twitter — real-time posts and breaking news

• Search prediction markets — odds and crowd-sourced probability estimates from Polymarket

• AI-powered research — let Groq's models search, browse, and synthesize for you

Most tools are free and require no API keys. A few need setup (detailed below).

Related MCP server: DuckDuckGo MCP Server

Quick Start (2 minutes)

Step 1: Install

uv tool install git+https://github.com/sydasif/web-search-mcp.git

Step 2: Add to Your MCP Client

Paste this into your MCP client config (Claude Desktop, VS Code, Cursor, etc.):

{
  "mcpServers": {
    "web-search": {
      "command": "web-search-mcp"
    }
  }
}

That's it. You now have web search in your LLM.

Step 3: Optional — Enable More Tools

Want Reddit, Hacker News, GitHub, Polymarket, or X/Twitter search? They work out of the box — no extra config needed.

Want AI-powered research via Groq? Add an API key:

{
  "mcpServers": {
    "web-search": {
      "command": "web-search-mcp",
      "env": {
        "SEARCH_MCP_GROQ_API_KEY": "gsk_your_key_here"
      }
    }
  }
}

Get a free Groq API key here — Groq's free tier is generous.

What Can I Do With It?

Here are some real-world examples to get you started.

Search the Web

Ask your LLM: "Search for the latest Python 3.13 features"

Behind the scenes:
  web_search("latest Python 3.13 features", search_type="text")

Read an Article

Ask your LLM: "Read this article and summarize it: https://docs.python.org/3/whatsnew/3.13.html"

Behind the scenes:
  fetch_page(url="https://docs.python.org/3/whatsnew/3.13.html")

Search Reddit for Real Opinions

Ask your LLM: "What do people on Reddit think about Rust vs Go for web backends?"

Behind the scenes:
  reddit_search(query="Rust vs Go web backend", subreddits=["rust", "golang"])

Search Hacker News for Tech Discourse

Ask your LLM: "Find Hacker News discussions about LLM fine-tuning in 2024"

Behind the scenes:
  hackernews_search(query="LLM fine-tuning 2024")

Search GitHub Issues

Ask your LLM: "Find issues about memory leaks in React Server Components"

Behind the scenes:
  github_search(query="memory leak React Server Components")

Search X/Twitter for Breaking News

Ask your LLM: "What's trending about AI on X right now?"

Behind the scenes:
  x_search(query="AI", depth="default")

Search Prediction Markets

Ask your LLM: "What are the current odds on the next US presidential election?"

Behind the scenes:
  polymarket_search(topic="US presidential election")

AI-Powered Deep Research

Ask your LLM: "Research the state of quantum computing in 2025 and validate the findings"

Behind the scenes:
  groq_research(query="state of quantum computing 2025")

Choosing the Right Tool

Not sure which tool to use? Here's a quick guide:

How the Tools Connect

The best results come from combining tools. Here's the workflow that works best:

Step 1: Discover (free, fast) Use DuckDuckGo tools to find relevant sources quickly.

web_search("Python async performance benchmarks 2025")

Step 2: Read (free, fast) Pull the clean text from the most promising URLs.

fetch_page(url="https://example.com/benchmark-results")

Step 3: Validate (requires Groq API key) Let AI cross-check and expand on what you found.

groq_research("Validate these Python async benchmark findings and find any contradictions")

This discovery → reading → validation pattern gives you the most reliable results.

Environment Variables

Getting X/Twitter Cookies

The x_search tool needs two cookies from a logged-in X session. Here's how to get them:

1. Log into x.com in your browser

2. Open DevTools (press F12)

3. Go to Application → Cookies → https://x.com

4. Find and copy the values for auth_token and ct0

5. Add them to your MCP config as AUTH_TOKEN and CT0

These cookies last about 24 hours. If X search stops working, refresh them.

Fetch Backend Options

The fetch_page tool can use different HTTP backends. Some websites block automated requests, so having options helps.

The auto backend handles this for you — it tries the fast option first and automatically switches if needed.

Tool Reference

DuckDuckGo Tools (free, no API key)

Reddit Search (free, no API key)

Hacker News Search (free, no API key)

GitHub Search (free, no API key)

Polymarket Search (free, no API key)

X/Twitter Search (requires cookies)

Groq Tools (requires API key)

Which Groq model should I use?

• groq/compound-mini (default) — Faster, cheaper, handles most tasks well. Start here.

• groq/compound — More thorough, uses more tools internally. Use for complex research.

• openai/gpt-oss-20b — For groq_browse only. Good balance of speed and quality.

• openai/gpt-oss-120b — For groq_browse only. Best quality, but slower.

Troubleshooting

"No results found" for web search

DuckDuckGo sometimes returns empty results for very specific or unusual queries. Try:

• Broadening your search terms

• Removing special characters

• Using a simpler query

Cloudflare blocks when reading URLs

This happens when a website detects automated requests. The fetch_page tool handles this automatically:

• Default backend (auto) tries httpx first, then falls back to curl with Chrome impersonation

• If you're still blocked, try setting backend="curl" explicitly

X/Twitter search not working

X cookies expire after ~24 hours. To fix:

1. Log into x.com again

2. Get fresh auth_token and ct0 cookies (see Getting X/Twitter Cookies above)

3. Update your MCP config and restart the server

Groq tools returning errors

• Check that SEARCH_MCP_GROQ_API_KEY is set correctly

• Make sure your Groq API key has credits (free tier available)

• If you see "Query too long", shorten your query to under 150 characters

Rate limiting (429 errors)

DuckDuckGo will block you if you search too fast. The built-in rate limiter helps, but:

• Reduce SEARCH_MCP_RATE_LIMIT_SEARCH if you're hitting limits

• Add small delays between rapid searches

• Use max_results to get fewer results per query

Server won't start

Make sure you're using uv (not pip):

# Install
uv tool install git+https://github.com/sydasif/web-search-mcp.git

# Run directly
uv run web-search-mcp

Frequently Asked Questions

Do I need API keys?

For most tools, no. Web search, Reddit, Hacker News, GitHub, and Polymarket all work without any API key. You only need a Groq API key for the AI-powered tools (browse, research, analyze_page) and X/Twitter cookies for X search.

Is this free?

The DuckDuckGo, Reddit, Hacker News, GitHub, and Polymarket tools are completely free. Groq has a generous free tier. X/Twitter uses your browser session.

How is this different from just searching in the browser?

Your LLM can't access the web on its own. This server gives it that ability through the Model Context Protocol. The LLM can search, read, analyze, and synthesize information in a single conversation turn.

Can I use this with any LLM client?

Yes, as long as it supports MCP — Claude Desktop, VS Code with Copilot, Cursor, Zed, and more.

What about rate limits?

The server includes built-in rate limiting to avoid getting blocked by DuckDuckGo. You can configure the limits via environment variables (defaults: 30 searches/min, 20 fetches/min).

Development

Setup

git clone https://github.com/sydasif/web-search-mcp.git
cd web-search-mcp
uv sync

Run Tests

# All tests
uv run pytest

# With coverage
uv run pytest --cov=web_search_mcp

# Single test file
uv run pytest tests/test_models.py

# Specific test
uv run pytest tests/test_models.py::test_search_request_valid_defaults

Linting

uv run ruff check .
uv run ruff check --fix  # auto-fix issues

Type Checking

uv run mypy web_search_mcp/

Run the Server

uv run web-search-mcp

License

MIT License

Acknowledgment: This project is built on publicly available APIs, open-source libraries, and community research. DuckDuckGo, Reddit RSS, Hacker News Algolia API, GitHub REST API, Polymarket Gamma API, and Groq's API are all used in accordance with their respective terms.