Skip to main content
Glama
sydasif

Web Search MCP

by sydasif
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

Web Search MCP

Python 3.11+ Code Style: Ruff FastMCP HOL Guard Scanner

A comprehensive Model Context Protocol (MCP) server built with FastMCP that provides LLMs with real-time, high-fidelity access to the web. This server aggregates multiple search engines, social platforms, and developer tools into a single interface, allowing AI agents to perform deep research, track community sentiment, and analyze technical documentation.

Design docs → wiki — tool selection guide, decision matrix, recommended workflows, tools status & known quirks, plugin setup, and development standards.

🚀 Features

The server provides a diverse suite of tools categorized by their primary use case:

🌐 General Web Search & Retrieval

Tool

Description

Best For

search_web

Fast web search via DuckDuckGo or Exa (SDK). Supports domain-scoping, date filtering, news mode, and geographic region. Default auto-provider tries DDG first, falls back to Exa.

Quick lookups, high-volume searches, pagination, broad coverage

fetch_page

High-fidelity text extraction from URLs with bot-detection bypass, SSRF protection (blocks private/internal IPs), and multiple output formats.

Deep reading of search results, standalone URL fetching

💬 Social & Community Intelligence

Tool

Description

Best For

search_reddit

Keyless search for community discussions, opinions, and real-world user experiences via RSS + Shreddit enrichment.

Product reviews, community sentiment, troubleshooting

search_hackernews

Technical discourse, startup news, and developer opinions via the Algolia HN API.

Tech news, startup discussions, developer opinions

search_github

Search for Issues and PRs to track bugs, feature requests, and community sentiment. Requires gh CLI or GITHUB_TOKEN.

Bug tracking, feature requests, community sentiment

get_github_issue

Fetch full conversation threads from GitHub Issues/PRs, sorted by reactions with author/date/reactions metadata.

Deep-diving into specific issues/PRs

search_x

Real-time discourse and breaking news via Xquik API or vendored Bird CLI (requires session cookies or API key).

Breaking news, community reactions, engagement signals

search_linkedin

Search people, companies, jobs, posts via DuckDuckGo + Jina Reader (r.jina.ai). No API key needed.

Professional profiles, company research, job search

🎓 Academic & Reference

Tool

Description

Best For

search_arxiv

Specialized search for academic papers with Lucene field prefixes (au:, ti:, cat:, abs:).

Research papers, citations, literature reviews

search_wikipedia

Factual summaries and background research via the MediaWiki API.

Factual summaries, background research, citations

Related MCP server: DuckDuckGo MCP Server

📋 Prerequisites

Requirement

Version

Notes

Python

3.11+

Required

uv

Latest

Recommended for installation and environment management

Optional External Tools

Tool

Required For

Installation

gh CLI

Authenticated GitHub search & issue retrieval (higher rate limits)

brew install gh / github.com/cli/cli

Node.js

Vendored Bird CLI for X/Twitter search (not needed with XQUIK_API_KEY)

22+ recommended; brew install node@22 / nodejs.org

⚙️ Installation

You have three options depending on your use case:

Option A: Quick Run (via uvx)

Fastest way to try it out without cloning the repo. Add to your MCP client config:

{
  "mcpServers": {
    "Web-Research": {
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/sydasif/web-search-mcp.git",
        "web-search-mcp"
      ]
    }
  }
}

Option B: Permanent Install

Fastest startup times with a globally installed tool:

uv tool install git+https://github.com/sydasif/web-search-mcp.git

Then configure your MCP client:

{
  "mcpServers": {
    "Web-Research": {
      "command": "web-search-mcp"
    }
  }
}

Option C: Development Install

If you want to modify the code or contribute:

git clone https://github.com/sydasif/web-search-mcp.git
cd web-search-mcp
uv sync
uv run web-search-mcp

Verify It's Working

Once the server is running, try a simple search:

search_web(query="current weather in Tokyo")

🔐 Configuration & Authentication

Most tools work out of the box with zero configuration. The following environment variables are only needed for premium or authenticated features.

Environment Variables Reference

Variable

Required For

How to Get It

EXA_API_KEY

Exa AI semantic search (optional fallback)

Sign up at exa.ai

GITHUB_TOKEN

Higher GitHub API rate limits (optional)

Generate a GitHub PAT

AUTH_TOKEN

X/Twitter search via Bird CLI (required)

Session cookie from x.com (see below)

CT0

X/Twitter search via Bird CLI (required)

Session cookie from x.com (see below)

XQUIK_API_KEY

X/Twitter search via Xquik API (alternative to cookies)

Sign up at xquik.ai

Setting Up GitHub Authentication

Option 1 — Recommended: Use gh CLI

gh auth login

The server detects your local session automatically.

Option 2: Manual Token

export GITHUB_TOKEN="ghp_your_token_here"

Setting Up X/Twitter Authentication

X/Twitter search requires either session cookies or an API key.

Option 1 — Session Cookies (Bird CLI):

  1. Log into x.com in your browser.

  2. Open DevTools (F12) → Application (or Storage) → Cookies → x.com.

  3. Copy the values for auth_token and ct0.

  4. Export them in the shell where the MCP server runs:

    export AUTH_TOKEN="your_auth_token"
export CT0="your_ct0"

    Note: These are session cookies. If searches return 401s, refresh them by logging out and back in.

Option 2 — Xquik API Key (Recommended):

  1. Sign up at xquik.ai to get an API key.

  2. Export it:

    export XQUIK_API_KEY="your_xquik_key"

    This bypasses the Node.js Bird CLI dependency entirely.

Setting Up Exa AI (Optional)

Exa provides semantic search and JS-heavy page fallback:

export EXA_API_KEY="your_exa_key"

💡 Usage Examples

Web Research

# Broad search (auto: DDG first, falls back to Exa on error or zero results)
search_web(query="Latest NVIDIA H200 benchmarks")

# Force DDG explicitly
search_web(query="uv package manager", provider="ddg")

# Force Exa explicitly
search_web(query="uv package manager", provider="exa")

# Targeted documentation search
search_web(query="useEffect cleanup", domain="react.dev")

# News with region filter
search_web(query="elections", search_type="news", region="us-en", provider="exa")

# Date-filtered search
search_web(query="uv package manager", time_range="w", provider="auto")

# Deep read a page
fetch_page(url="https://docs.python.org/3/library/os.html")

Technical Analysis

# Track GitHub issues/PRs
search_github(query="uv package manager")

# Get full GitHub issue thread
get_github_issue(url="https://github.com/astral-sh/uv/issues/1")

Community Sentiment

# Reddit discussions
search_reddit(query="Best mechanical keyboards 2024", subreddits=["MechanicalKeyboards"])

# Hacker News technical discourse
search_hackernews(query="MCP server architecture")

# LinkedIn professional search
search_linkedin(query="site reliability engineer", content_type="people")
search_linkedin(query="machine learning startup", content_type="companies")
search_linkedin(query="kubernetes devops", content_type="jobs")
search_linkedin(query="AI agents", content_type="posts")

Academic Research

# arXiv paper search with field prefixes
search_arxiv(query="au:Goodfellow AND cat:cs.LG")
search_arxiv(query="transformer attention", sort_by="submitted_date")

# Wikipedia background research
search_wikipedia(query="Quantum computing")

🏗️ Project Structure

web_search_mcp/
├── server.py              # Entry point: FastMCP init, @mcp.tool registrations
├── search/                # Search engine implementations
│   ├── ddg.py             # DuckDuckGo search + trafilatura page fetch
│   └── exa.py             # Exa SDK search & content fetch (lazy-init client)
├── social/                # Community platform integrations
│   ├── github.py          # GitHub Search API + gh CLI issue rendering
│   ├── hackernews.py      # Algolia HN API + comment enrichment
│   ├── linkedin/          # LinkedIn search via DDG + Jina Reader
│   │   ├── __init__.py    # LinkedIn search tool registration
│   │   └── client.py      # DDG search + Jina Reader enrichment
│   ├── reddit/            # RSS + Shreddit keyless pipeline
│   │   ├── client.py      # HTTP client with RSS parsing
│   │   ├── parsers.py     # RSS/HTML parsers
│   │   └── shreddit.py    # Shreddit comment enrichment
│   └── x.py               # X/Twitter search via Xquik API or vendored Bird CLI
├── tools/                 # Specialized reference utilities
│   ├── arxiv.py           # arXiv paper search (Lucene field prefixes)
│   └── wikipedia.py       # Wikipedia MediaWiki API
├── _config/               # Settings, env vars, rate limits, depth tiers
│   ├── settings.py        # pydantic-settings (EXA_API_KEY, SEARCH_MCP_ prefix)
│   └── limits.py          # Per-platform quick/default/deep limits, timeouts
├── _http/                 # Shared HTTP + SSRF protection
│   └── client.py          # validate_url, http_client, get_json_client
├── _models/               # Pydantic request/response models
│   ├── requests.py        # SearchRequest
│   ├── responses.py       # ErrorResponse, SearchResponse, PageResponse
│   └── types.py           # Depth, ResponseFormat, SearchType, FetchOutputFormat
├── _utils/                # Shared helpers
│   ├── formatting.py      # Markdown formatters, date/epoch utils
│   ├── rate_limiter.py    # Token-bucket rate limiter
│   └── scoring.py         # Relevance scoring
└── vendor/                # Vendored third-party tools
    └── bird-search/       # Node.js CLI for X/Twitter search (fallback when XQUIK_API_KEY unset)

🛠️ Tool Implementation Flow

When adding a new tool:

  1. Implement logic in the appropriate module (search/, social/, or tools/)

  2. Define models in _models/ (request/response types)

  3. Register in server.py using @mcp.tool decorator with a clear docstring (serves as the tool's description for the LLM)

📐 Design Decisions

  • search-backend-split — Why search_web unifies DuckDuckGo and Exa behind a single provider parameter instead of exposing two separate tools.

🧪 Testing

# Run all tests
uv run pytest

# Run a single test file
uv run pytest tests/test_module.py

# Run a specific test
uv run pytest tests/test_module.py::test_function_name

# Run with coverage
uv run pytest --cov=web_search_mcp

🔧 Troubleshooting

Problem

Likely Cause

Solution

Auth errors on a tool

Env var not set in the server's shell

Export the variable in the same shell where the MCP server process runs

GitHub returns empty results

Not authenticated

Run gh auth login or set GITHUB_TOKEN

search_x returns 401

Expired X session cookies

Re-extract auth_token and ct0 from x.com

fetch_page blocked by Cloudflare

Bot detection

Try backend="curl" parameter

search_arxiv returns 503

Upstream arXiv maintenance

Wait a few minutes and retry

Tool says "Query cannot be empty"

Missing or blank query

Provide a non-empty search query

🤝 Contributing

  1. Fork the repository.

  2. Create a feature branch: git checkout -b feat/my-new-tool

  3. Ensure all tests pass: uv run pytest

  4. Submit a pull request with a detailed description of the changes.

📄 License

This project is licensed under the MIT License.

Available Tools

3 tools
fetch_pageA
Read-onlyIdempotent

Extracts the full text content from a web page URL. Use this to read the details of a specific result found via web_search.

Args: url: The URL to fetch and extract content from output_format: Format for extracted content ('csv', 'html', 'json', 'markdown', 'python', 'txt', 'xml', 'xmltei') include_metadata: Whether to include document metadata (title, author, date, etc.) include_tables: Whether to include table content in extraction include_comments: Whether to include comment content in extraction include_images: Whether to include image descriptions in extraction deduplicate: Whether to remove duplicated content max_length: Maximum length of content to return (default 15000) timeout: Request timeout in seconds (default 30) backend: HTTP backend to use ('httpx' for lightweight, 'curl' to bypass bot detection, 'auto' to try httpx first then fallback to curl)

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
output_formatNotxt
include_metadataNo
include_tablesNo
include_commentsNo
include_imagesNo
deduplicateNo
max_lengthNo
timeoutNo
backendNoauto

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate readOnlyHint=true, destructiveHint=false, idempotentHint=true, and openWorldHint=true. The description adds behavioral context by detailing backend options (curl for bot detection) and extraction options, but does not mention potential rate limits or failure scenarios.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with a concise summary and a usage sentence, followed by a well-structured bullet list of parameters. Every sentence adds value without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the tool's purpose and all parameters thoroughly. Given an output schema exists (not shown), the description does not need to detail return values. Minor gaps include lack of error handling or typical response behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description compensates by listing all parameters with explanations in the Args section, clarifying enums and defaults. However, some parameters like include_metadata lack details on what metadata is included.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Extracts the full text content from a web page URL', which is a specific verb and resource. It also distinguishes itself from siblings by recommending use after web_search.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly advises using it to read details of a specific result from web_search, providing clear context. However, it does not include exclusions or alternatives beyond the implicit sibling distinction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_docsA
Read-onlyIdempotent

Searches specifically for technical documentation or content on a specific domain.

Args: query: What you're looking for domain: The domain to search (e.g. 'docs.python.org', 'stackoverflow.com')

Returns: Search results from the specified domain

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYes
domainNodocs.python.org

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already disclose readOnlyHint=true, destructiveHint=false, idempotentHint=true, openWorldHint=true. The description adds that it searches a domain and returns results, but does not elaborate on behavioral traits like pagination, rate limits, or result structure beyond the output schema. With annotations carrying the safety profile, the description's added value is moderate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with 6 sentences, front-loading the purpose, then clearly listing parameter descriptions and return value. Every sentence adds value without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (2 parameters, both described), annotations covering behavioral aspects, and existence of an output schema, the description provides sufficient context for the agent to correctly select and invoke the tool. It covers purpose, parameters, and return high-level description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description bears full burden. It provides meaningful explanations for both parameters: 'query: What you're looking for' and 'domain: The domain to search (e.g. 'docs.python.org', 'stackoverflow.com').' The query description is slightly vague but adequate, and the domain example adds clarity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool searches for technical documentation on a specific domain, using a specific verb ('searches') and resource ('technical documentation'). This distinguishes it from sibling tools like web_search (general web search) and fetch_page (page retrieval).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly implies usage for domain-specific searches, providing a clear context but no explicit when-not-to-use or alternative comparisons. It states it searches 'specifically for technical documentation... on a specific domain,' which guides the agent to use it over general search when targeting a particular site.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 3 tool updatesv0.2.0
    • First observedfetch_page
    • First observedsearch_docs
    • First observedweb_search
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.3/5.0

Scored across 3 tools

Disambiguation5/5

Each tool has a clearly distinct purpose: web_search for general web/news search, search_docs for domain-specific search, and fetch_page for retrieving content from a URL. No overlap.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern (fetch_page, search_docs, web_search) using snake_case, which is predictable and clear.

Tool Count4/5

With 3 tools, the server is minimal but well-scoped for its purpose (search and fetch). It covers the essential operations without being overly complex or thin.

Completeness5/5

The tool surface covers the full workflow: general search, specialized search, and content extraction. No obvious missing operations; web_search returns structured results for further processing.

Maintenance

ActivityActive
ResponsivenessWithin a week

Resources

Related MCP Connectors

Related MCP Servers

  • DuckDuckGo MCP Server

    988664li-star
    A
    license
    B
    quality
    D
    maintenance
    Enables web search through DuckDuckGo and webpage content fetching with intelligent text extraction. Features built-in rate limiting and LLM-optimized result formatting for seamless integration with language models.
    2
    MIT

  • ORZ MCP

    sunwu51
    F
    license
    A
    quality
    D
    maintenance
    Provides web search capabilities across multiple engines and webpage content fetching to simplified Markdown. It enables AI assistants to access real-time internet information and extract text from specific URLs.
    2
    13
    -

Appeared in Searches