Web Search MCP
The server exposes three read-only web research tools: unified web/news search, page content extraction, and domain-specific documentation search.
web_search: run web or news queries with region/date/safesearch filters, pagination, backend selection, and markdown or structured JSON output.fetch_page: extract page content from a URL in multiple formats, with options for metadata, tables, comments, images, deduplication, max length, timeout, and HTTP/curl backends.search_docs: search technical documentation or content on a specified domain, defaulting todocs.python.org.All tools are read-only, idempotent, open-world, non-destructive, and can return structured error responses.
Note: the README advertises many additional social, academic, and reranking tools, but the provided server schema only exposes these three tools.
Provides web and news search capabilities using DuckDuckGo, supporting text and news search with configurable parameters such as region, time range, and result format.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Web Search MCP search for latest AI news"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
OmniSearch
A comprehensive Model Context Protocol (MCP) server built with FastMCP that provides LLMs with real-time, high-fidelity access to the web. This server aggregates multiple search engines, social platforms, and developer tools into a single interface, allowing AI agents to perform deep research, track community sentiment, and analyze technical documentation.
Design docs ā wiki ā tool selection guide, decision matrix, recommended workflows, tools status & known quirks, plugin setup, and development standards.
š Features
The server provides a diverse suite of tools categorized by their primary use case:
š General Web Search & Retrieval
Tool | Description | Best For |
| Fast web search via DuckDuckGo or Exa (SDK). Supports domain-scoping, date filtering, news mode, and geographic region. Default auto-provider tries DDG first, falls back to Exa. | Quick lookups, high-volume searches, pagination, broad coverage |
| High-fidelity text extraction from URLs with bot-detection bypass, SSRF protection (blocks private/internal IPs), and multiple output formats. | Deep reading of search results, standalone URL fetching |
š¬ Social & Community Intelligence
Tool | Description | Best For |
| Keyless search for community discussions, opinions, and real-world user experiences via RSS + Shreddit enrichment + Arctic-Shift score backfill. | Product reviews, community sentiment, troubleshooting |
| Technical discourse, startup news, and developer opinions via the Algolia HN API. | Tech news, startup discussions, developer opinions |
| Search for Issues and PRs to track bugs, feature requests, and community sentiment. Requires | Bug tracking, feature requests, community sentiment |
| Fetch full conversation threads from GitHub Issues/PRs, sorted by reactions with author/date/reactions metadata. | Deep-diving into specific issues/PRs |
| Real-time discourse and breaking news via Xquik API or vendored Bird CLI (requires session cookies or API key). | Breaking news, community reactions, engagement signals |
| Search people, companies, jobs, posts via DuckDuckGo + Jina Reader (r.jina.ai). No API key needed. | Professional profiles, company research, job search |
| Curated AI discourse clusters from ~1000 high-signal X accounts via digg-pp-cli. TLDRs + top X posts per cluster. | AI trends, curated high-signal discourse |
| Editorial tech news from live archive (2005+) via techmeme-pp-cli. Headlines with source attribution. | Tech industry news, major announcements |
| Prediction markets with real-money odds via public Gamma API. Multi-query expansion + acronym credit. | Forecasting, sentiment, real-money odds on events |
š Academic & Reference
Tool | Description | Best For |
| Specialized search for academic papers with Lucene field prefixes ( | Research papers, citations, literature reviews |
| Factual summaries and background research via the MediaWiki API. | Factual summaries, background research, citations |
| Rerank candidate texts against a query by relevance. Three backends: | Post-search relevance refinement, LLM context prioritization |
Related MCP server: DuckDuckGo MCP Server
š Prerequisites
Requirement | Version | Notes |
Python | 3.11+ | Required |
Latest | Recommended for installation and environment management |
Optional External Tools
Tool | Required For | Installation |
| Authenticated GitHub search & issue retrieval (higher rate limits) |
|
Node.js | Vendored Bird CLI for X/Twitter search (not needed with | 22+ recommended; |
| Digg AI 1000 curated clusters (free, no auth) |
|
| Techmeme tech news archive search (free, no auth) |
|
| Local corpus PDF extraction (optional, for future |
|
Optional Python Extras
Extra | Install Command | Enables |
|
| Neural reranking backend (~23MB ONNX, CPU, no torch) |
|
| sentence-transformers backend (requires torch) |
|
| BM25/sklearn evaluation utilities |
āļø Installation
You have three options depending on your use case:
Option A: Quick Run (via uvx)
Fastest way to try it out without cloning the repo. Add to your MCP client config:
{
"mcpServers": {
"OmniSearch": {
"command": "uvx",
"args": [
"--from",
"git+https://github.com/sydasif/web-search-mcp.git",
"omnisearch"
]
}
}
}Option B: Permanent Install
Fastest startup times with a globally installed tool:
uv tool install git+https://github.com/sydasif/web-search-mcp.gitThen configure your MCP client:
{
"mcpServers": {
"OmniSearch": {
"command": "omnisearch"
}
}
}Option C: Development Install
If you want to modify the code or contribute:
git clone https://github.com/sydasif/web-search-mcp.git
cd web-search-mcp
uv sync
uv run omnisearchVerify It's Working
Once the server is running, try a simple search:
search_web(query="current weather in Tokyo")Option D: Plugin Install
Install as a plugin for quick access from any project:
claude plugin marketplace add sydasif/web-search-mcp
claude plugin install --scope user web-search@sydasif-web-search-mcpOMP Plugin Install
omp plugin marketplace add sydasif/web-search-mcp
omp plugin install --scope user web-search@sydasif-web-search-mcpš Configuration & Authentication
Most tools work out of the box with zero configuration. The following environment variables are only needed for premium or authenticated features.
Environment Variables Reference
Variable | Required For | How to Get It |
| Exa AI semantic search (optional fallback) | Sign up at exa.ai |
| Higher GitHub API rate limits (optional) | Generate a GitHub PAT |
| X/Twitter search via Bird CLI (required) | Session cookie from x.com (see below) |
| X/Twitter search via Bird CLI (required) | Session cookie from x.com (see below) |
| X/Twitter search via Xquik API (alternative to cookies) | Sign up at xquik.ai |
| Optional SearXNG instance for keyless web search fallback | Your SearXNG instance URL |
| Local document directories for private search (colon-separated paths) ā planned | Paths to your |
| Fixed default source set (comma-separated, e.g. | Source names from available sources |
| Opt-in additive sources (e.g. | Source names |
| Hard-exclude sources (e.g. | Source names |
| Enable default fact-checking for Polymarket odds, GitHub stars, etc. ā planned |
|
Setting Up GitHub Authentication
Option 1 ā Recommended: Use gh CLI
gh auth loginThe server detects your local session automatically.
Option 2: Manual Token
export GITHUB_TOKEN="ghp_your_token_here"Setting Up X/Twitter Authentication
X/Twitter search requires either session cookies or an API key.
Option 1 ā Session Cookies (Bird CLI):
Log into
x.comin your browser.Open DevTools (F12) ā Application (or Storage) ā Cookies ā
x.com.Copy the values for
auth_tokenandct0.Export them in the shell where the MCP server runs:
export AUTH_TOKEN="your_auth_token" export CT0="your_ct0"Note: These are session cookies. If searches return 401s, refresh them by logging out and back in.
Option 2 ā Xquik API Key (Recommended):
Sign up at xquik.ai to get an API key.
Export it:
export XQUIK_API_KEY="your_xquik_key"This bypasses the Node.js Bird CLI dependency entirely.
Setting Up Exa AI (Optional)
Exa provides semantic search and JS-heavy page fallback:
export EXA_API_KEY="your_exa_key"š” Usage Examples
Web Research
# Broad search (auto: DDG first, falls back to Exa on error or zero results)
search_web(query="Latest NVIDIA H200 benchmarks")
# Force DDG explicitly
search_web(query="uv package manager", provider="ddg")
# Force Exa explicitly
search_web(query="uv package manager", provider="exa")
# Targeted documentation search
search_web(query="useEffect cleanup", domain="react.dev")
# News with region filter
search_web(query="elections", search_type="news", region="us-en", provider="exa")
# Date-filtered search
search_web(query="uv package manager", time_range="w", provider="auto")
# Deep read a page
fetch_page(url="https://docs.python.org/3/library/os.html")Technical Analysis
# Track GitHub issues/PRs
search_github(query="uv package manager")
# Get full GitHub issue thread
get_github_issue(url="https://github.com/astral-sh/uv/issues/1")Community Sentiment
# Reddit discussions
search_reddit(query="Best mechanical keyboards 2024", subreddits=["MechanicalKeyboards"])
# Hacker News technical discourse
search_hackernews(query="MCP server architecture")
# LinkedIn professional search
search_linkedin(query="site reliability engineer", content_type="people")
search_linkedin(query="machine learning startup", content_type="companies")
search_linkedin(query="kubernetes devops", content_type="jobs")
search_linkedin(query="AI agents", content_type="posts")Academic Research
# arXiv paper search with field prefixes
search_arxiv(query="au:Goodfellow AND cat:cs.LG")
search_arxiv(query="transformer attention", sort_by="submitted_date")
# Wikipedia background research
search_wikipedia(query="Quantum computing")AI Trends & Forecasting
# Digg AI 1000 - curated clusters from high-signal X accounts
search_digg(query="AI agents")
search_digg(query="Claude Code", depth="deep")
# Techmeme - editorial tech news
search_techmeme(query="Apple WWDC")
search_techmeme(query="AI funding", depth="deep")
# Polymarket - prediction markets with real-money odds
search_polymarket(query="US election 2024")
search_polymarket(query="Fed rate cut")
search_polymarket(query="AI breakthrough", depth="deep") # expanded query searchResult Refinement
# Rerank search snippets to top 3 most relevant (default: token_overlap, zero-dep)
rerank_results(
query="how to cancel asyncio tasks",
documents=["snippet1...", "snippet2...", "snippet3..."],
top_n=3,
)
# Neural reranking with FlashRank (uv pip install 'web_search_mcp[flashrank]')
rerank_results(query="python async", documents=["..."], backend="flashrank")
# Hybrid scoring with metadata boosts and deduplication
rerank_results(
query="kubernetes networking",
documents=["..."],
backend="hybrid",
documents_meta=[{"engagement": 500, "freshness_days": 3, "authority": 0.9, "parent_id": "doc1"}],
top_n=5,
)šļø Project Structure
web_search_mcp/
āāā server.py # Entry point: FastMCP init, @mcp.tool registrations
āāā search/ # Search engine implementations
ā āāā ddg.py # DuckDuckGo search + trafilatura page fetch
ā āāā exa.py # Exa SDK search & content fetch (lazy-init client)
āāā social/ # Community platform integrations
ā āāā github.py # GitHub Search API + gh CLI issue rendering
ā āāā hackernews.py # Algolia HN API + comment enrichment
ā āāā linkedin/ # LinkedIn search via DDG + Jina Reader
ā ā āāā __init__.py # LinkedIn search tool registration
ā ā āāā client.py # DDG search + Jina Reader enrichment
ā āāā reddit/ # RSS + Shreddit keyless pipeline
ā ā āāā client.py # HTTP client with RSS parsing
ā ā āāā parsers.py # RSS/HTML parsers
ā ā āāā models.py # Shreddit card parsing
ā ā āāā engine.py # Search pipeline + arctic-shift backfill
ā ā āāā arctic_shift.py # Arctic-Shift score enrichment
ā ā āāā _utils.py # Shared HTML/regex helpers
ā āāā x.py # X/Twitter search via Xquik API or vendored Bird CLI
āāā tools/ # Specialized reference utilities
ā āāā arxiv.py # arXiv paper search (Lucene field prefixes)
ā āāā wikipedia.py # Wikipedia MediaWiki API
ā āāā digg.py # Digg AI 1000 clusters via digg-pp-cli
ā āāā techmeme.py # Techmeme tech news via techmeme-pp-cli
ā āāā polymarket.py # Polymarket prediction markets via Gamma API
ā āāā rerank.py # Rerank documents against a query (token_overlap / flashrank / hybrid)
āāā _config/ # Settings, env vars, rate limits, depth tiers
ā āāā settings.py # pydantic-settings (EXA_API_KEY, SEARCH_MCP_ prefix)
ā āāā limits.py # Per-platform quick/default/deep limits, timeouts
āāā _http/ # Shared HTTP + SSRF protection
ā āāā client.py # validate_url, http_client, get_json_client
āāā _models/ # Pydantic request/response models
ā āāā requests.py # SearchRequest
ā āāā responses.py # ErrorResponse, SearchResponse, PageResponse
ā āāā types.py # Depth, ResponseFormat, SearchType, FetchOutputFormat
āāā _utils/ # Shared helpers
ā āāā dates.py # Time-range and date-window utilities
ā āāā formatting.py # Markdown formatters, date/epoch utils
ā āāā rate_limiter.py # Token-bucket rate limiter
ā āāā scoring.py # Relevance scoring
āāā vendor/ # Vendored third-party tools
āāā bird-search/ # Node.js CLI for X/Twitter search (fallback when XQUIK_API_KEY unset)š ļø Tool Implementation Flow
When adding a new tool:
Implement logic in the appropriate module (
search/,social/, ortools/)Define models in
_models/(request/response types)Register in
server.pyusing@mcp.tooldecorator with a clear docstring (serves as the tool's description for the LLM)
š Design Decisions
search-backend-split ā Why
search_webunifies DuckDuckGo and Exa behind a singleproviderparameter instead of exposing two separate tools.rerank-tool ā Why
rerank_resultsuses three scoring backends (token_overlap/flashrank/hybrid) with metadata boosts, dedup, and chunk diversity.
š§Ŗ Testing
# Run all tests
uv run pytest
# Run a single test file
uv run pytest tests/test_module.py
# Run a specific test
uv run pytest tests/test_module.py::test_function_name
# Run with coverage
uv run pytest --cov=web_search_mcpš§ Troubleshooting
Problem | Likely Cause | Solution |
Auth errors on a tool | Env var not set in the server's shell | Export the variable in the same shell where the MCP server process runs |
GitHub returns empty results | Not authenticated | Run |
| Expired X session cookies | Re-extract |
| Bot detection | Exa fallback is automatic; ensure |
| Upstream arXiv maintenance | Wait a few minutes and retry |
Tool says "Query cannot be empty" | Missing or blank query | Provide a non-empty search query |
š¤ Contributing
Fork the repository.
Create a feature branch:
git checkout -b feat/my-new-toolEnsure all tests pass:
uv run pytestSubmit a pull request with a detailed description of the changes.
š License
This project is licensed under the MIT License.
Available Tools
3 toolsfetch_pageARead-onlyIdempotent
Extracts the full text content from a web page URL. Use this to read the details of a specific result found via web_search.
Args: url: The URL to fetch and extract content from output_format: Format for extracted content ('csv', 'html', 'json', 'markdown', 'python', 'txt', 'xml', 'xmltei') include_metadata: Whether to include document metadata (title, author, date, etc.) include_tables: Whether to include table content in extraction include_comments: Whether to include comment content in extraction include_images: Whether to include image descriptions in extraction deduplicate: Whether to remove duplicated content max_length: Maximum length of content to return (default 15000) timeout: Request timeout in seconds (default 30) backend: HTTP backend to use ('httpx' for lightweight, 'curl' to bypass bot detection, 'auto' to try httpx first then fallback to curl)
Inputs: url (required), output_format, include_metadata, include_tables, +6 more. Returns: result
search_docsARead-onlyIdempotent
Searches specifically for technical documentation or content on a specific domain.
Args: query: What you're looking for domain: The domain to search (e.g. 'docs.python.org', 'stackoverflow.com')
Returns: Search results from the specified domain
Inputs: query (required), domain. Returns: result
web_searchARead-onlyIdempotent
Unified search tool for web content and news.
Args: query: Search query string search_type: Type of search ('text' or 'news') max_results: Max number of results to return (default 5) time_range: Time filter ('d', 'w', 'm', 'y') or None region: Geographic region (e.g. 'us-en', 'uk-en') or None safesearch: Safe search level ('moderate', 'off', 'on') page: Page number for pagination (default 1) backend: Backend to use ('auto', 'legacy', 'api') response_format: Output format - 'markdown' for human-readable, 'json' for structured data
Returns: str: Markdown-formatted search results (when response_format="markdown") SearchResponse: Raw search results (when response_format="json") ErrorResponse: Error response if applicable
Inputs: query (required), search_type, max_results, time_range, +5 more. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogā¦
TDQS
Scored across 3 tools
Each tool has a clearly distinct purpose: web_search for general web/news search, search_docs for domain-specific search, and fetch_page for retrieving content from a URL. No overlap.
All tool names follow a consistent verb_noun pattern (fetch_page, search_docs, web_search) using snake_case, which is predictable and clear.
With 3 tools, the server is minimal but well-scoped for its purpose (search and fetch). It covers the essential operations without being overly complex or thin.
The tool surface covers the full workflow: general search, specialized search, and content extraction. No obvious missing operations; web_search returns structured results for further processing.
Maintenance
Related MCP Connectors
Web search and page reading for AI agents, including JSON APIs, RSS feeds and YouTube.
Give your AI today's internet: live web search, real-time breaking news, full articles and papers.
LLM-ready web search + instant answers + URL-to-clean-text fetch for agents and RAG.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Related MCP Servers
- AlicenseBqualityDmaintenanceProvides web search capabilities through Baidu with content fetching and parsing features, allowing LLMs to search the web and extract webpage content.157 PyPI28MIT
- AlicenseBqualityDmaintenanceEnables web search through DuckDuckGo and webpage content fetching with intelligent text extraction. Features built-in rate limiting and LLM-optimized result formatting for seamless integration with language models.2MIT
- FlicenseAqualityCmaintenanceProvides web search capabilities across multiple engines and webpage content fetching to simplified Markdown. It enables AI assistants to access real-time internet information and extract text from specific URLs.213-
- AlicenseAqualityDmaintenanceProvides web access capabilities for LLMs including search, fetching, content extraction, PDF reading, image viewing, and screenshots.346MIT