Web Research Assistant MCP Server

Comprehensive Model Context Protocol (MCP) server that provides web research and discovery capabilities. Includes 16 tools, 4 resources, and 5 prompts for searching, crawling, and analyzing web content, powered by your local Docker SearXNG instance, Exa AI neural search, Scrapling, and Pixabay API:

1. web_search — federated search across multiple engines via SearXNG

2. batch_web_search — run up to 20 searches in one call instead of looping web_search

3. search_examples — find code examples, tutorials, and articles (defaults to recent content)

4. search_images — find high-quality stock photos, illustrations, and vectors via Pixabay

5. crawl_url — full page content extraction with advanced crawling

6. batch_crawl_urls — read up to 20 pages in parallel in one call instead of looping crawl_url

7. package_info — detailed package metadata from npm, PyPI, crates.io, Go

8. package_search — discover packages by keywords and functionality

9. github_repo — repository health metrics and development activity

10. translate_error — find solutions for error messages and stack traces from Stack Overflow (auto-detects CORS, fetch, and web errors)

11. api_docs — auto-discover and crawl official API documentation with examples (works for any API - no hardcoded URLs)

12. extract_data — extract structured data (tables, lists, fields, JSON-LD) from web pages with automatic detection

13. compare_tech — compare technologies side-by-side with NPM downloads, GitHub stars, and aspect analysis (React vs Vue, PostgreSQL vs MongoDB, etc.)

14. get_changelog — NEW! Get release notes and changelogs with breaking change detection (upgrade safely from version X to Y)

15. check_service_status — NEW! Instant health checks for 25+ services (Stripe, AWS, GitHub, OpenAI, etc.) - "Is it down or just me?"

The server's MCP instructions tell agents to batch: several pages or searches go into one batch_crawl_urls or batch_web_search call, never one crawl_url or web_search call per item.

All tools feature comprehensive error handling, response size limits, usage tracking, and clear documentation for optimal AI agent integration.

MCP Resources (Direct Data Lookups)

• package://{registry}/{name} - Package info from npm, PyPI, crates.io, or Go modules

• github://{owner}/{repo} - Repository information and health metrics

• status://{service} - Service health status for 120+ services

• changelog://{registry}/{package} - Package release notes and changelogs

MCP Prompts (Reusable Workflows)

• research_package - Comprehensive package evaluation

• debug_error - Structured error debugging with solutions

• compare_technologies - Side-by-side technology comparison

• evaluate_repository - GitHub repository health assessment

• check_service_health - Multi-service status monitoring

Quick Start

Option 1: Full Docker Setup (Recommended)

Everything runs in Docker - no Python installation needed:

./docker-start.sh

This starts both SearXNG and the MCP server in containers. See DOCKER_SETUP.md for details.

Option 2: Python + Docker SearXNG

1. Set up SearXNG (5 minutes):

   # Using Docker (recommended)
   docker run -d -p 2288:8080 searxng/searxng:latest

   Then configure search engines - see SEARXNG_SETUP.md for optimized settings.

2. Install the MCP server:

   uvx web-research-assistant  # or: pip install web-research-assistant

3. Configure Claude Desktop - add to claude_desktop_config.json:

   {
     "mcpServers": {
       "web-research-assistant": {
         "command": "uvx",
         "args": ["web-research-assistant"]
       }
     }
   }

4. Restart Claude Desktop and start researching!

⚠️ For best results: Configure SearXNG with GitHub, Stack Overflow, and other code-focused search engines. See SEARXNG_SETUP.md for the recommended configuration.

Related MCP server: SearXNG MCP Server

Prerequisites

Required

• Python 3.10+

• A running SearXNG instance on http://localhost:2288

  • 📖 See SEARXNG_SETUP.md for complete Docker setup guide

  • ⚠️ IMPORTANT: For best results, enable these search engines in SearXNG:

    • GitHub, Stack Overflow, GitLab (for code search - critical!)

    • DuckDuckGo, Brave (for web search)

    • MDN, Wikipedia (for documentation)

    • Reddit, HackerNews (for tutorials and discussions)

    • See SEARXNG_SETUP.md for the full optimized configuration

Optional

• Exa API key for neural search - Get API key (recommended for better search results)

• Pixabay API key for image search - Get free key

• Playwright browsers for advanced crawling (installed with scrapling install)

Developer Setup (if running from source)

uv tool install uv  # if you do not already have uv
uv sync --all-extras --all-groups  # creates the virtual environment
uv run scrapling install           # installs Chromium for Scrapling

You can also use pip install -e . if you prefer pip over uv (deps come from pyproject.toml).

Installation

Option 1: Using uvx (Recommended - No installation needed!)

uvx web-research-assistant

This runs the server directly from PyPI without installing it globally.

Option 2: Install with pip

pip install web-research-assistant
web-research-assistant

Option 3: Install with uv

uv tool install web-research-assistant
web-research-assistant

By default the server communicates over stdio, which makes it easy to wire into Claude Desktop or any other MCP host.

MCP Client Configuration

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

Option 1: Using uvx (Recommended - No installation needed!)

{
  "mcpServers": {
    "web-research-assistant": {
      "command": "uvx",
      "args": ["web-research-assistant"]
    }
  }
}

Option 2: Using installed package

{
  "mcpServers": {
    "web-research-assistant": {
      "command": "web-research-assistant"
    }
  }
}

OpenCode

Add to ~/.config/opencode/opencode.json:

Using uvx (Recommended)

{
  "mcp": {
    "web-research-assistant": {
      "type": "local",
      "command": ["uvx", "web-research-assistant"],
      "enabled": true
    }
  }
}

Using installed package

{
  "mcp": {
    "web-research-assistant": {
      "type": "local",
      "command": ["web-research-assistant"],
      "enabled": true
    }
  }
}

Development (Running from source)

For Claude Desktop:

{
  "mcpServers": {
    "web-research-assistant": {
      "command": "uv",
      "args": [
        "--directory",
        "/ABSOLUTE/PATH/TO/web-research-assistant",
        "run",
        "web-research-assistant"
      ]
    }
  }
}

For OpenCode:

{
  "mcp": {
    "web-research-assistant": {
      "type": "local",
      "command": [
        "uv",
        "--directory",
        "/ABSOLUTE/PATH/TO/web-research-assistant",
        "run",
        "web-research-assistant"
      ],
      "enabled": true
    }
  }
}

Restart your MCP client afterwards. The MCP tools will be available immediately.

Tool behavior

Tool responses are capped at 30,000 characters by default. crawl_url returns an exact start_char continuation value when a page is longer than the requested chunk.

The batch tools share one 50,000-character budget across their items: short pages come back whole and long ones are cut evenly. A cut batch_crawl_urls result ends with the exact arguments that continue every cut page in one more call. Items still running after 45 seconds are reported as timed out, so the batch returns before typical 60-second client timeouts.

Resources

MCP Resources provide direct data access via URI templates - perfect for quick lookups without tool calls.

Prompts

MCP Prompts are reusable message templates that guide AI assistants through common workflows.

Configuration

Environment variables let you adapt the server without touching code:

Development

The codebase is intentionally modular and organized:

web-research-assistant/
├── src/searxng_mcp/     # Source code
│   ├── config.py        # Configuration and environment
│   ├── search.py        # SearXNG integration
│   ├── exa.py           # Exa AI neural search client
│   ├── crawler.py       # Scrapling wrapper
│   ├── images.py        # Pixabay client
│   ├── registry.py      # Package registries (npm, PyPI, crates, Go)
│   ├── github.py        # GitHub API client
│   ├── errors.py        # Error parser (language/framework detection)
│   ├── api_docs.py      # API docs discovery (NO hardcoded URLs)
│   ├── tracking.py      # Usage analytics
│   ├── batch.py         # Batch tool helpers (parallel runs, shared output budget)
│   ├── instructions.py  # MCP instructions sent to every agent
│   └── server.py        # MCP server + 16 tools
├── docs/                # Documentation (27 files)
└── [config files]

Each module is well under 400 lines, making the codebase easy to understand and extend.

Usage Analytics

All tools automatically track usage metrics including:

• Tool invocation counts and success rates

• Response times and performance trends

• Common use case patterns (via the reasoning parameter)

• Error frequencies and types

Analytics data is stored in ~/.config/web-research-assistant/usage.json. The optional reasoning parameter improves use-case grouping, but omitting analytics context never blocks a tool call.

Documentation

Comprehensive documentation is available in the docs/ directory:

• Project Status - Current status, metrics, roadmap

• API Docs Implementation - NEW tool documentation

• Error Translator Design - Error translator details

• Tool Ideas Ranked - Prioritization and progress

• SearXNG Configuration - Recommended setup

• Quick Start Examples - Usage examples

See the docs README for a complete index.