Web Search MCP

An MCP server that provides eight tools: a web search powered by SearXNG, five Crawl4AI-powered tools (web_fetch, web_screenshot, web_pdf, web_execute_js, web_crawl), and two Wayback Machine tools (web_snapshots, web_archive).

Architecture

graph LR
    Client["MCP Client<br/>(Claude, Cursor, etc.)"] -->|web_search| Server["MCP Server"]
    Client -->|web_fetch| Server
    Client -->|web_screenshot| Server
    Client -->|web_pdf| Server
    Client -->|web_execute_js| Server
    Client -->|web_crawl| Server
    Client -->|web_snapshots| Server
    Client -->|web_archive| Server
    Server --> SearXNG
    SearXNG --> Redis
    Server --> Crawl4AI
    Server --> Wayback["Wayback Machine"]

The web_search tool queries SearXNG for search results. The Crawl4AI tools handle content extraction, screenshots, PDFs, JS execution, and multi-URL crawling. The Wayback Machine tools list and retrieve archived pages.

The full stack deploys as 4 services: Redis, SearXNG, Crawl4AI, and this MCP server.

Related MCP server: Search MCP Server

Tools

The server exposes eight MCP tools:

web_search

Lightweight web search via SearXNG. Returns structured results.

Returns a JSON array of { url, title, description } results.

web_fetch

Fetch a single URL and return its content as clean markdown via Crawl4AI.

Returns the page content as markdown.

web_screenshot

Capture a full-page PNG screenshot of a URL via Crawl4AI.

Returns a base64-encoded PNG image.

web_pdf

Generate a PDF document of a URL via Crawl4AI.

Returns a base64-encoded PDF.

web_execute_js

Execute JavaScript snippets on a URL via Crawl4AI and return the full crawl result.

Returns the full CrawlResult JSON including markdown, links, media, and JS execution results.

web_crawl

Crawl one or more URLs and extract their content using Crawl4AI.

Returns the extracted content from each URL.

web_snapshots

List Wayback Machine snapshots for a URL.

Returns a JSON array of snapshots with timestamps, status codes, and archive URLs.

web_archive

Retrieve an archived page from the Wayback Machine.

Returns the archived page content.

Connecting to the Server

All examples below assume your server is running at https://your-server.up.railway.app/mcp with an API key. Replace the URL and key with your own values.

Claude Code (CLI)

claude mcp add web_search \
  --transport http \
  https://your-server.up.railway.app/mcp \
  --header "Authorization: Bearer your-api-key"

Project-level config (.mcp.json)

Add to .mcp.json at the root of any project to make the tool available to all collaborators:

{
  "mcpServers": {
    "web_search": {
      "type": "http",
      "url": "https://your-server.up.railway.app/mcp",
      "headers": {
        "Authorization": "Bearer your-api-key"
      }
    }
  }
}

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "web_search": {
      "type": "http",
      "url": "https://your-server.up.railway.app/mcp",
      "headers": {
        "Authorization": "Bearer your-api-key"
      }
    }
  }
}

Replace Claude Code's Built-in Web Search & Web Fetch (Optional)

By default, Claude Code uses its own WebSearch and WebFetch tools. You can replace them with this server's web_search and web_fetch tools for privacy-respecting, self-hosted results.

1. Add the MCP server globally:

claude mcp add web_search --scope user \
  --transport http \
  https://your-server.up.railway.app/mcp \
  --header "Authorization: Bearer your-api-key"

2. Disable the built-in tools by editing ~/.claude/settings.json:

{
  "permissions": {
    "deny": ["WebSearch", "WebFetch"]
  }
}

3. Guide Claude via ~/.claude/CLAUDE.md so it uses your tools:

## Search & Fetch

- Use the web_search MCP tool for all web searches
- Use the web_fetch MCP tool to fetch and read web pages
- Do not attempt to use the built-in WebSearch or WebFetch tools

4. Verify by running /mcp inside Claude Code to check the server is connected, then ask Claude to search for something or fetch a URL.

Deployment

Railway

This project deploys as 4 services on Railway:

1. Redis - Cache for SearXNG

2. SearXNG - Meta search engine

3. Crawl4AI - Browser automation service

4. MCP Server - This web-search-mcp server

Deploy Steps:

1. Fork/Clone this repository to your GitHub account

2. Create a new project on Railway

3. Deploy Redis:

   • Click "Add Service" → "Database" → "Redis"

4. Deploy SearXNG:

   • Click "Add Service" → "Image"

   • Image: searxng/searxng:latest

   • Add port: 8080

   • Set SEARXNG_SECRET_KEY and SEARXNG_REDIS_URL environment variables

5. Deploy Crawl4AI:

   • Click "Add Service" → "Image"

   • Image: unclecode/crawl4ai:latest

   • Add port: 11235

   • (Optional) Set CRAWL4AI_API_TOKEN environment variable

6. Deploy MCP Server:

   • Click "Add Service" → "GitHub Repo"

   • Select your forked repository

   • Railway will auto-detect the Dockerfile

   • Set environment variables:

     • API_KEY - Your secret API key (use ${{random(32)}} to generate)

     • CRAWL4AI_URL - http://crawl4ai.railway.internal:11235 (or your Crawl4AI service URL)

7. Get your URL and test:

   curl -X POST https://your-service.up.railway.app/mcp \
     -H "Authorization: Bearer your-api-key" \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"tools/list","id":1}'

Other Platforms

The server can be deployed to any platform that supports Docker or Node.js:

• Fly.io - Use the Dockerfile

• Render - Use the Dockerfile

• DigitalOcean App Platform - Use the Dockerfile

• Self-hosted - Use docker-compose

Quick Start (Local)

1. Clone and install

git clone https://github.com/your-username/web-search-mcp
cd web-search-mcp
pnpm install

2. Configure environment

cp .env.example .env.local
# Edit .env.local and set your API_KEY

3. Start Crawl4AI

docker compose up -d crawl4ai

4. Run the MCP server

CRAWL4AI_URL=http://localhost:11235 pnpm run start

The server is available at http://localhost:3000/mcp.

5. Or run everything in Docker

docker compose up

Authentication

The API_KEY environment variable is required.

On Railway, set the key manually or use ${{random(32)}} to auto-generate. For local development, set it in your .env.local file.

Clients provide the key as a Bearer token in the Authorization header (shown in the examples above) or as an ?api_key= query parameter. The /health endpoint is unauthenticated.

License

MIT