<div align="center">

<img src="./assets/mywebsearch-logo.png" alt="WebSearch Forge logo" width="148" height="148">

# WebSearch Forge MCP

**A lightweight, unified WebSearch MCP for AI agents.**

Search, read, crawl, parse, and research the public web through one MCP server.

[Quick Start](#quick-start) · [Tools](#tools) · [Architecture](#architecture) · [Configure Engines](#configure-search-engines) · [中文文档](./README.zh-CN.md)

![Python](https://img.shields.io/badge/Python-3.10%2B-3776AB?logo=python&logoColor=white)
![MCP](https://img.shields.io/badge/MCP-stdio-111827)
![SearXNG](https://img.shields.io/badge/SearXNG-gateway-2AA889)
![Windows](https://img.shields.io/badge/Windows-supported-0078D4?logo=windows&logoColor=white)

</div>

## What It Is

WebSearch Forge MCP is a self-contained MCP service for agent-driven web research:

~~~text
Question → search sources → fetch pages → extract content → crawl related pages → compile evidence
~~~

It is split into clear layers:

- `transports` exposes MCP stdio and the optional FastAPI adapter.
- `core` handles configuration, caching, security checks, orchestration, and research workflows.
- `providers` implement search, fetching, extraction, crawling, and media features.
- SearXNG is the single search gateway for Bing, Baidu, Brave, DuckDuckGo, and other configured engines.

## Highlights

### One MCP server, six capabilities

Connect once to `transports/mcp_stdio.py`:

| Tool | Purpose |
| --- | --- |
| `search_web` | Find candidate sources through SearXNG |
| `fetch_web_content` | Fetch and extract one public URL |
| `search_and_fetch` | Search, then read the top results |
| `deep_research` | Run multiple searches and compile a report |
| `crawl_site` | Recursively crawl a bounded same-domain site |
| `youtube_transcript` | Retrieve YouTube captions |

### Unified search gateway

The `engines` argument is passed to SearXNG as a filter. WebSearch Forge does not independently fan out to search websites:

~~~text
WebSearch Forge MCP → SearXNG → Bing / Baidu / Brave / DuckDuckGo
~~~

### Lightweight by default

MCP stdio needs no exposed application port or database server. The cache is SQLite. Optional packages add Trafilatura, Readability, stealth requests, Playwright, Office/PDF parsing, Scrapy, YouTube captions, and FastAPI without changing the MCP contract.

### Bounded crawling

`crawl_site` supports a zero-dependency native backend and an optional Scrapy backend. It enforces page and depth limits, stays on the starting host, resolves relative links, and isolates page failures.

## Quick Start

The following commands are for Windows PowerShell.

### Install

~~~powershell
cd D:\my-websearch\my_websearch
py -3.12 -m pip install -r requirements.txt
~~~

### Start SearXNG

~~~powershell
docker version
cd D:\my-websearch\my_websearch
docker compose up -d searxng
docker compose ps
~~~

The default gateway is `http://127.0.0.1:8080`. Configuration lives in [config/searxng/settings.yml](./config/searxng/settings.yml).

### Configure the MCP client

Use [mcp_config.example.json](./mcp_config.example.json) and keep an absolute path:

~~~json
{
  "mcpServers": {
    "websearch-forge": {
      "command": "py",
      "args": [
        "-3.12",
        "D:\\my-websearch\\my_websearch\\transports\\mcp_stdio.py"
      ],
      "env": {
        "SEARXNG_URL": "http://127.0.0.1:8080",
        "CACHE_TTL": "300"
      }
    }
  }
}
~~~

The stdio adapter forces UTF-8 on Windows. Loopback traffic bypasses machine-wide proxy variables by default; set `PROXY_URL` explicitly when needed.

## Tools

### `search_web`

Search through SearXNG without downloading page bodies.

~~~json
{
  "name": "search_web",
  "arguments": {
    "query": "Python 3.14 new features",
    "engines": ["bing", "baidu", "brave"],
    "limit": 5,
    "time_range": "month"
  }
}
~~~

### `fetch_web_content`

Fetch and extract one public URL. HTML, PDF, DOCX, XLSX, PPTX, CSV, Markdown, and plain text are supported.

~~~json
{
  "name": "fetch_web_content",
  "arguments": {
    "url": "https://www.python.org",
    "max_chars": 10000,
    "stealth_mode": "off",
    "render_mode": "auto",
    "extraction_mode": "auto"
  }
}
~~~

The response includes final URL, HTTP status, title, extraction method, word count, content, and discovered links.

### `search_and_fetch`

Search first, then fetch the top results independently. A failed page is recorded on that item and does not cancel the batch.

~~~json
{
  "name": "search_and_fetch",
  "arguments": {
    "query": "FastAPI MCP server",
    "limit": 3,
    "max_chars": 12000
  }
}
~~~

### `deep_research`

Run related queries concurrently, fetch the strongest results, and return a Markdown report with source-level failures.

~~~json
{
  "name": "deep_research",
  "arguments": {
    "queries": ["SearXNG engine configuration", "MCP stdio deployment"],
    "breadth": 3,
    "max_chars": 12000
  }
}
~~~

### `crawl_site`

Crawl a same-host site with hard page and depth limits.

~~~json
{
  "name": "crawl_site",
  "arguments": {
    "url": "https://www.python.org",
    "max_pages": 10,
    "max_depth": 2,
    "backend": "native",
    "stealth_mode": "off"
  }
}
~~~

`native` is the default. Install Scrapy and set `backend` to `scrapy` to use the optional backend. Each page reports URL, depth, status, fetch method, title, content, and word count.

### `youtube_transcript`

Retrieve YouTube captions with optional source and translation languages.

## Response Shape

~~~json
{
  "query": "OpenAI",
  "provider": "searxng",
  "engines": ["bing", "baidu", "brave"],
  "total_results": 3,
  "results": [
    {
      "title": "OpenAI | Research & Deployment",
      "url": "https://openai.com/",
      "description": "...",
      "source": "openai.com",
      "engine": "bing",
      "score": 1.0
    }
  ],
  "partial_failures": []
}
~~~

Successful partial results are preserved. Engine, page, and document errors are returned as structured failure entries.

## Architecture

~~~mermaid
flowchart LR
    A[Agent / MCP Client] -->|stdio JSON-RPC| B[transports/mcp_stdio.py]
    B --> C[core/service.py]
    C --> D[providers/search]
    D --> E[SearXNG]
    E --> F[Bing / Baidu / Brave / DDG]
    C --> G[providers/content]
    G --> H[HTTP / stealth / Playwright]
    C --> I[providers/crawl]
    C --> J[providers/media]
    C --> K[(SQLite TTL cache)]
~~~

- `transports` adapts protocols; MCP and FastAPI share the same service.
- `core` owns orchestration, cache policy, configuration, and URL security.
- `providers/search` talks to SearXNG and validates engine names.
- `providers/content` handles HTTP, stealth transport, rendering, and extraction.
- `providers/crawl` contains native and Scrapy crawling backends.
- `providers/media` contains the YouTube transcript provider.

## Configure Search Engines

The project-owned SearXNG source is:

~~~text
config/searxng/settings.yml
~~~

Two settings decide whether an engine can be called:

1. `settings.yml` enables the engine inside SearXNG.
2. `providers/search/registry.py` lists the accepted name in `SUPPORTED_ENGINES`.

Restart after changes:

~~~powershell
cd D:\my-websearch\my_websearch
docker compose up -d --force-recreate searxng
~~~

If SearXNG does not provide the engine yet, implement that SearXNG engine first.

## Security and Reliability

- Only HTTP and HTTPS URLs are accepted.
- Localhost, loopback, private IPv4, link-local, and private IPv6 targets are rejected.
- Redirect destinations are validated again.
- Search and fetch operations use a SQLite TTL cache, 300 seconds by default.
- Partial failures do not discard successful work.
- stdout is reserved for MCP JSON-RPC.

## Optional Capabilities

| Capability | Enablement |
| --- | --- |
| Trafilatura / Readability | Install `requirements-api.txt` |
| Stealth requests | Install `curl-cffi` and use `stealth_mode=high` |
| JavaScript rendering | Install Playwright and use `render_mode=browser` |
| PDF / DOCX / XLSX / PPTX | Install matching document packages |
| Scrapy crawling | Install Scrapy and use `backend=scrapy` |
| FastAPI HTTP service | Run `py -3.12 -m transports.api` |
| YouTube captions | Install `youtube-transcript-api` |

## Project Layout

~~~text
my_websearch/
├── assets/                     # Project logo
├── config/searxng/             # SearXNG settings.yml
├── core/                       # Config, cache, security, orchestration
├── providers/
│   ├── search/                 # SearXNG gateway and engine allow-list
│   ├── content/                # Requests, rendering, extraction
│   ├── crawl/                  # Native and Scrapy backends
│   └── media/                  # YouTube transcript provider
├── transports/                 # MCP stdio and FastAPI adapters
├── tests/                      # Dependency-free self-checks
├── docker-compose.yml          # Local SearXNG gateway
├── mcp_config.example.json     # MCP client template
├── requirements.txt            # Dependency entry point
├── README.md                  # English documentation
└── README.zh-CN.md            # 中文文档
~~~

## Development Check

~~~powershell
cd D:\my-websearch
py -3.12 -m my_websearch.tests.test_server
~~~

Expected output:

~~~text
my_websearch self-check: ok
~~~

Optional FastAPI service:

~~~powershell
cd D:\my-websearch\my_websearch
py -3.12 -m transports.api
~~~

Then open <http://127.0.0.1:8787/docs>.

## License

No license is imposed yet. Add a root-level `LICENSE` file before public distribution.