web-search-mcp

An all-in-one, fully local, zero external API, zero API key MCP tool that provides DeepSeek Harness + LM Studio with:

1. Web search — directly scrapes search engine result pages accessible from mainland China (Baidu / Bing China / 360 / Sogou), without calling any search API;

2. Full-page parsing — uses Crawl4AI (local Chromium) to extract page structure + text + images;

3. Image description — uses a local LM Studio vision model to convert images into Chinese text descriptions (image understanding happens server-side, thus bypassing DSH's limitation of discarding binary images).

Five MCP tools:

The scraped markdown is triple-denoised by default: ① PruningContentFilter-filtered version (when available); ② strips top navigation bars + removes footer/copyright/ad noise lines; ③ max_chars cap (default 20000 characters, truncated when exceeded). This prevents ads and other useless content from needlessly consuming context.

llm_extract uses a local LLM to fully solve body text extraction: ① rule-filter the page → ② SMALL_MODEL quickly extracts key points chunk by chunk → ③ LARGE_MODEL summarizes into a coherent summary.

⚠️ Model switching to save VRAM (default): when model_switching=true in config.py, single-instance sequential switching — automatically switches to qwen3.5-4b when the small model is needed (and disables thinking), then switches back to the large model qwen/qwen3.8-27b for summarization. Only one model is loaded at any given time, avoiding insufficient VRAM. Set to false to use dual instances in parallel (requires sufficient VRAM).

Configuration File (just edit this)

All configurable settings are centralized in config.py — for future maintenance, only edit this one file:

Environment variables (e.g., the env section of DSH's cordis.patch.yml) can still override config.py defaults, but for daily use, editing config.py is sufficient. Restart DSH after changes for them to take effect.

Architecture

                    ┌──────────────────────────────┐
                    │  web-search-mcp (本进程)        │
关键词 ─────────────►│  1. 抓取 百度/必应/360/搜狗 结果页 │──► 搜索结果(标题/URL/摘要)
                    │  2. Crawl4AI 整页解析           │──► markdown / links / images
                    │  3. 下载图片 ─► LM Studio 视觉模型 │──► 图片中文描述(文本)
                    └──────────────────────────────┘
                              ▲ MCP stdio
                    ┌─────────┴──────────┐
                    │ DeepSeek Harness    │  (cordis.yml 里的 @deepseek-ai/dsh-mcp-client)
                    │ LM Studio(主模型)    │
                    └────────────────────┘

• Search, scraping, and image description all happen locally; the only network access is "opening the webpage itself" (unavoidable for any web search), no third-party APIs, no API keys, data never leaves your machine.

• Image description is server-side vision: Crawl4AI only extracts the image URLs; this tool downloads the images, calls LM Studio's vision model, converts the images to text, and returns that to DSH. Therefore, DSH's MCP bridging layer (which discards binary images) is not an issue.

Installation

1. Environment

• Python 3.10+ (Crawl4AI recommends 3.11 / 3.12; if you hit dependency issues on 3.13, fall back to 3.12)

• Docker installed is optional (this project does not require Docker; SearXNG is also not required — search works by direct scraping)

• LM Studio running with a model loaded

2. Install dependencies (use a mirror in mainland China)

cd web-search-mcp
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt

# 下载 Crawl4AI 用的 Chromium(仅抓取功能需要;只用搜索可跳过)
playwright install chromium

Dependency notes: httpx + beautifulsoup4 are required (search + MCP transport); lxml is optional (falls back to the standard library if not installed); crawl4ai is only needed for the scraping feature. The MCP transport layer is hand-written using only the Python standard library and does not depend on mcp/pydantic, so in the worst case, installing only httpx + beautifulsoup4 is enough for search.

3. Configure the LM Studio vision model (optional, but required for image description)

Load a vision model that supports image input in LM Studio, such as Qwen2.5-VL-7B-Instruct, MiniCPM-V, or LLaVA.

Set the environment variables (or write them to .env, but this tool does not automatically read .env; set them in your startup command):

⚠️ Single instance vs. dual instance: LM Studio typically loads only one model at a time. If your main chat model is not a vision model, it is recommended to start another LM Studio instance (on a different port, e.g., 1235) dedicated to the vision model, then point VISION_BASE_URL to http://localhost:1235/v1.

Integrating with DeepSeek Harness

Add a section to the plugin list in cordis.yml (see cordis.example.yml for an example):

- id: mcp-websearch
  name: '@deepseek-ai/dsh-mcp-client'
  config:
    serverName: websearch
    transport: stdio
    command: python
    args: ['C:/Users/LiangYuelin/Desktop/workspace/web-search-mcp/server.py']
    cwd: 'C:/Users/LiangYuelin/Desktop/workspace/web-search-mcp'
    env:
      VISION_BASE_URL: 'http://localhost:1234/v1'
      VISION_MODEL: 'qwen2.5-vl-7b-instruct'
      VISION_API_KEY: 'lm-studio'
    toolCallTimeoutMs: 300000   # 抓取 + 图片描述较慢,务必调大

• If you use a venv, change command to .venv/Scripts/python.exe (absolute path).

• After integration, the model will see three tools: mcp__websearch__search_web, mcp__websearch__scrape_url, and mcp__websearch__search_and_extract.

Usage Examples

The model will naturally call tools, for example:

• "Search for 'latest advances in LLM RAG'" → search_web(query="大模型 RAG 最新进展", engine="bing")

• "Scrape and parse this webpage, and tell me what the images in it are" → scrape_url(url="https://...", describe_images=true)

• "Search for 'Bitcoin market' and summarize the top 3 articles" → search_and_extract(query="比特币 行情", engine="bing", max_results=3)

Search engine selection:

Deployment Status (local machine)

Completed and tested locally:

• Dependencies installed: crawl4ai 0.9.2 + playwright + lxml + Chromium (via a domestic mirror);

• DSH config written to ~/.dsh/profiles/web/cordis.patch.yml;

• All four search engines (Baidu/Bing/360/Sogou) return results;

• Baidu redirect links are correctly resolved;

• MCP stdio protocol fully working (initialize / tools/list / tools/call / error handling / Chinese UTF-8);

• scrape_url (full-page markdown + links + images) and search_and_extract (search → resolve → scrape → extract images) pass end-to-end.

The only remaining manual step (required for image description):

1. Open LM Studio → start the local server (port 1234);

2. Load the vision model qwen/qwen3.8-27b (with mmproj, supports image input);

3. Restart DSH (dsh web), and you will see the three mcp__websearch__* tools in the model.

File Descriptions

• server.py —— MCP server entry point (hand-written MCP stdio, zero mcp/pydantic dependencies)

• engines.py —— Search engine scraping module (Baidu/Bing/360/Sogou)

• vision.py —— LM Studio vision model image description

• cache.py —— Disk cache module (scrape results / image description reuse, pure standard library)

• config.py —— Centralized configuration (all configurable items)

• requirements.txt —— Dependencies

• .env.example —— Vision model environment variable example

• cordis.example.yml —— DSH integration config example

Performance Optimizations (Route A · Shipped)

Four optimizations for a local environment with 20GB VRAM + 32GB RAM, without adding any new models:

F5 requires the optional Pillow dependency (already in requirements.txt); if not installed, downsampling is skipped automatically and all other features are unaffected. The cache directory defaults to .cache/ inside the project; set CACHE_ENABLED=false to disable entirely.

Known Limitations

• Search results may occasionally contain ads (Baidu's baidu.php?url=... are ad links that cannot be resolved; they will be skipped/error out during scraping, which is expected);

• Search engine anti-scraping may cause occasional failures; just switch engines;

• Scraping large pages / pages with many images is slow; be sure to increase toolCallTimeoutMs in the DSH config;

• Image description quality depends on your local vision model itself.