MCP Web Fetch Server

An all-in-one Python MCP server for web research: fetch pages, search the web, batch-fetch, extract links, summarize via client-side sampling, and (optionally) read/write local files — all for LLM agents like Cursor. Supports local stdio and Streamable HTTP for remote access, and exercises essentially every MCP protocol capability (Tools, Resources, Prompts, Completions, Sampling, Elicitation, Roots, Progress, Logging).

Documentation:

Features

Tools

• fetch_url — page content as markdown, with chunked reading (start_index, max_length)

• fetch_metadata_tool — HEAD request metadata

• batch_fetch — fetch multiple URLs concurrently, with per-URL error isolation and progress

• web_search — search the live web (DuckDuckGo, no API key, with SearXNG fallback), the local corpus, or both, via source=auto|web|corpus|both

• image_search — web image search (via SearXNG), returning image URLs, titles and source pages

• describe_image — have the local vision model read any image URL and describe it, including text in it

• extract_links — structured link/image extraction from a page

• summarize_url — asks the connected client's LLM to summarize a page (MCP sampling)

• read_file / write_file / list_dir — sandboxed local file access (opt-in, disabled by default)

Offline corpus (optional)

Ingest your own documents and the same tools serve them instead of the web — an offline archive a local model can browse as if it were the internet.

• rag_search — hybrid semantic + exact-term search over your documents, with citable URLs

• rag_answer — an answer written from your documents alone, with numbered citations

• rag_explain — why a search ranked the way it did, arm by arm

• corpus_stats — what the archive contains

• Prompts: explore_archive, research_corpus, cite_claim, compare_documents, summarize_category

• fetch_url, web_search, extract_links transparently serve the archive when FETCH_NET_MODE is offline or hybrid

uv sync --extra rag
ollama serve                                  # the local model lives on the host
mcp-fetch-server ingest ./my-documents        # PDF, DOCX, PPTX, EPUB, HTML, MD, TXT
mcp-fetch-server enrich                       # titles, summaries, tags, questions
mcp-fetch-server taxonomy bootstrap           # propose categories, then edit the file
mcp-fetch-server classify                     # file documents under them
mcp-fetch-server embed                        # build the search index
mcp-fetch-server search "your question"       # or --answer for a written answer
mcp-fetch-server watch ./my-documents         # keep the corpus in step with a folder
./scripts/install-watch-unit.sh               # ...or run that as a systemd unit (WSL)

mcp-fetch-server doctor checks the whole stack before you start, and mcp-fetch-server eval --from-corpus measures retrieval quality so accuracy changes can be verified rather than assumed. Qdrant runs embedded on local disk when FETCH_QDRANT_URL is empty, so no Docker is needed.

See docs/RAG_PLAN.md for the full picture.

Other MCP capabilities

• Resources: config://settings, history://recent, fetch-cache://{encoded_url}

• Prompts: fetch, research_topic, summarize_page, extract_key_facts, compare_sources

• Completions: URL/depth autocomplete for prompt and resource arguments

• Elicitation: write_file confirms before overwriting an existing file

• Roots: local file tools honor client-exposed directories in addition to FETCH_LOCAL_FILES_ROOT

• Progress notifications: batch_fetch and summarize_url report progress as they run

• Management GUI: web dashboard at /admin for status, config, history, cache, and tools

Security

• SSRF protection with resolve-then-check and redirect re-validation

• robots.txt compliance (override with ignore_robots_txt=true)

• Optional domain allowlist

• Local file tools sandboxed to one configured directory, path-traversal safe

• Bearer token auth + rate limiting for HTTP mode

• /health endpoint

• Windows .exe build (no Python required for end users)

Related MCP server: myscrape

Quick Start

One-command install (any platform)

# Linux / macOS / WSL
curl -fsSL https://raw.githubusercontent.com/narimanamiri/mcp-fetch-server/master/scripts/install.sh -o install.sh
bash install.sh --yes

# Windows
irm https://raw.githubusercontent.com/narimanamiri/mcp-fetch-server/master/scripts/install.ps1 -OutFile install.ps1
powershell -ExecutionPolicy Bypass -File install.ps1 -Yes

Clones the project if it is not already present, installs uv and the dependencies, writes a .env with a freshly generated MCP_AUTH_TOKEN, checks Ollama and pulls the models, brings up Qdrant (or picks embedded mode when Docker is absent), and runs doctor to verify. Re-running never overwrites an existing .env.

Useful flags (--help for all): --mode offline, --qdrant embedded, --no-models, --rerank, --corpus PATH, --watch.

Option A: Docker — full stack (recommended for server deployment)

Runs the MCP server, admin GUI, and SearXNG together.

Linux / macOS:

cd mcp-fetch-server
cp .env.docker.example .env          # edit MCP_AUTH_TOKEN
chmod +x scripts/docker-up.sh
./scripts/docker-up.sh
# or: docker compose up -d --build

Windows (PowerShell):

cd "E:\my python projects\MCP\mcp-fetch-server"
copy .env.docker.example .env
.\scripts\docker-up.ps1

Connect Cursor over HTTP (Bearer token required):

{
  "mcpServers": {
    "web-fetch": {
      "url": "http://127.0.0.1:8000/mcp",
      "headers": { "Authorization": "Bearer your-token-from-env" }
    }
  }
}

Local files for read_file/write_file map to the ./workspace folder on your host.

Option B: Windows executable (local / Cursor stdio)

cd "E:\my python projects\MCP\mcp-fetch-server"
.\dist\mcp-fetch-server.exe --transport stdio

Build the exe yourself: .\scripts\build_exe.ps1 → outputs dist\mcp-fetch-server.exe

Option C: Python + uv (development)

cd "E:\my python projects\MCP\mcp-fetch-server"
uv sync --dev
copy .env.example .env
uv run mcp-fetch-server --transport stdio

Connect to Cursor

See the User Manual — Connect to Cursor for step-by-step instructions.

Minimal .cursor/mcp.json using the executable:

{
  "mcpServers": {
    "web-fetch": {
      "command": "E:/my python projects/MCP/mcp-fetch-server/dist/mcp-fetch-server.exe",
      "args": ["--transport", "stdio"],
      "env": { "PYTHONIOENCODING": "utf-8" }
    }
  }
}

Management GUI

A built-in web dashboard lets you monitor and manage the server without using Cursor:

The dashboard shows uptime, registered tools, redacted configuration, recent fetch history, cached page previews, and a button to clear history/cache. It auto-refreshes every 30 seconds.

If MCP_AUTH_TOKEN is set, enter it in the dashboard's auth bar (stored in your browser session only). Disable the GUI with FETCH_ADMIN_ENABLED=false.

Optional: SearXNG search fallback

web_search uses DuckDuckGo by default and falls back to SearXNG when that scrape fails. When you use Docker (docker compose up), SearXNG is started automatically and the MCP container is preconfigured to reach it at http://searxng:8080.

For local (non-Docker) use, you can still run only SearXNG:

docker compose up -d searxng

Set FETCH_SEARXNG_URL=http://localhost:8080 in .env. See searxng/settings.yml.

Remote HTTP Mode (without full Docker stack)

$env:MCP_AUTH_TOKEN = "your-long-random-token"
.\dist\mcp-fetch-server.exe --transport streamable-http --host 127.0.0.1 --port 8000

• MCP endpoint: http://127.0.0.1:8000/mcp

• Health check: http://127.0.0.1:8000/health

Project Structure

mcp-fetch-server/
├── dist/mcp-fetch-server.exe   # Windows executable
├── src/mcp_fetch_server/       # Source code (tools, resources, prompts, security, ...)
├── tests/                      # 84 pytest tests
├── docs/                       # User manual + technical docs
├── scripts/docker-up.sh        # Linux/macOS stack startup
├── scripts/docker-up.ps1       # Windows stack startup
├── docker-compose.yml          # Full stack: MCP server + SearXNG
├── .env.docker.example         # Environment template for Docker Compose
├── Dockerfile                  # MCP server image
├── workspace/                  # Host folder mounted for local file tools (Docker)
├── searxng/settings.yml        # SearXNG config (JSON API enabled)
├── src/mcp_fetch_server/admin.py  # Management web GUI
└── .cursor/mcp.json            # Cursor config

Development

uv run pytest
uv run ruff check .
./scripts/docker-up.sh          # Linux / macOS
docker compose down

Windows only:

.\scripts\build_exe.ps1
.\scripts\docker-up.ps1

License

MIT