MCP Web Scraper

A standalone Model Context Protocol (MCP) server that scrapes static HTML with BeautifulSoup. LLM analysis is handled by your MCP client (e.g. Odysseus with Ollama, or Open WebUI) — this server only provides scraping tools.

Designed for container deployment over Streamable HTTP.

What it does

Exposes two MCP tools:

Limitations: static HTML only (no JavaScript rendering), no anti-bot bypass. Respect site terms and rate limits.

Prerequisites

• Docker

Quick start (GHCR image)

After your first GitHub Release, pull the published image:

docker pull ghcr.io/<owner>/mcp-web-scraper:latest

docker run -d \
  --name mcp-web-scraper \
  -p 127.0.0.1:8000:8000 \
  ghcr.io/<owner>/mcp-web-scraper:latest

Verify health:

curl http://127.0.0.1:8000/health

MCP endpoint: http://127.0.0.1:8000/mcp

Make the GHCR package public

1. Go to your GitHub profile → Packages → mcp-web-scraper

2. Package settings → Change visibility → Public

Local build

cp .env.example .env
docker compose up --build -d

Connect to Odysseus

Odysseus uses native Streamable HTTP MCP (transport http) via the Python SDK's streamablehttp_client. No MCPO bridge required.

Odysseus requirements this server meets

Important: Leave MCP_API_KEY empty when using Odysseus. Odysseus does not send a Bearer token for HTTP MCP servers, and a 401 response would trigger its OAuth flow.

1. Run this MCP server (publish port 8000 or attach to the same Docker network as Odysseus).

2. In Odysseus Settings → MCP (admin), add a server:

3. Configure your LLM in Odysseus separately — the agent uses scraping tools from this server and its own model for analysis.

Compose overlay with Odysseus

Add to your Odysseus docker-compose.yml or an override file:

services:
  mcp-web-scraper:
    image: ghcr.io/<owner>/mcp-web-scraper:latest
    ports:
      - "127.0.0.1:8000:8000"
    restart: unless-stopped

  odysseus:
    depends_on:
      - mcp-web-scraper

Use http://mcp-web-scraper:8000/mcp as the MCP URL inside Odysseus.

Shared network overlay (separate compose projects)

If Odysseus and this server run in different docker compose projects, attach this service to the Odysseus network:

# Confirm the Odysseus network name (usually odysseus_default)
docker network ls | rg odysseus

docker compose -f docker-compose.yml -f docker-compose.odysseus.yml up -d

Register URL: http://mcp-web-scraper:8000/mcp

Odysseus in Docker (most common)

Odysseus's backend resolves the MCP URL inside its container. The hostname mcp-web-scraper only works when both containers share a Docker network; otherwise you get Temporary failure in name resolution and Odysseus may return HTTP 500.

Quickest fix — MCP published on the host (MCP_BIND=0.0.0.0, the default):

Odysseus's docker-compose.yml already sets extra_hosts: host.docker.internal:host-gateway. Verify from the Odysseus container:

docker exec -it odysseus-odysseus-1 curl -sf http://host.docker.internal:8000/health

Shared-network fix — use the service name instead of the host:

docker network ls | rg odysseus   # e.g. odysseus_default
docker compose -f docker-compose.yml -f docker-compose.odysseus.yml up -d

Then register http://mcp-web-scraper:8000/mcp.

Which MCP URL to use

Do not use your host LAN IP unless MCP_BIND publishes port 8000 on 0.0.0.0. The default compose file uses 0.0.0.0; set MCP_BIND=127.0.0.1 only if you want host-local access.

Troubleshooting Odysseus POST /api/mcp/servers → 500

The browser error is reported by Odysseus (localhost:7000), not this MCP server. A failed MCP connection normally returns HTTP 200 with "status": "error" in the JSON body — a 500 means Odysseus raised an unhandled exception.

1. Read Odysseus logs while saving the server (replace the container name if different):

   docker logs -f odysseus-odysseus-1 2>&1 | rg -i "mcp|error|traceback"

   Or, if Odysseus runs directly on the host, check the terminal where app.py is running.

2. Test reachability from Odysseus (run inside the Odysseus container):

   docker exec -it odysseus-odysseus-1 curl -sf http://mcp-web-scraper:8000/health
   # or, MCP on host:
   docker exec -it odysseus-odysseus-1 curl -sf http://host.docker.internal:8000/health

   Expect {"status":"ok"}. If this fails, fix networking before re-saving in the UI.

3. Confirm this server is up on the host:

   curl -sf http://127.0.0.1:8000/health

4. Leave MCP_API_KEY empty for Odysseus (see .env.example). Odysseus does not send Bearer tokens.

5. Use the exact path /mcp — not / or /sse.

6. Pull or rebuild the latest image after compatibility fixes:

   docker compose pull && docker compose up -d --build

Common log messages:

Connect to Open WebUI

Open WebUI v0.6.31+ supports MCP Streamable HTTP natively.

1. Admin Settings → External Tools → Add Server

2. Type: MCP (Streamable HTTP)

3. URL: http://host.docker.internal:8000/mcp

4. Auth: None (or Bearer if MCP_API_KEY is set)

5. Enable Function Calling: Native on your model

Configuration

Copy .env.example to .env:

Security

• Do not expose this server unauthenticated on the public internet.

• Set MCP_API_KEY for Open WebUI or other clients that support Bearer auth. Do not enable it for Odysseus.

• Scraping is unrestricted by default; only scrape sites you are permitted to access.

Creating a release

CI publishes to GHCR when a GitHub Release is created:

git tag v0.1.0
git push origin v0.1.0

Create a release from tag v0.1.0 on GitHub. The release workflow pushes:

• ghcr.io/<owner>/mcp-web-scraper:0.1.0

• ghcr.io/<owner>/mcp-web-scraper:0.1

• ghcr.io/<owner>/mcp-web-scraper:0

• ghcr.io/<owner>/mcp-web-scraper:latest

Development

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
ruff check src tests
pytest
mcp-web-scraper

License

MIT