Skip to main content
Glama
RebaubierWilliam

Web Scraper MCP Server

by RebaubierWilliam
OverviewSchemaRelated ServersScoreDiscussions
Python

Web Scraping with MCP

A Model Context Protocol (MCP) server that gives Claude (and any other MCP-compatible LLM client) the ability to scrape and extract data from live websites.

Inspired by luminati-io/web-scraping-with-mcp.

What it does

Tool

Description

fetch_page(url)

Loads a URL with a headless Chromium browser (JS rendered) and caches the HTML

extract_text(css_selector)

Extracts text from elements matching a CSS selector

extract_links()

Returns all hyperlinks found on the cached page

extract_table()

Extracts the first HTML table as structured JSON

extract_amazon()

Specialized extractor for Amazon product pages

list_cached_pages()

Lists all URLs currently in the cache

clear_cache(url?)

Clears one or all cached pages

Architecture

Claude Desktop / Cursor IDE
        │
        │  MCP (JSON-RPC over stdio)
        ▼
  scraper_mcp.py   ◄── FastMCP server
        │
        ├── Playwright  (headless browser, JS rendering)
        └── lxml        (HTML parsing / XPath / CSS selectors)

Installation

# 1. Clone this repo
git clone https://github.com/RebaubierWilliam/web-scraping-with-mcp.git
cd web-scraping-with-mcp

# 2. Create and activate a virtual environment
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install Playwright browsers
python -m playwright install chromium

Connect to Claude Desktop

  1. Open (or create) your Claude Desktop config file:

    • Windows: %APPDATA%\Claude\claude_desktop_config.json

    • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

  2. Add the following block, replacing the path:

{
  "mcpServers": {
    "web-scraper": {
      "command": "python",
      "args": ["C:/Users/YourName/Source/web-scraping-with-mcp/scraper_mcp.py"]
    }
  }
}

  1. Restart Claude Desktop. You should see a hammer icon confirming the tools are available.

Usage examples

Once connected to Claude Desktop, you can ask:

Fetch the page https://www.amazon.fr/dp/B09V3KXJPB and extract the product details.
Go to https://news.ycombinator.com and give me all the links.
Scrape the prices table from https://example.com/pricing

Run the server manually (for testing)

python scraper_mcp.py

The server communicates over stdio — it is meant to be launched by the MCP host, not used interactively.

Credits

This server cannot be installed

-
license - not tested
-
quality - not tested
D
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/RebaubierWilliam/web-scraping-with-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server