Skip to main content
Glama
Sekharz

web-scrapper-mcp

by Sekharz
OverviewSchemaRelated ServersScoreDiscussions
Python
Local

web-scrapper-mcp

A portable MCP server that scrapes SPA/JS-rendered webpages and extracts structured API endpoint data. Registered globally in VS Code — callable from any project without reimplementing scraping.

Why this exists

Most API documentation sites (FMP, Tapetide, etc.) are React SPAs. httpx fetches only an empty shell. This server uses headless Chromium (Playwright) to fully render JavaScript, then extracts structured content.

Related MCP server: chromium-mcp

Tech Stack

Installation

git clone <repo>
cd web-scrapper-mcp
uv sync
uv run playwright install chromium
cp .env.example .env

Configuration

Edit .env:

SCRAPER_OUTPUT_DIR=output      # where screenshots are saved
SCRAPER_TIMEOUT_MS=30000       # page load timeout
# HTTPS_PROXY=http://proxy:8080  # optional — for Zscaler/corporate proxy

CLI Usage

# Generic scrape → markdown
uv run cli.py --url https://httpbin.org/html

# Scrape → plain text
uv run cli.py --url https://example.com --mode text

# Extract API endpoints (auto-detect site type)
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs --mode endpoints

# Save endpoint JSON to file
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs \
  --mode endpoints --out output/fmp_endpoints.json

# Screenshot
uv run cli.py --url https://example.com --mode screenshot

MCP Server Tools

scrape_page(url, wait_for?, selector?, mode?, wait_ms?)

Generic SPA scraper. Returns the rendered page as markdown, HTML, or plain text.

Param

Default

Description

url

required

URL to scrape

wait_for

networkidle

Playwright wait state

selector

""

CSS selector to wait for

mode

markdown

markdown | html | text

wait_ms

2000

Extra settle delay (ms)

Returns: {content, page_title, url, word_count, mode}

extract_api_endpoints(url, site_type?)

Extracts structured endpoint data. Auto-detects site type or accepts explicit override.

Site types: swagger | redoc | stoplight | fmp | generic

Returns:

{
  "endpoints": [
    {
      "method": "GET",
      "path": "/api/v3/stock/price",
      "summary": "Stock Price",
      "description": "...",
      "tags": ["Stock"],
      "parameters": [{"name": "symbol", "in": "query", "required": true, "type": "string"}]
    }
  ],
  "count": 247,
  "site_type": "fmp",
  "url": "..."
}

screenshot_page(url, full_page?)

Takes a full-page screenshot. Returns the saved file path and base64-encoded PNG for inline display.

Returns: {saved_path, base64}

scrape_and_seed(url, db_path, source_name, site_type?)

Power tool: scrapes API docs and seeds a SQLite database directly. The target DB must have an endpoints table (created automatically if absent).

Returns: {seeded, skipped, source, db_path}

Project Structure

web-scrapper-mcp/
  main.py               ← FastMCP server (4 tools)
  scraper.py            ← Core Playwright engine
  extractors/
    __init__.py
    generic.py          ← Any URL → markdown/html/text
    api_docs.py         ← Auto-detect swagger/redoc/stoplight/fmp/generic
    fmp.py              ← FMP developer docs specific extractor
  cli.py                ← CLI: uv run cli.py --url ...
  pyproject.toml
  .env.example
  .gitignore
  README.md
  .plan/                ← Architecture notes (gitignored)
  output/               ← Screenshots and output files (gitignored)
Install Server
F
license - not found
A
quality
C
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Tools

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Sekharz/web-scrapper-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server