A
licenseA
qualityD
maintenanceMCP server for web fetching and automation using Chromium headless browser. Enables fetching pages as markdown, taking screenshots, automating interactions, and extracting data via CSS selectors.
5
MIT
web-scrapper-mcp
by Sekharz
Provides tools for extracting structured API endpoint data from Swagger-documented API documentation sites.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@web-scrapper-mcp Extract API endpoints from https://site.financialmodelingprep.com/developer/docs"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
web-scrapper-mcp
A portable MCP server that scrapes SPA/JS-rendered webpages and extracts structured API endpoint data. Registered globally in VS Code — callable from any project without reimplementing scraping.
Why this exists
Most API documentation sites (FMP, Tapetide, etc.) are React SPAs. httpx fetches only an empty shell. This server uses headless Chromium (Playwright) to fully render JavaScript, then extracts structured content.
Related MCP server: kage-mcp
Tech Stack
Python 3.14 + uv
Playwright Python (headless Chromium)
Installation
git clone <repo>
cd web-scrapper-mcp
uv sync
uv run playwright install chromium
cp .env.example .envConfiguration
Edit .env:
SCRAPER_OUTPUT_DIR=output # where screenshots are saved
SCRAPER_TIMEOUT_MS=30000 # page load timeout
# HTTPS_PROXY=http://proxy:8080 # optional — for Zscaler/corporate proxyCLI Usage
# Generic scrape → markdown
uv run cli.py --url https://httpbin.org/html
# Scrape → plain text
uv run cli.py --url https://example.com --mode text
# Extract API endpoints (auto-detect site type)
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs --mode endpoints
# Save endpoint JSON to file
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs \
--mode endpoints --out output/fmp_endpoints.json
# Screenshot
uv run cli.py --url https://example.com --mode screenshotMCP Server Tools
scrape_page(url, wait_for?, selector?, mode?, wait_ms?)
Generic SPA scraper. Returns the rendered page as markdown, HTML, or plain text.
Param | Default | Description |
| required | URL to scrape |
|
| Playwright wait state |
|
| CSS selector to wait for |
|
|
|
|
| Extra settle delay (ms) |
Returns: {content, page_title, url, word_count, mode}
extract_api_endpoints(url, site_type?)
Extracts structured endpoint data. Auto-detects site type or accepts explicit override.
Site types: swagger | redoc | stoplight | fmp | generic
Returns:
{
"endpoints": [
{
"method": "GET",
"path": "/api/v3/stock/price",
"summary": "Stock Price",
"description": "...",
"tags": ["Stock"],
"parameters": [{"name": "symbol", "in": "query", "required": true, "type": "string"}]
}
],
"count": 247,
"site_type": "fmp",
"url": "..."
}screenshot_page(url, full_page?)
Takes a full-page screenshot. Returns the saved file path and base64-encoded PNG for inline display.
Returns: {saved_path, base64}
scrape_and_seed(url, db_path, source_name, site_type?)
Power tool: scrapes API docs and seeds a SQLite database directly. The target DB must have an endpoints table (created automatically if absent).
Returns: {seeded, skipped, source, db_path}
Project Structure
web-scrapper-mcp/
main.py ← FastMCP server (4 tools)
scraper.py ← Core Playwright engine
extractors/
__init__.py
generic.py ← Any URL → markdown/html/text
api_docs.py ← Auto-detect swagger/redoc/stoplight/fmp/generic
fmp.py ← FMP developer docs specific extractor
cli.py ← CLI: uv run cli.py --url ...
pyproject.toml
.env.example
.gitignore
README.md
.plan/ ← Architecture notes (gitignored)
output/ ← Screenshots and output files (gitignored)Available Tools
4 toolsextract_api_endpointsA
Extract structured API endpoint data from an API documentation site.
Automatically detects the site type (swagger, redoc, stoplight, fmp, generic). Override detection by passing site_type explicitly.
Args: url: URL of the API documentation page. site_type: Force a specific extractor — 'swagger' | 'redoc' | 'stoplight' | 'fmp' | 'generic'.
Returns: { endpoints: [{method, path, operation_id, summary, description, tags, parameters}], count: int, site_type: str, url: str, }
Inputs: url (required), site_type.
scrape_and_seedA
Scrape API documentation and seed a finanal-style SQLite database directly.
The target database must have an 'endpoints' table with columns: path, method, operation_id, summary, description, tags, parameters, source
Args: url: URL of the API documentation page. db_path: Absolute path to the SQLite database file. source_name: Label for the 'source' column (e.g. 'fmp', 'tapetide'). site_type: Optional site type override for the extractor.
Returns: {seeded: int, skipped: int, source: str, db_path: str}
Inputs: url (required), db_path (required), source_name (required), site_type.
scrape_pageA
Scrape any SPA or JS-rendered webpage and return its content.
Args: url: The page URL to scrape. wait_for: Playwright wait state — 'networkidle' | 'load' | 'domcontentloaded'. selector: Optional CSS selector to wait for before returning content. mode: Output format — 'markdown' | 'html' | 'text'. wait_ms: Extra settle delay in milliseconds after the page loads.
Returns: {content, page_title, url, word_count, mode}
Inputs: url (required), mode, wait_ms, selector, +1 more.
screenshot_pageA
Take a full-page screenshot of a URL for visual debugging.
Returns the file path and the PNG encoded as base64 so the agent can display it inline.
Args: url: URL to screenshot. full_page: Capture the full scrollable page (default True).
Returns: {saved_path: str, base64: str}
Inputs: url (required), full_page.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
A4.1/5.0
Scored across 4 tools
Disambiguation5/5
Each tool has a clearly distinct purpose: extracting API endpoints, seeding a database, scraping general web pages, and taking screenshots. No two tools overlap in function.
Naming Consistency3/5
All names use snake_case, but the verb patterns are inconsistent: 'extract_api_endpoints' and 'scrape_page' follow verb_noun, while 'scrape_and_seed' uses 'and', and 'screenshot_page' is a noun_verb combination.
Tool Count5/5
4 tools is well-scoped for a web scraping server. Each tool covers a core functionality without unnecessary redundancy or missing essentials.
Completeness4/5
The server covers core scraping needs (API extraction, general scraping, screenshot, database seeding). Minor gaps like multi-page crawling or link extraction are absent but not critical for the stated purpose.
Related MCP Connectors
Web scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.
Web scraping MCP server: any URL to Markdown or JSON, with JavaScript rendering and batch jobs.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
One MCP server for 180+ live web-data APIs returning clean JSON from sites that block scrapers.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceA lightweight MCP server for parsing HTML, fetching URLs, rendering terminal-style screenshots, and executing JavaScript on static HTML without external dependencies.3MIT
- AlicenseAqualityDmaintenanceA headless web-scraping MCP server built on Scrapy, providing tools for polite fetching, CSS/XPath extraction, link/table extraction, sitemap and robots.txt reading, and bounded asynchronous crawls.10MIT
- AlicenseAqualityDmaintenanceAn MCP server that uses headless Chromium (Puppeteer) to capture pixel-perfect screenshots and extract DOM from URLs, with LLM-friendly step-based workflows.219 npm3MIT