Skip to main content
Glama
Sekharz

web-scrapper-mcp

by Sekharz
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

web-scrapper-mcp

A portable MCP server that scrapes SPA/JS-rendered webpages and extracts structured API endpoint data. Registered globally in VS Code — callable from any project without reimplementing scraping.

Why this exists

Most API documentation sites (FMP, Tapetide, etc.) are React SPAs. httpx fetches only an empty shell. This server uses headless Chromium (Playwright) to fully render JavaScript, then extracts structured content.

Related MCP server: kage-mcp

Tech Stack

Installation

git clone <repo>
cd web-scrapper-mcp
uv sync
uv run playwright install chromium
cp .env.example .env

Configuration

Edit .env:

SCRAPER_OUTPUT_DIR=output      # where screenshots are saved
SCRAPER_TIMEOUT_MS=30000       # page load timeout
# HTTPS_PROXY=http://proxy:8080  # optional — for Zscaler/corporate proxy

CLI Usage

# Generic scrape → markdown
uv run cli.py --url https://httpbin.org/html

# Scrape → plain text
uv run cli.py --url https://example.com --mode text

# Extract API endpoints (auto-detect site type)
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs --mode endpoints

# Save endpoint JSON to file
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs \
  --mode endpoints --out output/fmp_endpoints.json

# Screenshot
uv run cli.py --url https://example.com --mode screenshot

MCP Server Tools

scrape_page(url, wait_for?, selector?, mode?, wait_ms?)

Generic SPA scraper. Returns the rendered page as markdown, HTML, or plain text.

Param

Default

Description

url

required

URL to scrape

wait_for

networkidle

Playwright wait state

selector

""

CSS selector to wait for

mode

markdown

markdown | html | text

wait_ms

2000

Extra settle delay (ms)

Returns: {content, page_title, url, word_count, mode}

extract_api_endpoints(url, site_type?)

Extracts structured endpoint data. Auto-detects site type or accepts explicit override.

Site types: swagger | redoc | stoplight | fmp | generic

Returns:

{
  "endpoints": [
    {
      "method": "GET",
      "path": "/api/v3/stock/price",
      "summary": "Stock Price",
      "description": "...",
      "tags": ["Stock"],
      "parameters": [{"name": "symbol", "in": "query", "required": true, "type": "string"}]
    }
  ],
  "count": 247,
  "site_type": "fmp",
  "url": "..."
}

screenshot_page(url, full_page?)

Takes a full-page screenshot. Returns the saved file path and base64-encoded PNG for inline display.

Returns: {saved_path, base64}

scrape_and_seed(url, db_path, source_name, site_type?)

Power tool: scrapes API docs and seeds a SQLite database directly. The target DB must have an endpoints table (created automatically if absent).

Returns: {seeded, skipped, source, db_path}

Project Structure

web-scrapper-mcp/
  main.py               ← FastMCP server (4 tools)
  scraper.py            ← Core Playwright engine
  extractors/
    __init__.py
    generic.py          ← Any URL → markdown/html/text
    api_docs.py         ← Auto-detect swagger/redoc/stoplight/fmp/generic
    fmp.py              ← FMP developer docs specific extractor
  cli.py                ← CLI: uv run cli.py --url ...
  pyproject.toml
  .env.example
  .gitignore
  README.md
  .plan/                ← Architecture notes (gitignored)
  output/               ← Screenshots and output files (gitignored)

Available Tools

4 tools
extract_api_endpointsA

Extract structured API endpoint data from an API documentation site.

Automatically detects the site type (swagger, redoc, stoplight, fmp, generic). Override detection by passing site_type explicitly.

Args: url: URL of the API documentation page. site_type: Force a specific extractor — 'swagger' | 'redoc' | 'stoplight' | 'fmp' | 'generic'.

Returns: { endpoints: [{method, path, operation_id, summary, description, tags, parameters}], count: int, site_type: str, url: str, }

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
site_typeNo

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It explains the auto-detection behavior and return structure. It does not mention edge cases like invalid URLs or unrecognized site types, but for a read-only extraction tool, the main behavior is well-covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is structured with purpose, note about auto-detection, and clearly labeled args and returns. It is reasonably concise, though the return structure could be considered slightly redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simple tool with 2 parameters and no output schema, the description covers the main aspects: what it does, how to use, and what output to expect. It lacks error handling details but is otherwise complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, but the description provides clear meaning for both parameters: url (the documentation page URL) and site_type (with explicit allowed values to force a specific extractor). This adds significant value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool extracts structured API endpoint data from API documentation sites. It distinguishes from sibling tools like scrape_page and screenshot_page, which are for general scraping and screenshots.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description specifies that it works on API documentation sites and can auto-detect site types or be overridden, providing clear usage context. It does not explicitly state when not to use or compare to siblings, but the sibling names imply alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrape_and_seedA

Scrape API documentation and seed a finanal-style SQLite database directly.

The target database must have an 'endpoints' table with columns: path, method, operation_id, summary, description, tags, parameters, source

Args: url: URL of the API documentation page. db_path: Absolute path to the SQLite database file. source_name: Label for the 'source' column (e.g. 'fmp', 'tapetide'). site_type: Optional site type override for the extractor.

Returns: {seeded: int, skipped: int, source: str, db_path: str}

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
db_pathYes
site_typeNo
source_nameYes

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description bears full responsibility for behavioral disclosure. It mentions the required database table structure and return format, but lacks details on error conditions, idempotency, or whether existing data is overwritten.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is reasonably concise, with a brief summary followed by parameter definitions. It includes necessary details like the database table schema, which is valuable context, but could be slightly more compact without losing clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given four parameters, no output schema, and no annotations, the description provides the database schema and return format, and explains each parameter. However, it lacks details on error handling and preconditions, which would make it more complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Since schema description coverage is 0%, the description compensates by explaining each parameter clearly: url as 'URL of the API documentation page', db_path as 'Absolute path to the SQLite database file', source_name with examples, and site_type as an optional override. This adds significant meaning beyond the bare schema titles.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Scrape API documentation and seed a finanal-style SQLite database directly', using specific verbs and resources. It distinguishes from siblings by combining scraping and seeding, while siblings like 'extract_api_endpoints' only extract and 'scrape_page' only scrape.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not explicitly state when to use this tool versus siblings. The purpose of combining scraping and seeding is implied, but no direct guidance or exclusions are provided, leaving the agent to infer usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrape_pageA

Scrape any SPA or JS-rendered webpage and return its content.

Args: url: The page URL to scrape. wait_for: Playwright wait state — 'networkidle' | 'load' | 'domcontentloaded'. selector: Optional CSS selector to wait for before returning content. mode: Output format — 'markdown' | 'html' | 'text'. wait_ms: Extra settle delay in milliseconds after the page loads.

Returns: {content, page_title, url, word_count, mode}

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
modeNomarkdown
wait_msNo
selectorNo
wait_forNonetworkidle

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It mentions using Playwright for JS rendering and describes the return structure, but does not disclose potential side effects, authentication needs, or error handling behaviors.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a clear purpose sentence followed by an Args list and Returns. Every sentence provides necessary information, and it is appropriately sized without fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description includes return fields. It covers parameter details and the core scraping behavior. However, it lacks error handling or timeout information. Overall, it is fairly complete for the tool's complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% description coverage, so the description compensates by explaining each parameter's purpose (e.g., 'wait_for' as Playwright wait state, 'selector' as CSS selector). It adds meaning beyond the schema, though some descriptions are minimal.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it scrapes any SPA or JS-rendered webpage and returns content. It distinguishes from siblings like screenshot_page (screenshots) and extract_api_endpoints (extracts endpoints) by focusing on content extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description does not provide explicit guidance on when to use this tool versus siblings. It lacks any 'when not to use' or alternative suggestions, requiring the agent to infer usage from the tool name and description.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

screenshot_pageA

Take a full-page screenshot of a URL for visual debugging.

Returns the file path and the PNG encoded as base64 so the agent can display it inline.

Args: url: URL to screenshot. full_page: Capture the full scrollable page (default True).

Returns: {saved_path: str, base64: str}

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
full_pageNo

TDQS

A4.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations, but description discloses return format (saved_path and base64). Does not mention side effects, but operation is read-only by nature.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Very concise, front-loaded purpose, uses bullet points for args and returns. No unnecessary text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers functionality, parameters, and return values adequately. No output schema, but description explains return object. Complete for a simple tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, but description provides clear explanations for both parameters (url, full_page) including default for full_page. Adds meaning beyond schema types.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states it takes a full-page screenshot of a URL for visual debugging. Distinct from siblings like scrape_page (text extraction) and extract_api_endpoints.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly mentions 'for visual debugging', implying use case. Does not explicitly exclude alternatives, but context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 4 tool updatesv0.1.0
    • First observedextract_api_endpoints
    • First observedscrape_and_seed
    • First observedscrape_page
    • First observedscreenshot_page
Deploy Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.1/5.0

Scored across 4 tools

Disambiguation5/5

Each tool has a clearly distinct purpose: extracting API endpoints, seeding a database, scraping general web pages, and taking screenshots. No two tools overlap in function.

Naming Consistency3/5

All names use snake_case, but the verb patterns are inconsistent: 'extract_api_endpoints' and 'scrape_page' follow verb_noun, while 'scrape_and_seed' uses 'and', and 'screenshot_page' is a noun_verb combination.

Tool Count5/5

4 tools is well-scoped for a web scraping server. Each tool covers a core functionality without unnecessary redundancy or missing essentials.

Completeness4/5

The server covers core scraping needs (API extraction, general scraping, screenshot, database seeding). Minor gaps like multi-page crawling or link extraction are absent but not critical for the stated purpose.

Maintenance

ActivityStale
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • chromium-mcp

    gzigurella
    A
    license
    A
    quality
    D
    maintenance
    MCP server for web fetching and automation using Chromium headless browser. Enables fetching pages as markdown, taking screenshots, automating interactions, and extracting data via CSS selectors.
    5
    MIT

  • kage-mcp

    lxyct825-ops
    A
    license
    Not graded
    quality
    C
    maintenance
    A lightweight MCP server for parsing HTML, fetching URLs, rendering terminal-style screenshots, and executing JavaScript on static HTML without external dependencies.
    3
    MIT

  • scrapy-mcp

    eitan3
    A
    license
    A
    quality
    D
    maintenance
    A headless web-scraping MCP server built on Scrapy, providing tools for polite fetching, CSS/XPath extraction, link/table extraction, sitemap and robots.txt reading, and bounded asynchronous crawls.
    10
    MIT

  • mcp-page-capture

    chasesaurabh
    A
    license
    A
    quality
    D
    maintenance
    An MCP server that uses headless Chromium (Puppeteer) to capture pixel-perfect screenshots and extract DOM from URLs, with LLM-friendly step-based workflows.
    2
    13
    3
    MIT