# web-scrapper-mcp

A portable MCP server that scrapes SPA/JS-rendered webpages and extracts structured API endpoint data. Registered **globally** in VS Code — callable from any project without reimplementing scraping.

## Why this exists

Most API documentation sites (FMP, Tapetide, etc.) are React SPAs. `httpx` fetches only an empty shell. This server uses headless Chromium (Playwright) to fully render JavaScript, then extracts structured content.

## Tech Stack

- Python 3.14 + [uv](https://docs.astral.sh/uv/)
- [Playwright Python](https://playwright.dev/python/) (headless Chromium)
- [FastMCP](https://gofastmcp.com)
- [markdownify](https://github.com/matthewwithanm/python-markdownify)

## Installation

```bash
git clone <repo>
cd web-scrapper-mcp
uv sync
uv run playwright install chromium
cp .env.example .env
```

## Configuration

Edit `.env`:
```
SCRAPER_OUTPUT_DIR=output      # where screenshots are saved
SCRAPER_TIMEOUT_MS=30000       # page load timeout
# HTTPS_PROXY=http://proxy:8080  # optional — for Zscaler/corporate proxy
```

## CLI Usage

```bash
# Generic scrape → markdown
uv run cli.py --url https://httpbin.org/html

# Scrape → plain text
uv run cli.py --url https://example.com --mode text

# Extract API endpoints (auto-detect site type)
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs --mode endpoints

# Save endpoint JSON to file
uv run cli.py --url https://site.financialmodelingprep.com/developer/docs \
  --mode endpoints --out output/fmp_endpoints.json

# Screenshot
uv run cli.py --url https://example.com --mode screenshot
```

## MCP Server Tools

### `scrape_page(url, wait_for?, selector?, mode?, wait_ms?)`
Generic SPA scraper. Returns the rendered page as markdown, HTML, or plain text.

| Param | Default | Description |
|---|---|---|
| `url` | required | URL to scrape |
| `wait_for` | `networkidle` | Playwright wait state |
| `selector` | `""` | CSS selector to wait for |
| `mode` | `markdown` | `markdown` \| `html` \| `text` |
| `wait_ms` | `2000` | Extra settle delay (ms) |

**Returns:** `{content, page_title, url, word_count, mode}`

---

### `extract_api_endpoints(url, site_type?)`
Extracts structured endpoint data. Auto-detects site type or accepts explicit override.

**Site types:** `swagger` | `redoc` | `stoplight` | `fmp` | `generic`

**Returns:**
```json
{
  "endpoints": [
    {
      "method": "GET",
      "path": "/api/v3/stock/price",
      "summary": "Stock Price",
      "description": "...",
      "tags": ["Stock"],
      "parameters": [{"name": "symbol", "in": "query", "required": true, "type": "string"}]
    }
  ],
  "count": 247,
  "site_type": "fmp",
  "url": "..."
}
```

---

### `screenshot_page(url, full_page?)`
Takes a full-page screenshot. Returns the saved file path and base64-encoded PNG for inline display.

**Returns:** `{saved_path, base64}`

---

### `scrape_and_seed(url, db_path, source_name, site_type?)`
Power tool: scrapes API docs and seeds a SQLite database directly. The target DB must have an `endpoints` table (created automatically if absent).

**Returns:** `{seeded, skipped, source, db_path}`

## Project Structure

```
web-scrapper-mcp/
  main.py               ← FastMCP server (4 tools)
  scraper.py            ← Core Playwright engine
  extractors/
    __init__.py
    generic.py          ← Any URL → markdown/html/text
    api_docs.py         ← Auto-detect swagger/redoc/stoplight/fmp/generic
    fmp.py              ← FMP developer docs specific extractor
  cli.py                ← CLI: uv run cli.py --url ...
  pyproject.toml
  .env.example
  .gitignore
  README.md
  .plan/                ← Architecture notes (gitignored)
  output/               ← Screenshots and output files (gitignored)
```