# Web Content Scraper MCP Worker

A high-performance Cloudflare Worker project that presents an agent-ready **Model Context Protocol (MCP) Tool** and **REST API endpoint** for dynamic web content extraction. Built with **TypeScript**, **Cloudflare Workflows**, **Cloudflare Browser Rendering (`@cloudflare/puppeteer`)**, and the **MCP SDK (`@modelcontextprotocol/sdk`)**, monetized with the **x402 payment protocol** ($0.005 USDC per call on Base).

---

## 🌟 Key Features

1. **Dynamic Page Scraping & Headless Chrome**:
   - Uses **Cloudflare Browser Rendering** (`@cloudflare/puppeteer`) to execute dynamic JavaScript, handle client-side rendering (SPAs/React/Vue/Next.js), and bypass anti-bot layout hurdles.
2. **Boilerplate & Ad Stripping**:
   - In-browser DOM parsing automatically strips scripts, stylesheets, navigation bars, headers, footers, sidebars, cookie banners, popups, and advertisement blocks.
3. **Structured Content Output**:
   - **Markdown Format** (`default`): Converts page hierarchy into clean structured Markdown (`# Headings`, paragraphs, lists, blockquotes, code blocks, links, and images).
   - **JSON Format**: Extracts metadata (`title`, `author`, `siteName`, `description`), text content, headings array, link objects, and image objects.
4. **Cloudflare Workflows Integration**:
   - Includes `ScraperWorkflow` for step-by-step asynchronous processing, step retries, and error isolation.
5. **x402 Protocol Monetization**:
   - Protected endpoints require **$0.005 USDC** payment per execution on the **Base network** via the x402 microtransaction standard.

---

## 🚀 MCP Tool Specification

### Tool Name: `scrape_web_content`

**Description**:
> "Scrapes a dynamic web page, bypasses client-side rendering, removes clutter/ads, and extracts core content as clean Markdown or structured JSON. Use this tool when you need to read web articles, documentation, or extract main body text from any URL."

**Parameters**:
- `url` (*string, required*): The HTTP/HTTPS web page URL to scrape.
- `format` (*string, optional*): `"markdown"` or `"json"` (default: `"markdown"`).

**Response Schema**:
```json
{
  "title": "Article Title",
  "sourceUrl": "https://example.com/article",
  "extractedContent": "# Article Title\n\nCore content formatted in clean markdown...",
  "wordCount": 482
}
```

---

## 📡 REST API Endpoints

### 1. Health Diagnostics
```http
GET /health
```
**Response**:
```json
{
  "status": "ok",
  "service": "web-scraper-mcp-worker",
  "timestamp": "2026-08-05T16:00:00.000Z",
  "browserConfigured": true,
  "workflowConfigured": true,
  "monetization": {
    "protocol": "x402",
    "price_usdc": "0.005",
    "network": "base",
    "payee_address": "0x742d35Cc6634C0532925a3b844Bc454e4438f44e"
  }
}
```

---

### 2. Scraping Endpoint (x402 Protected)
```http
POST /api/scrape
Content-Type: application/json

{
  "url": "https://example.com/blog/dynamic-article",
  "format": "markdown"
}
```

---

### 3. Workflow Trigger Endpoint
```http
POST /api/workflow/scrape
Content-Type: application/json

{
  "url": "https://example.com/docs/api",
  "format": "json"
}
```

---

### 4. MCP JSON-RPC Protocol Endpoint (x402 Protected)
```http
POST /mcp
Content-Type: application/json

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "scrape_web_content",
    "arguments": {
      "url": "https://news.ycombinator.com",
      "format": "markdown"
    }
  }
}
```

---

## 💳 x402 Monetization Flow

Unauthenticated requests to protected endpoints return an `HTTP 402 Payment Required` status code with x402 headers:

```http
HTTP/1.1 402 Payment Required
WWW-Authenticate: X-402 payee="0x742d35Cc6634C0532925a3b844Bc454e4438f44e" asset="0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913" amount="5000" network="eip155:8453"
Content-Type: application/json

{
  "x402Version": 1,
  "error": "Payment Required",
  "payment": {
    "scheme": "exact",
    "network": "eip155:8453",
    "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
    "amount": "5000",
    "human_amount": "$0.005 USDC",
    "payee": "0x742d35Cc6634C0532925a3b844Bc454e4438f44e",
    "resource": "/api/scrape",
    "description": "Dynamic Web Scraper & Content Extractor ($0.005 USDC per execution)"
  }
}
```

Include payment authorization proof in the `X-PAYMENT` header to execute requests.

---

## 🛠 Local Development & Deployment

### Prerequisites
- Node.js 18+
- Cloudflare Workers Paid account (for Browser Rendering in production)

### Commands
```bash
# Install dependencies
npm install

# Type Check
npm run type-check

# Local Development Server
npm run dev

# Deploy to Cloudflare Workers
npm run deploy
```

---

## 📄 License
MIT License.