# web-scrape

CLI-first website scrape + anonymous browser fetch.

- **SSRF-safe** public `http(s)` only
- **robots.txt** honored (fail-open on missing/unreachable robots)
- **Protection taxonomy** — blocked / captcha / Cloudflare / login / rate-limit statuses instead of garbage text
- **Optional CapSolver** — Turnstile + Cloudflare “Just a moment…” clears (sticky residential proxy; CapSolver needs **socks5** for Webshare)
- **Thin MCP adapter** — same library over JSON-RPC HTTP

> Does **not** bypass protections by default. CapSolver is opt-in via env. No social-login profiles or cookies are reused.

## Install

```bash
git clone https://github.com/teslashibe/web-scrape.git
cd web-scrape
npm install
npx playwright install chromium   # for `fetch` / browser path
```

Node **≥ 20**. Uses `undici` **v7** (compatible with Node 20).

## CLI (preferred for agents)

```bash
# Readable browser fetch (Playwright)
./bin/web-scrape.mjs fetch https://example.com/ --json

# Structured single-page scrape (HTTP + HTML extract)
./bin/web-scrape.mjs scrape https://example.com/ --json

# MCP HTTP server (default :8091)
./bin/web-scrape.mjs serve --port 8091
```

Exit codes: `0` ok · `2` structured non-ok status · `1` usage/error.

## Library

```js
import { scrape, browserFetchURL } from "web-scrape";

const page = await browserFetchURL({ url: "https://example.com/" });
const brief = await scrape({ url: "https://example.com/" });
```

## Docker / Kubernetes

```bash
docker build -t web-scrape .
docker run --rm -p 8091:8091 \
  -e WEB_FETCH_TURNSTILE_PROVIDER=capsolver \
  -e WEB_FETCH_TURNSTILE_API_KEY=CAP-... \
  -e WEBSHARE_RESIDENTIAL_USER=... \
  -e WEBSHARE_RESIDENTIAL_PASS=... \
  -e WEBSHARE_USERNAME_TEMPLATE='{user}-{country}-1' \
  web-scrape
```

Expose Service port `8091`. Probe `GET /mcp/v1/ready` (200 ready / 503 not_ready).

## MCP

```
POST /mcp/v1          JSON-RPC 2.0 (initialize, tools/list, tools/call)
GET  /mcp/v1/health   liveness + limits + metrics
GET  /mcp/v1/ready    readiness
```

Tools:

| Tool | Purpose |
| --- | --- |
| `browser_fetch_url` | Ephemeral Playwright fetch → readable text or structured status |
| `scrape_website_context` | Single-page HTTP scrape → title/summary/product/audience fields |

## CapSolver + proxy (optional)

| Env | Purpose |
| --- | --- |
| `WEB_FETCH_TURNSTILE_PROVIDER=capsolver` | Enable provider |
| `WEB_FETCH_TURNSTILE_API_KEY` | CapSolver key |
| `WEBSHARE_RESIDENTIAL_USER` / `PASS` | Residential proxy |
| `WEBSHARE_USERNAME_TEMPLATE` | Prefer sticky `{user}-{country}-1` (required for `cf_clearance` IP affinity) |
| `WEB_SCRAPER_PROXY_URL` | Full proxy URL override |
| `BROWSER_FETCH_TIMEOUT_MS` | Playwright deadline (default rises when CapSolver+proxy set; max 180s) |

CapSolver `AntiCloudflareTask` is sent as `socks5:host:port:user:pass` (no page `html` — CapSolver rejects it as `invalid html`). Playwright egress stays HTTP proxy. Managed “Just a moment…” pages use AntiCloudflareTask even when a Turnstile iframe is present. Never log proxy credentials or API keys.

## Tests

```bash
npm test          # unit fixtures (no live network)
npm run validate  # loopback MCP discovery/call
npm run smoke     # status-matrix smoke
```

## License

Apache-2.0. CapSolver / Webshare are optional third-party paid services — you bring your own keys.