remote-mcp-scraper-vercel

A production-ready Remote Model Context Protocol (MCP) server for Vercel Serverless on Node.js, built with Next.js App Router and TypeScript.

It exposes scraping-oriented MCP tools over the legacy HTTP + SSE transport:

Tools

All tool inputs are validated with strict zod schemas; outputs include structuredContent.

Related MCP server: Shark-no-Kari

Stack

• Next.js 16 (App Router, Node.js runtime) — route handlers on Vercel Functions

• @modelcontextprotocol/sdk — McpServer + SSEServerTransport, armSseKeepAlive heartbeats

• puppeteer-core + @sparticuz/chromium — brotli-packed chrome-headless-shell sized for the ~50 MB serverless bundle budget

• undici — request API (full header control incl. Sec-*, unlike fetch) + ProxyAgent

• socks — SOCKS4/5 CONNECT support via a custom undici connector (TLS-in-tunnel for HTTPS origins)

• zod — input/output validation

• vitest + eslint + tsc — validation (npm run validate)

Project layout

app/
  page.tsx                  minimal status page
  layout.tsx
  api/
    sse/route.ts            GET — opens MCP session over SSE
    message/route.ts        POST — JSON-RPC ingress for a session
    health/route.ts         liveness
lib/
  mcp.ts                    McpServer factory + tool registration
  sessions.ts               in-memory session registry + TTL sweep
  node-shims.ts             ServerResponse/IncomingMessage adapters over Web Streams
  fetcher.ts                stealth fetch client (UA rotation, proxies, retries, caps)
  user-agents.ts            rotating browser-identity profiles
  browser.ts                dynamic launcher: CDP → local binary → bundled chromium
  movie.ts                  TMDB / OMDb / Wikipedia metadata providers
  schemas.ts                zod input schemas
  auth.ts                   optional MCP_AUTH_TOKEN bearer gate
  rate-limit.ts             per-IP fixed-window limiter
tests/                      vitest unit tests
vercel.json                 function memory (1024 MB) + maxDuration + headers

Local development

npm install
cp .env.example .env   # fill in what you need (all optional for a first run)
npm run dev            # http://localhost:3000

Validate everything:

npm run validate       # eslint + tsc --noEmit + vitest + next build

Smoke-testing the MCP transport

# terminal 1 — open the SSE stream, note the sessionId in the `endpoint` event
curl -N http://localhost:3000/api/sse

# terminal 2 — drive the session
SID=<sessionId from the endpoint event>
curl -X POST "http://localhost:3000/api/message?sessionId=$SID" \
  -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"curl","version":"0.0"}}}'
curl -X POST "http://localhost:3000/api/message?sessionId=$SID" \
  -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","method":"notifications/initialized"}'
curl -X POST "http://localhost:3000/api/message?sessionId=$SID" \
  -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"ping","arguments":{"message":"hi"}}}'

Responses arrive as event: message frames on the SSE stream.

MCP client configuration

Any client that supports the SSE transport can connect to https://<your-deployment>/api/sse. For mcp-remote-style bridges or Claude Desktop:

{
  "mcpServers": {
    "scraper": {
      "url": "https://<your-deployment>.vercel.app/api/sse",
      "headers": { "Authorization": "Bearer <MCP_AUTH_TOKEN>" }
    }
  }
}

Deployment (Vercel)

npm i -g vercel
vercel          # preview
vercel --prod   # production

vercel.json already configures 1024 MB function memory and a 60 s maxDuration for both API routes — required for the bundled Chromium binary. Set environment variables in the Vercel dashboard (or vercel env add).

Environment variables

Proxy configuration

fetch_url and take_screenshot accept a proxy argument and honour the env vars above. Supported schemes:

• http:// / https:// — via undici ProxyAgent (CONNECT tunnelling, optional auth in the URL).

• socks5://, socks5h://, socks4://, socks4a:// — via socks + a custom undici connector; HTTPS origins get TLS inside the tunnel. Credentials may be embedded (socks5://user:pass@host:1080).

• For take_screenshot, the proxy is passed to Chromium as --proxy-server and embedded credentials are applied with page.authenticate().

Proxy credentials are redacted from all error messages.

Security & privacy

• Auth: optional MCP_AUTH_TOKEN bearer (header or ?token=); no auth is otherwise enforced — treat an unauthenticated deployment as public tooling.

• Rate limiting: fixed-window per-IP limiter on tool calls (instance-local, best-effort).

• Allowlisting: none — the tools fetch arbitrary http(s) URLs. If you expose this publicly, set MCP_AUTH_TOKEN and consider fronting it with your own allowlist; see limitations for SSRF context.

• No credential leakage: proxy userinfo is stripped from URLs used in errors, logs, and browser flags.

• Data handling: no persistence — request bodies/screenshots live only for the duration of a call; sessions live in instance memory.

Responsible use

fetch_url makes requests look like ordinary browser traffic to avoid incidental breakage — it is not a bypass for authentication, authorization, paywalls, CAPTCHAs, or site rate-limit policy, and must not be used as one. Honour robots.txt, the target site's terms of service, and applicable law. The MCP_AUTH_TOKEN gate and rate limiter are abuse dampers, not access control for the sites you scrape.

Known limitations

• Session affinity: the in-memory session map lives inside one function instance. POST /api/message must reach the same warm instance as GET /api/sse; a cold start or scale-out returns 404 unknown_session. For multi-instance deployments, switch to the MCP Streamable HTTP transport (StreamableHTTPServerTransport) with an external session store (e.g. Redis), or keep traffic single-instance.

• Function lifetime: SSE connections die at maxDuration (60 s default); MCP clients should reconnect and re-initialize.

• Cold starts: the first take_screenshot on a cold instance extracts ~50 MB of brotli-packed Chromium — expect a few seconds; warm calls are sub-second.

• Binary ceiling: @sparticuz/chromium ships chrome-headless-shell — no GPU rendering, no PDF printing; setGraphicsMode is disabled.

• Rate limiter is per-instance and memory-resident (not distributed).

• Node ≥ 20.9 required (Vercel nodejs20.x+ runtime).