ScrapeUnblocker Remote MCP Server
OfficialREADME.md
# ScrapeUnblocker remote MCP server
A hosted (HTTP) [Model Context Protocol](https://modelcontextprotocol.io) server for
[ScrapeUnblocker](https://scrapeunblocker.com?utm_source=mcp&utm_medium=integration&utm_campaign=mcp-remote), deployed as a Vercel serverless
function. It lets **claude.ai** (web and mobile), Claude Desktop, Claude Code, and any
other MCP client fetch any web page's HTML - or AI-parsed JSON, or Google results -
through ScrapeUnblocker's anti-bot API, billed to your own account.
> Prefer a local install with no hosting? Use the stdio package instead:
> [`scrapeunblocker-mcp`](https://www.npmjs.com/package/scrapeunblocker-mcp).
Also listed on [Smithery](https://smithery.ai/servers/scrapeunblocker/scrapeunblocker).
## Endpoint
```
https://mcp.scrapeunblocker.com/mcp
```
You need a ScrapeUnblocker account - [create a free one](https://app.scrapeunblocker.com?utm_source=mcp&utm_medium=integration&utm_campaign=mcp-remote).
There are two ways to authenticate, and **you only need one of them**.
**A. Sign in with OAuth (recommended)** - nothing to copy or paste. The server is an
OAuth 2.1 Resource Server backed by Auth0, so the client runs a standard PKCE flow, and
the server resolves your ScrapeUnblocker key server-side from the account you signed in
with (the token is never passed through to the backend, per RFC 8707).
If your client asks for an OAuth client ID, use the public one - it is a PKCE client,
so it carries no secret and is safe to share:
```
5BM5Wk2dE4ABkDITmuKfemPLnn3QQ8jd
```
Leave the client secret field empty.
**B. Bring your own API key** - handy for scripts and clients without an OAuth flow.
Supply the key any of three ways:
1. `?key=YOUR_KEY` (or `?token=YOUR_KEY`) in the URL.
2. `Authorization: Bearer YOUR_KEY` header (a non-JWT value).
3. `x-scrapeunblocker-key: YOUR_KEY` header.
Prefer a header (or OAuth) whenever your client lets you set one. The `?key=` form exists
for clients that only accept a URL (ChatGPT, some connector UIs); a key in a URL can end
up in client logs or history, so treat that URL like the key itself and rotate the key in
the dashboard if it leaks.
Listing the tools (`initialize`, `tools/list`) needs no credentials at all, so MCP
directories and inspectors can introspect the server; actually running a tool does.
`https://mcp.scrapeunblocker.com/mcp/oauth` serves the same tools but answers nothing
without credentials: an unauthenticated client gets the 401 + `WWW-Authenticate`
challenge straight away. Use it with clients that start their OAuth flow only on that
401, such as Cursor with a static client ID (see below).
### OAuth configuration (maintainers)
| Env var | Purpose |
|---------|---------|
| `AUTH0_ISSUER` | Auth0 issuer URL, e.g. `https://TENANT.auth0.com/` (trailing slash). Enables OAuth together with `MCP_AUDIENCE`. |
| `MCP_AUDIENCE` | Canonical MCP URI = the Auth0 API Identifier, `https://mcp.scrapeunblocker.com/mcp`. The token `aud` must match this. |
| `MCP_EMAIL_CLAIM` | Optional namespaced claim carrying the user's email (set by an Auth0 Post-Login Action), e.g. `https://scrapeunblocker.com/email`. Falls back to the standard `email` claim. |
| `MCP_RESOURCE_METADATA_URL` | Optional override for the RFC 9728 metadata URL advertised in `WWW-Authenticate`. |
| `UTILS_API_BASE_URL` | utils-api base for the `email_to_key` lookup, e.g. `https://utils-api.scrapeunblocker.com`. |
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_REGION` | IAM creds (SigV4, `execute-api`) for calling utils-api. |
Discovery endpoints served (via `vercel.json` rewrites):
`/.well-known/oauth-protected-resource`, `/.well-known/oauth-protected-resource/mcp` and
`/.well-known/oauth-protected-resource/mcp/oauth`. All three name the same resource
(`MCP_AUDIENCE`), so a token for `/mcp` also works on `/mcp/oauth`.
## Add it to claude.ai
1. Customize → Connectors → **+** → **Add custom connector**.
2. **URL:** `https://mcp.scrapeunblocker.com/mcp`
3. Under **Advanced settings**, set **OAuth Client ID** to `5BM5Wk2dE4ABkDITmuKfemPLnn3QQ8jd`
and leave **OAuth Client Secret** empty.
4. **Add**, then sign in and **Authorize**. The tools appear across claude.ai web,
mobile, and Claude Desktop.
Or skip OAuth entirely and paste a personalised URL instead:
`https://mcp.scrapeunblocker.com/mcp?key=YOUR_API_KEY`
## Add it to Claude Code
```bash
claude mcp add --transport http scrapeunblocker "https://mcp.scrapeunblocker.com/mcp?key=YOUR_API_KEY"
```
## Add it to Cursor
Add this to `~/.cursor/mcp.json` (or the project's `.cursor/mcp.json`), then click
**Connect** next to the server in Cursor's MCP settings and sign in:
```json
{
"mcpServers": {
"scrapeunblocker": {
"url": "https://mcp.scrapeunblocker.com/mcp/oauth",
"auth": {
"CLIENT_ID": "D1jvSq2WpNu4Ue8WFWFTUpHsKtDfngkF",
"scopes": ["openid", "email", "offline_access", "mcp:use"]
}
}
}
}
```
The client ID is a public PKCE client registered for Cursor's redirect URLs, so there is
no secret. To use an API key instead, set `"url"` to
`https://mcp.scrapeunblocker.com/mcp` and add
`"headers": { "x-scrapeunblocker-key": "YOUR_API_KEY" }` in place of `auth`.
## Add it to ChatGPT
Use **ChatGPT on the web** ([chatgpt.com](https://chatgpt.com)) - not the desktop
app - on a paid plan (Plus/Pro/Business/Enterprise) with developer mode. All the
tools are read-only, so Plus is enough.
1. **Settings → Plugins**, turn on **Developer mode**.
2. In **Plugins**, click **+** to open **New Plugin**.
3. Keep **Connection** on **Server URL**, set **Authentication** to **No Auth**,
tick the risk box, and paste the URL with your key:
`https://mcp.scrapeunblocker.com/mcp?key=YOUR_API_KEY`
4. **Create**. The tools are then available in any chat.
Full walkthrough with screenshots:
[docs.scrapeunblocker.com/sdks/chatgpt](https://docs.scrapeunblocker.com/sdks/chatgpt).
## Tools
| Tool | What it does |
|------|--------------|
| `fetch_html` | Fetch the fully rendered HTML of a URL. |
| `browse_page` | Run `steps` on a page in a real browser (search, click, type, paginate), then return the resulting HTML. Not read-only: it acts on the website. |
| `capture_xhr` | Load a page in a real browser and return the JSON its own API calls received (every XHR/fetch whose URL contains a fragment such as `/api/search`), with optional `steps` to make the page fire the call. For sites that render from a JSON API refusing direct requests. Not read-only when steps act on the site. |
| `fetch_parsed` | Fetch a page and return AI-parsed structured JSON. |
| `google_search` | Run a Google search and return the results page as JSON: organic results with real URLs, favicons, sitelinks and ratings, ads, AI Overview, People also ask, related searches with links, knowledge panel, news, videos, local and shopping results. Set `full_urls: true` to get every result's exact page URL: Google gives out the last links up to about 40 seconds after the search, so the call takes that much longer (billed the same). Without it, a result whose link has not come yet has the site's home page as its `url`. |
| `fetch_markdown` | Fetch a page and return it as clean Markdown: the main content with absolute links, without scripts, styles, navigation or footer. Usually 2-25% the size of the HTML, so it is the cheap way for a model to read a page. |
| `search_and_read` | Search Google and read the top results in one call: Google's AI Overview (when there is one), the first 1-5 result pages as Markdown, and the other results as title, URL and snippet. Billed as one search plus one request per page read. |
| `list_elements` | Return a page's interactive elements (buttons, inputs, selects, links, forms) with ready-to-use selectors, as JSON, to build `steps`. |
| `list_plugins` | List the ready-made site plugins (100+: Amazon, Walmart, Zillow, Booking.com, Indeed, Google Trends, YouTube, App Store, ...) with their parameters. Free - it calls nothing. |
| `run_plugin` | Run one site plugin and return its structured JSON. |
| `amazon_search`, `amazon_product`, `amazon_reviews`, `walmart_search`, `ebay_search`, `google_shopping`, `google_local`, `maps_place_details`, `google_images`, `youtube_search`, `youtube_transcript`, `zillow_search`, `indeed_jobs`, `trustpilot_reviews`, `booking_hotels` | The most-used site plugins as tools of their own, each with its parameters in the tool schema - one call, no `list_plugins` lookup. Same JSON as `run_plugin`. |
### Interacting with a page (`steps`)
`browse_page` takes a `steps` array - an ordered list of browser
actions run in a real browser **after** the page loads, before the HTML is
captured. Supported actions: `wait_for`, `wait_for_text`, `wait`, `click`,
`type` (human-like keystrokes), `select`, `press_key`, `scroll`. A typical flow
is: call `list_elements` to discover selectors, then call `browse_page` with
`steps` to drive the page. It is the one tool not marked read-only, because
steps act on the website.
Steps are **non-idempotent** - the request runs once and is not retried. If a
step fails (bad selector, element never appeared), the tool returns an HTTP 422
result naming the offending step (`step_index`, `action`, `reason`, `selector`)
plus the page HTML at the point of failure, so you can correct the step.
## Deploy (maintainers)
This is a standard Vercel project - no build step, the function lives in `api/mcp.ts`.
`npm run typecheck` and `npm test` (the tools against a local stand-in for the API)
check it before a push.
```bash
vercel # preview
vercel --prod # production
```
Then point the `mcp.scrapeunblocker.com` domain at the Vercel project. Scraping calls
can be slow and some plugins take longer than a minute, so the function's `maxDuration`
is set to 300s in `vercel.json`.
## License
MIT
This server cannot be deployed
Maintenance
ActivityActive
ResponsivenessLimited data