# ScrapeUnblocker MCP server

A [Model Context Protocol](https://modelcontextprotocol.io) server that lets
Claude (and any other MCP client) fetch **any web page's HTML** - or run one of
100+ ready-made **site plugins** that return structured JSON - through the
[ScrapeUnblocker](https://scrapeunblocker.com?utm_source=mcp&utm_medium=integration&utm_campaign=mcp-server) scraping API, bypassing anti-bot
protection (Cloudflare, DataDome, PerimeterX, Akamai, Shape).

You bring your **own** API key. Nothing is shared or proxied through us.

## Tools

| Tool | What it does |
|------|--------------|
| `fetch_html` | Fetch the fully rendered HTML of a URL. |
| `browse_page` | Run interactive browser `steps` on a page (click, type, select, scroll, wait), then return the resulting HTML. Not read-only: it acts on the website. |
| `list_elements` | List a page's notable elements with a ready-to-use selector for each (selector discovery). |
| `fetch_parsed` | Fetch a page and return AI-parsed structured JSON. |
| `google_search` | Run a Google search and return the results page as JSON: organic results with real URLs, favicons, sitelinks and ratings, ads, AI Overview, People also ask, related searches with links, knowledge panel, news, videos, local and shopping results. Set `full_urls: true` to get every result's exact page URL: Google gives out the last links up to about 40 seconds after the search, so the call takes that much longer (billed the same). Without it, a result whose link has not come yet has the site's home page as its `url`. |
| `fetch_markdown` | Fetch a page and return it as clean Markdown: the main content with absolute links, without scripts, styles, navigation or footer. Usually 2-25% the size of the HTML, so it is the cheap way for a model to read a page. |
| `search_and_read` | Search Google and read the top results in one call: Google's AI Overview (when there is one), the first 1-5 result pages as Markdown, and the other results as title, URL and snippet. Billed as one search plus one request per page read. |
| `list_plugins` | List the ready-made site plugins (100+: Amazon, Walmart, Zillow, Booking.com, Indeed, Google Trends, YouTube, App Store, Ahrefs, ...) with their parameters. Free - it calls nothing. |
| `run_plugin` | Run one site plugin and return its structured JSON. |
| `amazon_search`, `amazon_product`, `amazon_reviews`, `walmart_search`, `ebay_search`, `google_shopping`, `google_local`, `maps_place_details`, `google_images`, `youtube_search`, `youtube_transcript`, `zillow_search`, `indeed_jobs`, `trustpilot_reviews`, `booking_hotels` | The most-used site plugins as tools of their own, each with its parameters in the tool schema - one call, no `list_plugins` lookup. Same JSON as `run_plugin`. |

### Site plugins (structured JSON for known sites)

For many popular sites there is a plugin that returns the data already
parsed - a Target search, a Zillow listing search, an app's App Store reviews,
Google Trends for a few keywords. The model calls `list_plugins` (optionally
with `search: "zillow"` or a `category`) to find the plugin and its
parameters, then `run_plugin` with `name` and `params` (the most-used plugins, listed in
the table above, can also be called directly as their own tools):

```json
{ "name": "target_search", "params": { "query": "coffee maker", "max_results": 20 } }
```

Every plugin and parameter is documented on the
[plugin pages](https://docs.scrapeunblocker.com/plugins/overview?utm_source=mcp&utm_medium=integration&utm_campaign=mcp-server).
Each run is billed like an API request.

### Browser steps (interact, then capture)

Some pages only reveal what you need after you interact with them - accept a
cookie banner, click a tab, type into a search box and submit, or scroll to
trigger lazy loading. Pass a `steps` array to `browse_page` and those
actions run in a **real browser**, in order, after the page loads; the resulting
HTML is then returned.

Available actions:

| Action | Fields |
|--------|--------|
| `wait_for` | `selector`, `selector_type?` (`css`/`xPath`/`className`/`tagName`), `timeout_ms?` |
| `wait_for_text` | `value`, `timeout_ms?` |
| `wait` | `value` (ms) |
| `click` | `selector`, `selector_type?`, `timeout_ms?` |
| `type` | `selector`, `selector_type?`, `value`, `clear?`, `timeout_ms?` (typed human-like) |
| `select` | `selector`, `selector_type?`, `value`, `timeout_ms?` |
| `press_key` | `value` (`Enter`, `Tab`, `Escape`, `Backspace`, `Delete`, `Space`, `Arrow*`, `Home`, `End`, `PageUp`, `PageDown`) |
| `scroll` | `value` (`"bottom"` or a pixel offset) |

Steps are **not idempotent** - they run once per call - and they act on the
website, so `browse_page` is the one tool not marked read-only. If a step fails,
`browse_page` returns which step failed, why, and the page HTML at that moment so
you can fix the selector and retry.

### List elements (discover selectors first)

`list_elements` loads a page and returns a JSON list of its notable elements
(links, inputs, buttons, selects, ...), each with a ready-to-use `selector` plus
`tag`, `text` and useful attributes (`name`, `id`, `type`, `placeholder`,
`aria_label`, `href`, ...):

```json
{ "url": "https://example.com", "count": 42, "elements": [ { "tag": "input", "selector": "#search", "type": "text", "placeholder": "Search", "aria_label": "Search" } ] }
```

The natural workflow is **discover, then act**: call `list_elements` to find the
selectors you need, then pass matching `steps` to `browse_page` to click/type/
select and capture the resulting HTML.

## Get an API key

Sign up and grab your key at **[app.scrapeunblocker.com](https://app.scrapeunblocker.com?utm_source=mcp&utm_medium=integration&utm_campaign=mcp-server)**. The server
reads it from the `SCRAPEUNBLOCKER_KEY` environment variable.

## Install

### Claude Code

The easiest route is the official **plugin**, which installs this server for you,
prompts for your API key (stored in your OS keychain rather than an environment
variable), and adds a `/scrape-url` command plus reference skills:

```
/plugin marketplace add ScrapeUnblocker/claude-code-plugin
/plugin install scrapeunblocker@scrapeunblocker
```

See [ScrapeUnblocker/claude-code-plugin](https://github.com/ScrapeUnblocker/claude-code-plugin).

To add the bare server instead:

```bash
claude mcp add scrapeunblocker \
  --env SCRAPEUNBLOCKER_KEY=your_api_key_here \
  -- npx -y scrapeunblocker-mcp
```

### Claude Desktop

Add this to your `claude_desktop_config.json`
(Settings → Developer → Edit Config):

```json
{
  "mcpServers": {
    "scrapeunblocker": {
      "command": "npx",
      "args": ["-y", "scrapeunblocker-mcp"],
      "env": {
        "SCRAPEUNBLOCKER_KEY": "your_api_key_here"
      }
    }
  }
}
```

Restart Claude Desktop and the ScrapeUnblocker tools appear.

### Any other MCP client

Run the server over stdio:

```bash
SCRAPEUNBLOCKER_KEY=your_api_key_here npx -y scrapeunblocker-mcp
```

## Example prompts

- "Fetch the HTML of https://www.example-shop.com/product/123 and list the price."
- "This page keeps blocking me: <url>. Use fetch_html to get it."
- "List the elements on <url>, then use browse_page to type 'laptop' into the search box, press Enter, wait for the results, and give me the HTML."
- "Search Google for 'best running shoes 2026' and give me the top 5 links."
- "Read https://en.wikipedia.org/wiki/Web_scraping as Markdown and summarise it in five bullets."
- "Search the web for the current population of Vilnius and cite the pages you read."
- "Find coffee makers on Target under $50 and compare them with Walmart's."
- "Show Google Trends for 'coffee' vs 'tea' in the US over the last year."
- "What are the newest App Store reviews of WhatsApp saying?"

## Development

```bash
npm install
npm run build      # bundles to dist/ with tsup
npm run typecheck
npm test           # builds, then runs the tools against a local stand-in API
SCRAPEUNBLOCKER_KEY=... node dist/index.js   # run the server
```

## License

MIT