# 🐶 ScrapingDog MCP Server

**The complete [Model Context Protocol](https://modelcontextprotocol.io) server for [ScrapingDog](https://www.scrapingdog.com) — all 77 API endpoints as AI-ready tools.**

Give Claude (or any MCP client) the power to scrape any webpage, search Google, pull Amazon product data, read LinkedIn profiles, fetch YouTube transcripts, track Google Trends, and much more — through one API key.

[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Node >= 18](https://img.shields.io/badge/node-%3E%3D18-brightgreen.svg)](https://nodejs.org)
[![MCP](https://img.shields.io/badge/protocol-MCP-8A2BE2.svg)](https://modelcontextprotocol.io)

Built by extracting the **entire official documentation** (~90 pages at [scrapingdog.com/documentation](https://www.scrapingdog.com/documentation/)) into machine-readable specs, then **verifying endpoints against the live API** — including fixing endpoints the official docs get wrong (see [Deviations](#-deviations-from-the-official-docs)).

---

## ⚡ Quick start

**1. Get an API key** — free 200-credit trial, no credit card: [api.scrapingdog.com/register](https://api.scrapingdog.com/register)

**2. Install** (pick your client):

### Claude Code (recommended: user scope = available in every project)

```bash
claude mcp add --scope user scrapingdog -e SCRAPINGDOG_API_KEY=YOUR_KEY -- npx -y github:alessandrobenigni/ScrapingDog-MCP
```

### Claude Desktop

Add to `claude_desktop_config.json` (Settings → Developer → Edit Config):

```json
{
  "mcpServers": {
    "scrapingdog": {
      "command": "npx",
      "args": ["-y", "github:alessandrobenigni/ScrapingDog-MCP"],
      "env": { "SCRAPINGDOG_API_KEY": "YOUR_KEY" }
    }
  }
}
```

### Cursor / Windsurf / any MCP client

Same JSON shape as above in the client's MCP config (`.cursor/mcp.json`, `~/.codeium/windsurf/mcp_config.json`, …).

### From a local clone (fastest startup, easiest to hack on)

```bash
git clone https://github.com/alessandrobenigni/ScrapingDog-MCP.git
cd ScrapingDog-MCP
npm install
claude mcp add --scope user scrapingdog -e SCRAPINGDOG_API_KEY=YOUR_KEY -- node "$(pwd)/src/index.js"
```

**3. Use it** — just ask your AI naturally:

> *"Search Google for 'best espresso machine 2026' and summarize the top results"*
> *"Scrape this product page and extract the price"*
> *"Get the transcript of this YouTube video"*
> *"What's trending on Google Trends for 'AI agents' in the US this year?"*
> *"Pull the LinkedIn company profile for Anthropic"*
> *"How many ScrapingDog credits do I have left?"* (→ `account` tool)

That's it. No proxy management, no CAPTCHAs, no blocked requests.

---

## 🧰 The 77 tools

Every tool maps 1:1 to a ScrapingDog endpoint, with every documented parameter typed and described. Credit costs are per successful request and are shown in each tool's description.

### 🌐 Web Scraping
| Tool | What it does | Credits |
|---|---|---|
| `scrape` | Scrape any URL → raw HTML. JS rendering (`dynamic`), premium residential proxies (`premium`), geotargeting (`country`), sticky sessions (`session_number`), Cloudflare/CAPTCHA bypass (`stealth_mode`), custom header forwarding | 1–25 |
| `scrape_post` | Forward a POST body (forms, POST-only APIs) through the proxy layer | 1+ |
| `screenshot` | Screenshot any page (viewport or full-page, png/jpg/webp) → returned as an actual image | 5 |

### 🔍 Google Search & AI
| Tool | What it does | Credits |
|---|---|---|
| `google_search` | Full SERP as JSON: organic results, ads, **AI Overviews**, knowledge graph, local pack, top stories, related questions/searches, inline images/videos/shopping, discussions, immersive products, pagination | 5 |
| `google_ai_mode` | Google's AI Mode answers | 10 |
| `google_ai_overview` | Standalone AI Overview extraction | 5 |
| `google_autocomplete` | Query suggestions | — |
| `google_lens` | Visual search by image URL | — |

### 🗺️ Google Maps, Trends, News
| Tool | What it does |
|---|---|
| `google_maps_search` / `google_maps_places` / `google_maps_reviews` / `google_maps_photos` / `google_maps_posts` | Complete Maps data: place search, place details, paginated reviews, photos, business posts |
| `google_trends` / `google_trends_autocomplete` / `google_trends_trending_now` | Interest over time, entity lookup, real-time trending topics |
| `google_news_search` / `google_news_v2` | News SERP + the newer topic-token News API |

### 🎓 Google Scholar & Patents
`google_scholar` · `google_scholar_profiles` · `google_scholar_author` · `google_scholar_author_citation` · `google_scholar_cite` · `google_patents` · `google_patent_details`

> Tip: `google_patents` returns `patent_id` as `patent/US11734097B1/en`; pass just the publication number (`US11734097B1`) to `google_patent_details`.

### 🛍️ Google Shopping & verticals
`google_shopping` · `google_immersive_product` · `google_images` · `google_videos` · `google_shorts` · `google_finance` · `google_jobs` · `google_local` · `google_hotels` · `google_flights` · `google_ads_transparency`

> Multi-step flows are documented in the tool descriptions — e.g. Flights chains `departure_token` → `booking_token`; Shopping filters chain via `shoprs`.

### 🔎 Other search engines
`bing_search` · `bing_shopping` · `duckduckgo_search` · `baidu_search` · `universal_search` (20 credits, engine-agnostic)

### 📦 Amazon
| Tool | Credits |
|---|---|
| `amazon_product` (by ASIN) | 1 (US) / 5 (other countries) |
| `amazon_search` | 1 (US) / 5 (other) / +25 premium |
| `amazon_reviews` · `amazon_offers` · `amazon_autocomplete` | 5 / 5 / — |

### 🛒 Walmart, eBay, Flipkart, Myntra
`walmart_product` · `walmart_search` · `walmart_reviews` · `walmart_autocomplete` · `ebay_search` · `ebay_product` · `flipkart_search` · `flipkart_product` · `myntra_search` · `myntra_product`

> Walmart/eBay/Flipkart/Myntra tools take the **full target-site URL** (locale lives in the domain, e.g. `ebay.co.uk`).

### 💼 LinkedIn, Jobs & Local
| Tool | What it does | Credits |
|---|---|---|
| `linkedin_person_profile` | Full person profile by public ID | 50–100 |
| `linkedin_company_profile` | Company/school profile | 10 |
| `linkedin_post` | Post data | 5 |
| `linkedin_jobs_search` / `linkedin_job_overview` | Job search + job details | 5 |
| `yelp_scraper` / `indeed_scraper` / `zillow_scraper` | Local businesses, job listings, real estate | 4 / 1 / 2 |

### 📱 Social & Video
`x_profile` · `x_post` · `tiktok_profile` · `tiktok_post` · `tiktok_ads` · `youtube_search` · `youtube_video` · `youtube_channel` · `youtube_comments` · `youtube_transcripts` (only 1 credit!)

### 🤖 AI & Utilities
| Tool | What it does | Credits |
|---|---|---|
| `chatgpt_scraper` | Send a prompt to ChatGPT, get the structured conversation back | 30 |
| `account` | Remaining credits, concurrency, plan info — costs nothing, check it anytime | 0 |
| `webhook` | Fire-and-forget async scraping: returns a `sid` immediately, result POSTs to your dashboard-configured webhook | — |

---

## 🏗️ How it works

```
specs/*.json  ──►  src/index.js  ──►  77 MCP tools over stdio
(source of truth)   (generic engine)
```

- **`specs/*.json`** — nine category files holding every endpoint: exact URL, method, credit cost, every documented parameter (name/type/required/default/description), quirks, and response structure. Extracted from the full official docs on 2026-07-10.
- **`src/index.js`** — a single ~200-line engine. At startup it loads the specs and registers one tool per endpoint. **To fix or extend an API, edit a JSON file — zero code changes.**
- **`docs/*.md`** — distilled human-readable API reference per category (param tables, response shapes, gotchas).
- **`test/smoke.js`** — end-to-end test: boots the server over stdio as a real MCP client and makes live API calls.

### Built-in robustness

| Concern | Handling |
|---|---|
| **Your API key** | Injected server-side from `SCRAPINGDOG_API_KEY` — never exposed as a tool parameter, never visible to the model |
| Rate limits / transient errors | Automatic retry (3 attempts, backoff) on 429 / 5xx |
| Slow scrapes | 90s client timeout (ScrapingDog's own limit is 60s) |
| Giant HTML pages | Capped at 400k chars with a clear truncation notice |
| Screenshots | Returned as real MCP image content (base64 png/jpg/webp), not text |
| API errors | Mapped to actionable hints (401 → bad key, 403 → plan doesn't include this API, 410 → target returned nothing, …) |

---

## 🧪 Verify your install

```bash
SCRAPINGDOG_API_KEY=YOUR_KEY npm test          # macOS/Linux
$env:SCRAPINGDOG_API_KEY="YOUR_KEY"; npm test  # Windows PowerShell
```

Boots the server, lists all 77 tools, and makes 4 live calls (`account`, `google_search`, `scrape`, `youtube_transcripts`). Uses ~7 credits.

---

## ⚠️ Deviations from the official docs

Found and fixed by testing against the live API (2026-07-10):

1. **YouTube endpoints** — the docs list `https://api.scrapingdog.com/youtube` for all five YouTube APIs. That URL returns the ScrapingDog web app, not data. The real endpoints (used by this server) are `/youtube/search`, `/youtube/video`, `/youtube/channel`, `/youtube/comments`, `/youtube/transcripts`.
2. **`youtube_video` parameter** — docs say `video_id`; the live API rejects it and requires `v`.
3. **Datacenter proxy mode** (`http://scrapingdog:APIKEY@proxy.scrapingdog.com:8081`) is a passthrough HTTP proxy, not a REST endpoint — documented in [docs/engines-utilities.md](docs/engines-utilities.md) but intentionally not exposed as a tool.
4. Where the docs omit a credit cost or response sample (a handful of endpoints), the tool description says so explicitly rather than guessing.

---

## 💳 Credits, limits & error codes

- **Free plan**: 200 credits/month. Every tool description states its cost; `account` is free and shows your balance.
- **Cost-saving tip**: `scrape` defaults to JS rendering (5 credits). Pass `dynamic: false` for static pages → 1 credit.
- Requests time out server-side after **60 seconds**; concurrency is limited per plan tier (excess → 429, which this server retries automatically).

| Code | Meaning |
|---|---|
| 200 | Success |
| 202 | Accepted, queued |
| 400 | Bad request — check parameters |
| 401 | Invalid or missing API key |
| 403 | Your plan doesn't include this API |
| 404 | Resource not found |
| 410 | Target URL returned no content |
| 429 | Rate limit — auto-retried |

---

## 🔧 Troubleshooting

| Symptom | Fix |
|---|---|
| Server exits immediately: `SCRAPINGDOG_API_KEY ... not set` | Pass the key via `-e` (Claude Code) or the `env` block (JSON configs) |
| `401 Unauthorized` on every call | Key is wrong/expired — check your [dashboard](https://api.scrapingdog.com/dashboard) |
| `403 Forbidden` on one specific tool | That API isn't in your plan tier |
| `npx` install is slow on first launch | It's cloning + installing once; subsequent launches use the cache. Or use the local-clone install |
| Tool returns truncated HTML | Expected on huge pages (400k-char cap) — scrape a more specific URL or use a structured tool instead of raw `scrape` |
| Windows: server won't start from a path with spaces | Quote the path in your config: `"args": ["C:\\path with spaces\\src\\index.js"]` |

---

## 🗺️ Repo map

```
├── src/index.js        # the server (spec-driven engine)
├── specs/              # machine-readable endpoint specs — the source of truth
│   ├── web-scraping.json
│   ├── google-search.json
│   ├── google-ai-maps-trends.json
│   ├── google-news-scholar-patents.json
│   ├── google-shopping-other.json
│   ├── engines-utilities.json
│   ├── amazon-walmart.json
│   ├── ecommerce.json
│   ├── profiles-jobs-local.json
│   └── social-youtube.json
├── docs/               # human-readable API reference per category
├── test/smoke.js       # live end-to-end test
└── .env.example
```

## 🤝 Contributing

ScrapingDog ships new APIs regularly. Adding one is trivial:

1. Add an endpoint object to the matching `specs/*.json` (or a new spec file) — `tool_name`, `endpoint`, `params`, `credits`, `description`, `response_summary`.
2. Restart the server. The tool exists.
3. Run `npm test`, open a PR.

## 📄 License

[MIT](LICENSE) © Alessandro Benigni

*Not affiliated with ScrapingDog — an independent MCP integration. ScrapingDog is a product of its respective owners. Use responsibly and in accordance with the terms of service of the sites you scrape.*