ScrapingBot MCP Server
You can use this MCP server to scrape public web data through ScrapingBot and monitor the ScrapingBot account.
Scrape raw HTML from any public page, with Chrome rendering, premium proxy, proxy country, max-character, and premium-fallback options.
Extract structured product data from e-commerce pages, including title, price, currency, images, brand, stock, and EAN.
Extract real-estate listing data such as price/rent, surface, rooms, location, and images.
Scrape hotel/rental pages and travel search results from Booking.com, Airbnb, Expedia, Hotels.com, and Abritel.
Extract grocery product data including price, image, brand, and nutritional values.
Get Google or Bing search results as JSON, including Google maps/trends options.
Scrape public social media profiles, posts, and searches on Instagram, TikTok, Facebook, LinkedIn, X/Twitter, and Threads.
Retrieve pending social media scrape results without additional credit.
Check account plan, credits used/remaining, reset date, and usage history.
Page scrapes cost credits based on options; search and social scrapes cost 30 credits; some account/result checks are free, and failed requests are refunded.
Enables scraping Airbnb rental listings and search results, including price, surface, rooms, location, and images, through ScrapingBot's booking scraper.
Enables scraping Expedia hotel or rental pages and search results for listing details such as price, location, and images.
Enables scraping public Facebook profiles, posts, and searches via ScrapingBot's social media tools.
Enables scraping Google search engine results as JSON.
Enables scraping Hotels.com hotel or rental pages and search results for listing details such as price, location, and images.
Enables scraping public Instagram profiles, posts, and searches via ScrapingBot's social media tools.
Enables scraping public Threads profiles, posts, and searches via ScrapingBot's social media tools.
Enables scraping public TikTok profiles, posts, and searches via ScrapingBot's social media tools.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@ScrapingBot MCP Server scrape the title and price from https://www.amazon.com/dp/B08N5WRWNW"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
ScrapingBot MCP server
Let Claude, Cursor and any Model Context Protocol client scrape the web with ScrapingBot: page HTML, products, real estate listings, hotels and rentals, search engine results and public social media pages.
You need a ScrapingBot account: the free plan gives 500 credits a month. Your username and API key are in your dashboard.
Installation
The server runs with npx from npm (@sostag/scraping-bot-mcp): nothing to install besides Node.js 18 or later.
Claude Desktop: one-click install
Download scraping-bot.mcpb and open it: Claude Desktop installs the extension and asks for your username and API key (stored securely). Nothing else to install.
Claude Code
claude mcp add scraping-bot \
-e SCRAPING_BOT_USERNAME=your_username \
-e SCRAPING_BOT_API_KEY=your_api_key \
-- npx -y @sostag/scraping-bot-mcpClaude Desktop (manual configuration)
Or add the server to claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\), then restart Claude Desktop:
{
"mcpServers": {
"scraping-bot": {
"command": "npx",
"args": ["-y", "@sostag/scraping-bot-mcp"],
"env": {
"SCRAPING_BOT_USERNAME": "your_username",
"SCRAPING_BOT_API_KEY": "your_api_key"
}
}
}
}claude.ai and other clients with "Sign in" (OAuth)
Add a custom connector with the URL https://mcp.scraping-bot.io/mcp: the client opens ScrapingBot, you sign in and allow it, and it scrapes with your credits without ever seeing your API key. Disconnect it at any time in your account settings ("Connected applications").
Remote server (Cursor and other clients that connect to a URL)
Nothing to install: point the client to https://mcp.scraping-bot.io/mcp with your credentials in two headers. Cursor (~/.cursor/mcp.json):
{
"mcpServers": {
"scraping-bot": {
"url": "https://mcp.scraping-bot.io/mcp",
"headers": {
"X-ScrapingBot-Username": "your_username",
"X-ScrapingBot-Api-Key": "your_api_key"
}
}
}
}Basic Auth (Authorization: Basic base64(username:apiKey)) works too. The server keeps nothing between requests and never logs your key.
Cursor and other clients (local)
Use the same command, args and env in the client's MCP settings (Cursor: ~/.cursor/mcp.json).
To pin a version, add it: @sostag/scraping-bot-mcp@1.2.0. The versions are listed on npm and in the releases. The server also runs straight from this repository, built on install (needs Git): npx -y github:sostag/scraping-bot-mcp.
Related MCP server: Spicrawl MCP Server
Tools
Tool | What it returns | Credits |
| HTML of any page (truncated to | 1, 5 with |
| Product: title, price, currency, images, brand, stock, EAN… | same |
| Listing: price or rent, surface, rooms, location, images… | same |
| Hotel or rental page, or a search results page (Booking.com, Airbnb, Expedia, Hotels.com, Abritel) | same |
| Grocery product: price, image, brand, nutritional values | same |
| Google or Bing results as JSON | 30 |
| Public Instagram, TikTok, Facebook, LinkedIn, X/Twitter and Threads profiles, posts and searches | 30 |
| The result of a social media scrape that was still pending | 0 |
| Your plan, credits used and remaining, when they reset | 0 |
| Credits spent per day, requests per API, success rate and latest requests (up to 90 days) | 0 |
Page tools take url plus the optional useChrome (JavaScript sites), premiumProxy (sites that block regular proxies) and proxyCountry (two-letter code, with premiumProxy). Some sites always get the right options from the API.
A request blocked by the site (403, captcha…) is retried once with a premium proxy, billed at the premium price only if it succeeds; the result then says so (notice, or a note above the HTML). Pass premiumFallback: false to turn this off.
Failed requests are refunded. Social media scrapes take from a few seconds to several minutes: scrape_social_media waits about a minute, then returns a responseId that get_social_media_result uses later, at no cost.
When a URL is malformed (missing https://, missing / after the domain…), the error explains it and gives the corrected address.
Security
Your API key stays on your computer: it is read from the environment and only sent to https://api.scraping-bot.io. Every call made by your AI agent uses your credits; you can follow them in your dashboard.
Development
npm install
npm test # builds, then tests every tool against a fake API (no credit used)
SCRAPING_BOT_USERNAME=... SCRAPING_BOT_API_KEY=... npm run test:live # real API, about 65 creditssrc/ is TypeScript, compiled to dist/ (not committed): the npm package ships it, and npx github:... builds it on install through the prepare script.
The remote mode is src/http.ts (Express, Streamable HTTP without sessions: credentials on every request, one API client and MCP server per request, per-IP rate limit, 64 KB bodies, no credentials in the logs), started by src/http-server.ts (npm run start:http, port 8080). For local development, npm run dev:http runs it on port 8090 against the local API (4000) and website (3000), reloading on changes; copy .env.example to .env for OAuth. The Dockerfile builds it; each release pushes the image to the DigitalOcean registry and the App Platform app (.do/app.yaml, mcp.scraping-bot.io) redeploys it.
OAuth (src/oauth.ts, on when OAUTH_SECRET is set): this server is the authorization server (the SDK's routes: metadata, /register, /authorize, /token, /revoke, PKCE) but stores nothing. Clients, codes and grants live in the ScrapingBot API (internal routes protected by OAUTH_SECRET); /authorize sends the user to the website's consent page (/oauth/consent) with a signed request. Access tokens (sbo_at_, 1 hour) are accepted by the API on /scrape and /account; refresh tokens (sbo_rt_, 90 days) rotate.
The repository is public: secretlint checks the staged files before each commit (scripts/secretlint-staged.mjs, installed by npx simple-git-hooks once after cloning) and the whole repository in the release workflow (npm run secretlint). Secrets go in environment variables, never in the code.
Releases are automatic: every push to main that changes the code builds, tests and publishes the npm package, a vX.Y.Z tag and a GitHub release with the Claude Desktop extension scraping-bot.mcpb (.github/workflows/release.yml; npm Trusted Publishing, no npm token). manifest.json describes the extension (its version comes from package.json); node scripts/build-mcpb.mjs out.mcpb builds it locally after npm run build. The commit messages choose the version: feat: raises the minor version, fix: and the others the patch version, ! or BREAKING CHANGE the major version. Nothing is committed to main (its package.json keeps 0.0.0-dev), so there is nothing to pull. Add [skip release] to a commit message to push without releasing. SCRAPING_BOT_API_URL points the server to another API (local development).
License
MIT
Available Tools
10 toolsget_accountGet my ScrapingBot creditsARead-only
Get the ScrapingBot account of the API key: plan, credits used and remaining this period, when they reset, and the upgrade page. Check it before a series of scrapes, and tell the user when few credits are left (under about 10%) or when a scrape fails for lack of credits (HTTP 402): the plans are at https://www.scraping-bot.io/pricing. Costs no credit.
Inputs: no named parameters.
get_social_media_resultGet a pending social media resultARead-only
Get the result of a scrape_social_media request that was still pending, from its scraper and responseId. Waits about a minute. Costs no credit.
Inputs: scraper (required), responseId (required).
get_usage_historyGet my ScrapingBot usage historyARead-only
Get the usage of the ScrapingBot account over the last days: credits spent per day, requests per API, success rate, and the latest requests (URL, API, success, credits, error). Useful to explain what used the credits or why requests fail. Costs no credit.
Inputs: days, recent.
scrape_bookingScrape a travel pageARead-only
Extract a hotel or rental page (title, price, images, rating, rooms) or a search results page from Booking.com, Airbnb, Expedia, Hotels.com or Abritel. Put the stay dates in the URL. Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
Inputs: url (required), useChrome, premiumProxy, proxyCountry, +1 more.
scrape_groceryScrape a grocery productARead-only
Extract a grocery product page (title, price, image, brand, nutritional values). Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
Inputs: url (required), useChrome, premiumProxy, proxyCountry, +1 more.
scrape_raw_htmlGet the HTML of a pageARead-only
Fetch the HTML source of any public web page through ScrapingBot's proxies. Use useChrome for pages built with JavaScript. Prefer the dedicated tools: scrape_retail for product pages, scrape_real_estate, scrape_booking, scrape_search_engine for Google or Bing, and scrape_social_media for Instagram, TikTok, Facebook, LinkedIn, X/Twitter or Threads (blocked here). Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
Inputs: url (required), maxChars, useChrome, premiumProxy, +3 more.
scrape_real_estateScrape a real estate listingARead-only
Extract a property listing (price or rent, surface, rooms, bedrooms, location, images...) from real estate sites such as SeLoger, Leboncoin, Rightmove or Zillow. Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
Inputs: url (required), useChrome, premiumProxy, proxyCountry, +1 more.
scrape_retailScrape a product pageARead-only
Extract structured product data (title, price, currency, images, brand, stock, EAN...) from an e-commerce product page such as Amazon, Cdiscount, Fnac or Zalando. Prefer it to scrape_raw_html for any product page. Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
Inputs: url (required), useChrome, premiumProxy, proxyCountry, +1 more.
scrape_search_engineSearch Google or BingARead-only
Get search engine results (organic results, related searches, people also ask...) as JSON. Use it instead of fetching google.com or bing.com pages with scrape_raw_html. Costs 30 credits; failed requests are refunded.
Inputs: search (required), page, engine, resultLang, +2 more.
scrape_social_mediaScrape a social media pageARead-only
Scrape a public social media profile, post or search (Instagram, TikTok, Facebook, LinkedIn, X/Twitter, Threads). Always use this tool for these networks, never scrape_raw_html. Choose the scraper and give its inputs: instagramProfile (account (name or profile URL); returns the 12 latest posts), instagramPost (url), tiktokProfile (url), tiktokHashtag (hashtag; max_video_count), facebookProfile (url (page or personal profile, without its posts)), facebookPost (url), facebookOrganization (url), facebookMarketplaceItem (url; country), facebookMarketplaceSearch (keyword, city, country; items_limit), linkedinProfile (url; language), linkedinCompanyProfile (url; language), linkedinPost (url), linkedinSearchResult (keywords; page, language), twitterProfile (url; number_of_tweets), twitterSearch (search; posts_max_count), threadsProfile (url), threadsPost (url). Waits for the result for about a minute, otherwise returns a responseId for get_social_media_result: use that tool to wait longer, as calling this one again is billed again. Costs 30 credits; failed requests are refunded.
Inputs: scraper (required), url, city, page, +11 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 10 tools
Most tools target clearly distinct verticals, and descriptions explicitly route users (use scrape_social_media for social networks, scrape_search_engine instead of scrape_raw_html for Google/Bing). The main overlaps are scrape_retail vs scrape_grocery (both extract product title/price/brand from e-commerce pages) and scrape_booking vs scrape_real_estate (Airbnb/Abritel rentals vs property listings), which could cause occasional misselection on borderline pages.
All ten tools follow a strict snake_case verb_noun pattern (get_account, get_usage_history, get_social_media_result, scrape_*), with scraping tools grouped under a single scrape_ prefix by vertical. No mixed conventions or camelCase.
Ten tools is well within the ideal range, and each earns its place: two zero-credit account/diagnostic tools, one async result poller, and seven scrapers covering distinct content verticals plus a raw fallback.
The surface covers account/credit inspection, usage diagnostics, retry of pending jobs, and the major scraping verticals plus a raw HTML fallback, which is solid end-to-end coverage for a scraping API. The gap is that async pending-result retrieval (get_social_media_result) exists only for social media, so long-running or expensive retail/real-estate/booking jobs have no documented way to be picked up later.
Maintenance
Related MCP Connectors
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
Unblocking and fresh web data for agents: URL to Markdown, YouTube, Maps, Amazon, jobs. Pay per call
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables agents to scrape, crawl, map, search, and extract web pages as clean markdown or structured JSON directly through MCP tools.AGPL 3.0
- AlicenseAqualityBmaintenanceEnables AI agents to scrape web pages into Markdown, text, HTML, JSON, PDFs, screenshots, and structured data, while managing batch scraping jobs, persistent browser sessions, usage/request history, and docs lookup through native MCP tools.27415 npm3Apache 2.0
- AlicenseAqualityBmaintenanceEnables MCP-compatible AI agents to search, scrape, crawl, map, and interact with live web pages, returning clean, structured, agent-ready content. It also supports recurring page monitoring, autonomous deep research, and credit-usage checks.261MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.-