Skip to main content
Glama
ScrapingBot MCP Server

ScrapingBot MCP Server

by sostag

ScrapingBot MCP server

Let Claude, Cursor and any Model Context Protocol client scrape the web with ScrapingBot: page HTML, products, real estate listings, hotels and rentals, search engine results and public social media pages.

You need a ScrapingBot account: the free plan gives 500 credits a month. Your username and API key are in your dashboard.

Installation

The server runs with npx from npm (@sostag/scraping-bot-mcp): nothing to install besides Node.js 18 or later.

Claude Desktop: one-click install

Download scraping-bot.mcpb and open it: Claude Desktop installs the extension and asks for your username and API key (stored securely). Nothing else to install.

Claude Code

claude mcp add scraping-bot \
  -e SCRAPING_BOT_USERNAME=your_username \
  -e SCRAPING_BOT_API_KEY=your_api_key \
  -- npx -y @sostag/scraping-bot-mcp

Claude Desktop (manual configuration)

Or add the server to claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\), then restart Claude Desktop:

{
  "mcpServers": {
    "scraping-bot": {
      "command": "npx",
      "args": ["-y", "@sostag/scraping-bot-mcp"],
      "env": {
        "SCRAPING_BOT_USERNAME": "your_username",
        "SCRAPING_BOT_API_KEY": "your_api_key"
      }
    }
  }
}

claude.ai and other clients with "Sign in" (OAuth)

Add a custom connector with the URL https://mcp.scraping-bot.io/mcp: the client opens ScrapingBot, you sign in and allow it, and it scrapes with your credits without ever seeing your API key. Disconnect it at any time in your account settings ("Connected applications").

Remote server (Cursor and other clients that connect to a URL)

Nothing to install: point the client to https://mcp.scraping-bot.io/mcp with your credentials in two headers. Cursor (~/.cursor/mcp.json):

{
  "mcpServers": {
    "scraping-bot": {
      "url": "https://mcp.scraping-bot.io/mcp",
      "headers": {
        "X-ScrapingBot-Username": "your_username",
        "X-ScrapingBot-Api-Key": "your_api_key"
      }
    }
  }
}

Basic Auth (Authorization: Basic base64(username:apiKey)) works too. The server keeps nothing between requests and never logs your key.

Cursor and other clients (local)

Use the same command, args and env in the client's MCP settings (Cursor: ~/.cursor/mcp.json).

To pin a version, add it: @sostag/scraping-bot-mcp@1.2.0. The versions are listed on npm and in the releases. The server also runs straight from this repository, built on install (needs Git): npx -y github:sostag/scraping-bot-mcp.

Related MCP server: Spicrawl MCP Server

Tools

Tool

What it returns

Credits

scrape_raw_html

HTML of any page (truncated to maxChars, 100,000 by default)

1, 5 with useChrome, 10 with premiumProxy, 25 with both

scrape_retail

Product: title, price, currency, images, brand, stock, EAN…

same

scrape_real_estate

Listing: price or rent, surface, rooms, location, images…

same

scrape_booking

Hotel or rental page, or a search results page (Booking.com, Airbnb, Expedia, Hotels.com, Abritel)

same

scrape_grocery

Grocery product: price, image, brand, nutritional values

same

scrape_search_engine

Google or Bing results as JSON

30

scrape_social_media

Public Instagram, TikTok, Facebook, LinkedIn, X/Twitter and Threads profiles, posts and searches

30

get_social_media_result

The result of a social media scrape that was still pending

0

get_account

Your plan, credits used and remaining, when they reset

0

get_usage_history

Credits spent per day, requests per API, success rate and latest requests (up to 90 days)

0

Page tools take url plus the optional useChrome (JavaScript sites), premiumProxy (sites that block regular proxies) and proxyCountry (two-letter code, with premiumProxy). Some sites always get the right options from the API.

A request blocked by the site (403, captcha…) is retried once with a premium proxy, billed at the premium price only if it succeeds; the result then says so (notice, or a note above the HTML). Pass premiumFallback: false to turn this off.

Failed requests are refunded. Social media scrapes take from a few seconds to several minutes: scrape_social_media waits about a minute, then returns a responseId that get_social_media_result uses later, at no cost.

When a URL is malformed (missing https://, missing / after the domain…), the error explains it and gives the corrected address.

Security

Your API key stays on your computer: it is read from the environment and only sent to https://api.scraping-bot.io. Every call made by your AI agent uses your credits; you can follow them in your dashboard.

Development

npm install
npm test                 # builds, then tests every tool against a fake API (no credit used)
SCRAPING_BOT_USERNAME=... SCRAPING_BOT_API_KEY=... npm run test:live   # real API, about 65 credits

src/ is TypeScript, compiled to dist/ (not committed): the npm package ships it, and npx github:... builds it on install through the prepare script.

The remote mode is src/http.ts (Express, Streamable HTTP without sessions: credentials on every request, one API client and MCP server per request, per-IP rate limit, 64 KB bodies, no credentials in the logs), started by src/http-server.ts (npm run start:http, port 8080). For local development, npm run dev:http runs it on port 8090 against the local API (4000) and website (3000), reloading on changes; copy .env.example to .env for OAuth. The Dockerfile builds it; each release pushes the image to the DigitalOcean registry and the App Platform app (.do/app.yaml, mcp.scraping-bot.io) redeploys it.

OAuth (src/oauth.ts, on when OAUTH_SECRET is set): this server is the authorization server (the SDK's routes: metadata, /register, /authorize, /token, /revoke, PKCE) but stores nothing. Clients, codes and grants live in the ScrapingBot API (internal routes protected by OAUTH_SECRET); /authorize sends the user to the website's consent page (/oauth/consent) with a signed request. Access tokens (sbo_at_, 1 hour) are accepted by the API on /scrape and /account; refresh tokens (sbo_rt_, 90 days) rotate.

The repository is public: secretlint checks the staged files before each commit (scripts/secretlint-staged.mjs, installed by npx simple-git-hooks once after cloning) and the whole repository in the release workflow (npm run secretlint). Secrets go in environment variables, never in the code.

Releases are automatic: every push to main that changes the code builds, tests and publishes the npm package, a vX.Y.Z tag and a GitHub release with the Claude Desktop extension scraping-bot.mcpb (.github/workflows/release.yml; npm Trusted Publishing, no npm token). manifest.json describes the extension (its version comes from package.json); node scripts/build-mcpb.mjs out.mcpb builds it locally after npm run build. The commit messages choose the version: feat: raises the minor version, fix: and the others the patch version, ! or BREAKING CHANGE the major version. Nothing is committed to main (its package.json keeps 0.0.0-dev), so there is nothing to pull. Add [skip release] to a commit message to push without releasing. SCRAPING_BOT_API_URL points the server to another API (local development).

License

MIT

Available Tools

10 tools
get_accountGet my ScrapingBot creditsA
Read-only

Get the ScrapingBot account of the API key: plan, credits used and remaining this period, when they reset, and the upgrade page. Check it before a series of scrapes, and tell the user when few credits are left (under about 10%) or when a scrape fails for lack of credits (HTTP 402): the plans are at https://www.scraping-bot.io/pricing. Costs no credit.

Inputs: no named parameters.

get_social_media_resultGet a pending social media resultA
Read-only

Get the result of a scrape_social_media request that was still pending, from its scraper and responseId. Waits about a minute. Costs no credit.

Inputs: scraper (required), responseId (required).

get_usage_historyGet my ScrapingBot usage historyA
Read-only

Get the usage of the ScrapingBot account over the last days: credits spent per day, requests per API, success rate, and the latest requests (URL, API, success, credits, error). Useful to explain what used the credits or why requests fail. Costs no credit.

Inputs: days, recent.

scrape_bookingScrape a travel pageA
Read-only

Extract a hotel or rental page (title, price, images, rating, rooms) or a search results page from Booking.com, Airbnb, Expedia, Hotels.com or Abritel. Put the stay dates in the URL. Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).

Inputs: url (required), useChrome, premiumProxy, proxyCountry, +1 more.

scrape_groceryScrape a grocery productA
Read-only

Extract a grocery product page (title, price, image, brand, nutritional values). Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).

Inputs: url (required), useChrome, premiumProxy, proxyCountry, +1 more.

scrape_raw_htmlGet the HTML of a pageA
Read-only

Fetch the HTML source of any public web page through ScrapingBot's proxies. Use useChrome for pages built with JavaScript. Prefer the dedicated tools: scrape_retail for product pages, scrape_real_estate, scrape_booking, scrape_search_engine for Google or Bing, and scrape_social_media for Instagram, TikTok, Facebook, LinkedIn, X/Twitter or Threads (blocked here). Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).

Inputs: url (required), maxChars, useChrome, premiumProxy, +3 more.

scrape_real_estateScrape a real estate listingA
Read-only

Extract a property listing (price or rent, surface, rooms, bedrooms, location, images...) from real estate sites such as SeLoger, Leboncoin, Rightmove or Zillow. Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).

Inputs: url (required), useChrome, premiumProxy, proxyCountry, +1 more.

scrape_retailScrape a product pageA
Read-only

Extract structured product data (title, price, currency, images, brand, stock, EAN...) from an e-commerce product page such as Amazon, Cdiscount, Fnac or Zalando. Prefer it to scrape_raw_html for any product page. Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).

Inputs: url (required), useChrome, premiumProxy, proxyCountry, +1 more.

scrape_search_engineSearch Google or BingA
Read-only

Get search engine results (organic results, related searches, people also ask...) as JSON. Use it instead of fetching google.com or bing.com pages with scrape_raw_html. Costs 30 credits; failed requests are refunded.

Inputs: search (required), page, engine, resultLang, +2 more.

scrape_social_mediaScrape a social media pageA
Read-only

Scrape a public social media profile, post or search (Instagram, TikTok, Facebook, LinkedIn, X/Twitter, Threads). Always use this tool for these networks, never scrape_raw_html. Choose the scraper and give its inputs: instagramProfile (account (name or profile URL); returns the 12 latest posts), instagramPost (url), tiktokProfile (url), tiktokHashtag (hashtag; max_video_count), facebookProfile (url (page or personal profile, without its posts)), facebookPost (url), facebookOrganization (url), facebookMarketplaceItem (url; country), facebookMarketplaceSearch (keyword, city, country; items_limit), linkedinProfile (url; language), linkedinCompanyProfile (url; language), linkedinPost (url), linkedinSearchResult (keywords; page, language), twitterProfile (url; number_of_tweets), twitterSearch (search; posts_max_count), threadsProfile (url), threadsPost (url). Waits for the result for about a minute, otherwise returns a responseId for get_social_media_result: use that tool to wait longer, as calling this one again is billed again. Costs 30 credits; failed requests are refunded.

Inputs: scraper (required), url, city, page, +11 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.4/5.0

    Scored across 10 tools

    Disambiguation4/5

    Most tools target clearly distinct verticals, and descriptions explicitly route users (use scrape_social_media for social networks, scrape_search_engine instead of scrape_raw_html for Google/Bing). The main overlaps are scrape_retail vs scrape_grocery (both extract product title/price/brand from e-commerce pages) and scrape_booking vs scrape_real_estate (Airbnb/Abritel rentals vs property listings), which could cause occasional misselection on borderline pages.

    Naming Consistency5/5

    All ten tools follow a strict snake_case verb_noun pattern (get_account, get_usage_history, get_social_media_result, scrape_*), with scraping tools grouped under a single scrape_ prefix by vertical. No mixed conventions or camelCase.

    Tool Count5/5

    Ten tools is well within the ideal range, and each earns its place: two zero-credit account/diagnostic tools, one async result poller, and seven scrapers covering distinct content verticals plus a raw fallback.

    Completeness4/5

    The surface covers account/credit inspection, usage diagnostics, retry of pending jobs, and the major scraping verticals plus a raw HTML fallback, which is solid end-to-end coverage for a scraping API. The gap is that async pending-result retrieval (get_social_media_result) exists only for social media, so long-running or expensive retail/real-estate/booking jobs have no documented way to be picked up later.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables agents to scrape, crawl, map, search, and extract web pages as clean markdown or structured JSON directly through MCP tools.
      AGPL 3.0
    • A
      license
      A
      quality
      B
      maintenance
      Enables AI agents to scrape web pages into Markdown, text, HTML, JSON, PDFs, screenshots, and structured data, while managing batch scraping jobs, persistent browser sessions, usage/request history, and docs lookup through native MCP tools.
      27
      415 npm
      3
      Apache 2.0
    • A
      license
      A
      quality
      B
      maintenance
      Enables MCP-compatible AI agents to search, scrape, crawl, map, and interact with live web pages, returning clean, structured, agent-ready content. It also supports recurring page monitoring, autonomous deep research, and credit-usage checks.
      26
      1
      MIT
    • F
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.
      -