ScrapingBot MCP Server
You can use this MCP server to scrape public web data through ScrapingBot and monitor the ScrapingBot account.
Scrape raw HTML from any public page, with Chrome rendering, premium proxy, proxy country, max-character, and premium-fallback options.
Extract structured product data from e-commerce pages, including title, price, currency, images, brand, stock, and EAN.
Extract real-estate listing data such as price/rent, surface, rooms, location, and images.
Scrape hotel/rental pages and travel search results from Booking.com, Airbnb, Expedia, Hotels.com, and Abritel.
Extract grocery product data including price, image, brand, and nutritional values.
Get Google or Bing search results as JSON, including Google maps/trends options.
Scrape public social media profiles, posts, and searches on Instagram, TikTok, Facebook, LinkedIn, X/Twitter, and Threads.
Retrieve pending social media scrape results without additional credit.
Check account plan, credits used/remaining, reset date, and usage history.
Page scrapes cost credits based on options; search and social scrapes cost 30 credits; some account/result checks are free, and failed requests are refunded.
Enables scraping Airbnb rental listings and search results, including price, surface, rooms, location, and images, through ScrapingBot's booking scraper.
Enables scraping Expedia hotel or rental pages and search results for listing details such as price, location, and images.
Enables scraping public Facebook profiles, posts, and searches via ScrapingBot's social media tools.
Enables scraping Google search engine results as JSON.
Enables scraping Hotels.com hotel or rental pages and search results for listing details such as price, location, and images.
Enables scraping public Instagram profiles, posts, and searches via ScrapingBot's social media tools.
Enables scraping public Threads profiles, posts, and searches via ScrapingBot's social media tools.
Enables scraping public TikTok profiles, posts, and searches via ScrapingBot's social media tools.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@ScrapingBot MCP Serverscrape the title and price from https://www.amazon.com/dp/B08N5WRWNW"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
ScrapingBot MCP server
Let Claude, Cursor and any Model Context Protocol client scrape the web with ScrapingBot: page HTML, products, real estate listings, hotels and rentals, search engine results and public social media pages.
You need a ScrapingBot account: the free plan gives 500 credits a month. Your username and API key are in your dashboard.
Installation
The server runs with npx from npm (@sostag/scraping-bot-mcp): nothing to install besides Node.js 18 or later.
Claude Desktop: one-click install
Download scraping-bot.mcpb and open it: Claude Desktop installs the extension and asks for your username and API key (stored securely). Nothing else to install.
Claude Code
claude mcp add scraping-bot \
-e SCRAPING_BOT_USERNAME=your_username \
-e SCRAPING_BOT_API_KEY=your_api_key \
-- npx -y @sostag/scraping-bot-mcpClaude Desktop (manual configuration)
Or add the server to claude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\), then restart Claude Desktop:
{
"mcpServers": {
"scraping-bot": {
"command": "npx",
"args": ["-y", "@sostag/scraping-bot-mcp"],
"env": {
"SCRAPING_BOT_USERNAME": "your_username",
"SCRAPING_BOT_API_KEY": "your_api_key"
}
}
}
}claude.ai and other clients with "Sign in" (OAuth)
Add a custom connector with the URL https://mcp.scraping-bot.io/mcp: the client opens ScrapingBot, you sign in and allow it, and it scrapes with your credits without ever seeing your API key. Disconnect it at any time in your account settings ("Connected applications").
Remote server (Cursor and other clients that connect to a URL)
Nothing to install: point the client to https://mcp.scraping-bot.io/mcp with your credentials in two headers. Cursor (~/.cursor/mcp.json):
{
"mcpServers": {
"scraping-bot": {
"url": "https://mcp.scraping-bot.io/mcp",
"headers": {
"X-ScrapingBot-Username": "your_username",
"X-ScrapingBot-Api-Key": "your_api_key"
}
}
}
}Basic Auth (Authorization: Basic base64(username:apiKey)) works too. The server keeps nothing between requests and never logs your key.
Cursor and other clients (local)
Use the same command, args and env in the client's MCP settings (Cursor: ~/.cursor/mcp.json).
To pin a version, add it: @sostag/scraping-bot-mcp@1.2.0. The versions are listed on npm and in the releases. The server also runs straight from this repository, built on install (needs Git): npx -y github:sostag/scraping-bot-mcp.
Related MCP server: Spicrawl MCP Server
Tools
Tool | What it returns | Credits |
| HTML of any page (truncated to | 1, 5 with |
| Product: title, price, currency, images, brand, stock, EAN… | same |
| Listing: price or rent, surface, rooms, location, images… | same |
| Hotel or rental page, or a search results page (Booking.com, Airbnb, Expedia, Hotels.com, Abritel) | same |
| Grocery product: price, image, brand, nutritional values | same |
| Google or Bing results as JSON | 30 |
| Public Instagram, TikTok, Facebook, LinkedIn, X/Twitter and Threads profiles, posts and searches | 30 |
| The result of a social media scrape that was still pending | 0 |
| Your plan, credits used and remaining, when they reset | 0 |
| Credits spent per day, requests per API, success rate and latest requests (up to 90 days) | 0 |
Page tools take url plus the optional useChrome (JavaScript sites), premiumProxy (sites that block regular proxies) and proxyCountry (two-letter code, with premiumProxy). Some sites always get the right options from the API.
A request blocked by the site (403, captcha…) is retried once with a premium proxy, billed at the premium price only if it succeeds; the result then says so (notice, or a note above the HTML). Pass premiumFallback: false to turn this off.
Failed requests are refunded. Social media scrapes take from a few seconds to several minutes: scrape_social_media waits about a minute, then returns a responseId that get_social_media_result uses later, at no cost.
When a URL is malformed (missing https://, missing / after the domain…), the error explains it and gives the corrected address.
Security
Your API key stays on your computer: it is read from the environment and only sent to https://api.scraping-bot.io. Every call made by your AI agent uses your credits; you can follow them in your dashboard.
Development
npm install
npm test # builds, then tests every tool against a fake API (no credit used)
SCRAPING_BOT_USERNAME=... SCRAPING_BOT_API_KEY=... npm run test:live # real API, about 65 creditssrc/ is TypeScript, compiled to dist/ (not committed): the npm package ships it, and npx github:... builds it on install through the prepare script.
The remote mode is src/http.ts (Express, Streamable HTTP without sessions: credentials on every request, one API client and MCP server per request, per-IP rate limit, 64 KB bodies, no credentials in the logs), started by src/http-server.ts (npm run start:http, port 8080). For local development, npm run dev:http runs it on port 8090 against the local API (4000) and website (3000), reloading on changes; copy .env.example to .env for OAuth. The Dockerfile builds it; each release pushes the image to the DigitalOcean registry and the App Platform app (.do/app.yaml, mcp.scraping-bot.io) redeploys it.
OAuth (src/oauth.ts, on when OAUTH_SECRET is set): this server is the authorization server (the SDK's routes: metadata, /register, /authorize, /token, /revoke, PKCE) but stores nothing. Clients, codes and grants live in the ScrapingBot API (internal routes protected by OAUTH_SECRET); /authorize sends the user to the website's consent page (/oauth/consent) with a signed request. Access tokens (sbo_at_, 1 hour) are accepted by the API on /scrape and /account; refresh tokens (sbo_rt_, 90 days) rotate.
The repository is public: secretlint checks the staged files before each commit (scripts/secretlint-staged.mjs, installed by npx simple-git-hooks once after cloning) and the whole repository in the release workflow (npm run secretlint). Secrets go in environment variables, never in the code.
Releases are automatic: every push to main that changes the code builds, tests and publishes the npm package, a vX.Y.Z tag and a GitHub release with the Claude Desktop extension scraping-bot.mcpb (.github/workflows/release.yml; npm Trusted Publishing, no npm token). manifest.json describes the extension (its version comes from package.json); node scripts/build-mcpb.mjs out.mcpb builds it locally after npm run build. The commit messages choose the version: feat: raises the minor version, fix: and the others the patch version, ! or BREAKING CHANGE the major version. Nothing is committed to main (its package.json keeps 0.0.0-dev), so there is nothing to pull. Add [skip release] to a commit message to push without releasing. SCRAPING_BOT_API_URL points the server to another API (local development).
License
MIT
Available Tools
10 toolsget_accountGet my ScrapingBot creditsARead-only
Get the ScrapingBot account of the API key: plan, credits used and remaining this period, when they reset, and the upgrade page. Check it before a series of scrapes, and tell the user when few credits are left (under about 10%) or when a scrape fails for lack of credits (HTTP 402): the plans are at https://www.scraping-bot.io/pricing. Costs no credit.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
It goes beyond standard metadata by disclosing crucial operational details like 'Costs no credit,' the specific failure code (HTTP 402), and the link to upgrade plans, adding significant behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
While slightly verbose, every part of the paragraph is high-value operational instruction. The critical information (what it does and when to use it) is front-loaded effectively.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of an output schema, the description provides a complete narrative of the account data, including renewal cycles and failure modes, leaving no critical data gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Since there are zero parameters, the baseline score is 4. The description effectively explains the scope of the data return, which is the only thing an agent needs to know about the 'input' operation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly specifies the data retrieved (plan, used, remaining, reset time, and upgrade page) for the ScrapingBot account, distinguishing it from general data scraping siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides explicit guidance on when to use the tool—'Check it before a series of scrapes'—and provides a clear operational trigger for invocation: low credit alerts or HTTP 402 failure.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_social_media_resultGet a pending social media resultARead-only
Get the result of a scrape_social_media request that was still pending, from its scraper and responseId. Waits about a minute. Costs no credit.
| Name | Required | Description | Default |
|---|---|---|---|
| scraper | Yes | ||
| responseId | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only cover read-only and open-world status, but the description adds two traits they cannot express: it blocks for roughly a minute and consumes no credit. That latency and cost disclosure is genuinely useful for an agent deciding whether to call it inline, though the still-pending failure mode is left unstated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence, front-loaded with the action and resource, with the two behavioral facts appended compactly. No filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description could say more about what the returned result looks like or how to handle a still-pending response; however, for a simple 2-parameter retrieval companion to scrape_social_media it covers everything needed to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description has to carry the load; it names both parameters and tells the agent where responseId comes from ('from its scraper and responseId'), but does not state that scraper must match the scraper used in the original pending request or explain the enum values.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Get the result of a scrape_social_media request') and explicitly ties it to the sibling tool that produces the pending request, so an agent can tell it apart from the scrape_* family without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clearly conditions use on a prior scrape_social_media call that returned 'still pending', and names that sibling as the originating alternative. It stops short of saying what to do if the result is still not ready after the wait (retry vs. give up), which is the one remaining routing decision.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_usage_historyGet my ScrapingBot usage historyARead-only
Get the usage of the ScrapingBot account over the last days: credits spent per day, requests per API, success rate, and the latest requests (URL, API, success, credits, error). Useful to explain what used the credits or why requests fail. Costs no credit.
| Name | Required | Description | Default |
|---|---|---|---|
| days | No | Period in days, 1 to 90. Default 30. | |
| recent | No | Number of latest requests to list, 0 to 50. Default 10. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false, and the description adds genuinely new operational context: the call "costs no credit," which matters on a metered scraping service. It also discloses the shape of the diagnostic data (per-day credits, failure errors), though it says nothing about latency or data freshness.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three front-loaded sentences: what it returns, why to use it, and its cost. The enumerated metric list is long but every item is informative since there is no output schema. Minor redundancy between "latest requests" and the earlier metric list.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the burden of describing return values, and it does so concretely, down to the fields of each recent-request entry (URL, API, success, credits, error). Combined with the fully documented schema, an agent has everything needed to call and interpret it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% — both `days` (1–90, default 30) and `recent` (0–50, default 10) are fully documented with ranges and defaults. The description mentions "over the last days" and "latest requests" but adds no syntax or usage nuance beyond the schema, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource — retrieving ScrapingBot account usage over the last days — and enumerates the returned metrics (credits per day, requests per API, success rate, latest requests). This clearly separates it from the scrape_* siblings, which perform scraping rather than reporting account telemetry.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Useful to explain what used the credits or why requests fail" gives explicit diagnostic contexts for calling it. It does not name a sibling alternative (e.g. get_account), but the use case is unambiguous and no exclusions are needed for a read-only reporting tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_bookingScrape a travel pageARead-only
Extract a hotel or rental page (title, price, images, rating, rooms) or a search results page from Booking.com, Airbnb, Expedia, Hotels.com or Abritel. Put the stay dates in the URL. Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Full URL of the page, starting with https:// | |
| useChrome | No | Render the page with a headless Chrome (JavaScript sites). Default false. | |
| premiumProxy | No | Use a premium proxy for sites that block regular proxies. Default false. | |
| proxyCountry | No | Two-letter country code of the proxy (requires premiumProxy), e.g. FR, US. | |
| premiumFallback | No | When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/openWorld annotations, it discloses credit costs, cost modifiers for useChrome and premiumProxy, refund behavior, protected-site billing, and automatic premium-proxy retry semantics. These are material operational details an agent could not infer from the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose and supported sites are front-loaded, followed by URL date guidance, return format, and billing behavior. It is dense but every major clause is useful, though the long list of protected-site examples could be tightened.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a scraping tool with no output schema, the description explains the JSON return and some fields, plus billing and retry behavior. Search-result return shape and pagination are not covered, leaving a small completeness gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds meaning by mapping useChrome and premiumProxy to specific credit costs and explaining premiumFallback billing. It does not add much for proxyCountry, but that parameter is already documented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a precise verb ('Extract') and resource ('hotel or rental page... or a search results page') and names the exact travel sites covered. This clearly separates it from the retail, grocery, social, and real-estate siblings by domain.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides clear usage context by naming supported travel sites and instructing users to put stay dates in the URL. It does not explicitly compare against alternatives such as scrape_raw_html or scrape_real_estate, but the domain context is strong.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_groceryScrape a grocery productARead-only
Extract a grocery product page (title, price, image, brand, nutritional values). Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Full URL of the page, starting with https:// | |
| useChrome | No | Render the page with a headless Chrome (JavaScript sites). Default false. | |
| premiumProxy | No | Use a premium proxy for sites that block regular proxies. Default false. | |
| proxyCountry | No | Two-letter country code of the proxy (requires premiumProxy), e.g. FR, US. | |
| premiumFallback | No | When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
This description excels in behavioral transparency by disclosing crucial details beyond the annotations, including the tiered cost structure (1, 5, 10, 25 credits), specific site triggers for high cost, and the complex retry/refund logic. This information is essential for financial and operational planning.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely dense and lengthy due to the high volume of operational, financial, and complex behavioral information it must convey. While it is efficient in its density, it sacrifices conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The definition is highly complete for a complex, high-risk tool. It covers the target data, the costs, the failures (refund policy), the success criteria for retries, and specific geographical exceptions, providing the agent with everything it needs to execute the tool safely and effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Since schema coverage is 100%, the baseline is 3. However, the description adds significant semantic value by explaining the *consequences* and *intent* of parameters like 'premiumProxy' and 'proxyCountry'—that they are required for circumventing site blocks and involve specific cost tiers.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The tool is clearly defined as scraping grocery products, specifying the data extracted (title, price, image, brand, etc.). The name and title are highly specific and immediately distinguish it from the other scraping tools in the sibling list (e.g., real_estate, booking).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides extensive functional context regarding when and why certain features are necessary (e.g., using premium proxies for blocked sites, using Chrome for JavaScript rendering). While it doesn't explicitly say 'Use this over X,' the functional constraints serve as excellent usage guidelines.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_raw_htmlGet the HTML of a pageARead-only
Fetch the HTML source of any public web page through ScrapingBot's proxies. Use useChrome for pages built with JavaScript. Prefer the dedicated tools: scrape_retail for product pages, scrape_real_estate, scrape_booking, scrape_search_engine for Google or Bing, and scrape_social_media for Instagram, TikTok, Facebook, LinkedIn, X/Twitter or Threads (blocked here). Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Full URL of the page, starting with https:// | |
| maxChars | No | Maximum number of characters returned. Default 100000. | |
| useChrome | No | Render the page with a headless Chrome (JavaScript sites). Default false. | |
| premiumProxy | No | Use a premium proxy for sites that block regular proxies. Default false. | |
| proxyCountry | No | Two-letter country code of the proxy (requires premiumProxy), e.g. FR, US. | |
| premiumFallback | No | When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off. | |
| waitForNetworkRequests | No | With useChrome, wait for the page's network requests to finish. Default false. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the readOnly/openWorld annotations by disclosing the credit cost model for each option, refunds on failure, the list of sites that force a premium proxy, and the automatic retry-with-premium-fallback behavior. It does not describe the response shape, but the cost and retry disclosures are exactly the kind of context the annotations cannot carry.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads purpose and alternates, then the cost model. Dense but every sentence carries routing or billing information; the long single paragraph buries the pricing detail a bit and could be split, but there is no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter, no-output-schema tool, it covers routing, cost, retry semantics, and the special-domain case. The only real gap is that it doesn't describe the return payload (truncation via maxChars is only implied by the schema), but that is minor given the schema's completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% so the baseline is 3; the description adds real meaning by tying useChrome and premiumProxy to concrete cost deltas and flagging which domains trigger premium pricing automatically. The premiumFallback behavior is restated with the billing condition, which reinforces rather than duplicates the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Fetch the HTML source of any public web page') plus the mechanism (ScrapingBot's proxies). It also distinguishes itself from every relevant sibling by name, so an agent can route correctly without opening other schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says to prefer dedicated tools and enumerates which ones for which domains (retail, real estate, booking, search engine, social media), and notes social media is blocked here. Clear when-to-use and when-not-to-use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_real_estateScrape a real estate listingARead-only
Extract a property listing (price or rent, surface, rooms, bedrooms, location, images...) from real estate sites such as SeLoger, Leboncoin, Rightmove or Zillow. Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Full URL of the page, starting with https:// | |
| useChrome | No | Render the page with a headless Chrome (JavaScript sites). Default false. | |
| premiumProxy | No | Use a premium proxy for sites that block regular proxies. Default false. | |
| proxyCountry | No | Two-letter country code of the proxy (requires premiumProxy), e.g. FR, US. | |
| premiumFallback | No | When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes far beyond annotations by detailing complex behaviors: pricing schemes, refund policies for failed requests, and the specific billing outcome of the retry mechanism for blocked sites. This operational knowledge is critical.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
While the description is detailed and lengthy, it is highly dense with value. It avoids fluff, with every sentence providing necessary operational constraints, such as cost or technical limitations.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of the tool (multiple optional parameters, pricing structure, and varied site requirements), the description is complete, providing all necessary behavioral and cost contexts required for safe and effective tool invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although schema coverage is 100%, the description adds necessary semantic depth by explaining how certain parameters like `premiumProxy` and `premiumFallback` interact with site policies, retry success, and billing outcomes.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description is explicit, stating it extracts data like price, surface, rooms, and location specifically from real estate sites (SeLoger, Zillow). It clearly defines the tool's resource and core function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It defines operational boundaries by mentioning specific supported sites (SeLoger, Rightmove) and warns that certain sites necessitate premium proxies, guiding the agent on when specific configurations are required.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_retailScrape a product pageARead-only
Extract structured product data (title, price, currency, images, brand, stock, EAN...) from an e-commerce product page such as Amazon, Cdiscount, Fnac or Zalando. Prefer it to scrape_raw_html for any product page. Returns JSON. Costs 1 credit, 5 with useChrome, 10 with premiumProxy, 25 with both (and for Google pages); failed requests are refunded. Some protected sites always use a premium proxy and cost 10 credits (e.g. amazon.fr, cdiscount.com, fnac.com, leboncoin.fr, seloger.com, rightmove.co.uk, zillow.com). A request blocked by the site is retried once with a premium proxy, billed at the premium price only if it succeeds (the result then says so).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Full URL of the page, starting with https:// | |
| useChrome | No | Render the page with a headless Chrome (JavaScript sites). Default false. | |
| premiumProxy | No | Use a premium proxy for sites that block regular proxies. Default false. | |
| proxyCountry | No | Two-letter country code of the proxy (requires premiumProxy), e.g. FR, US. | |
| premiumFallback | No | When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes far beyond the readOnlyHint/openWorldHint annotations: discloses credit pricing per option combination (1/5/10/25), refunds on failure, a named list of sites that force a premium proxy at 10 credits, and an automatic single retry on block with conditional billing that the result reports. These are exactly the cost and side-effect facts an agent needs before spending credits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose is front-loaded in the first sentence, followed by the cost/retry behavior, which is the right order. The middle section is dense with nested parentheticals (cost tiers, site lists) that take a second read, slightly reducing scannability, but no sentence is redundant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description still names the returned fields and the return format ('Returns JSON'), and it covers cost, refund, retry and protected-site behavior. For a 5-parameter, single-endpoint scrape tool with a fully documented schema, nothing material is left for the agent to guess.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents every parameter's meaning, which sets the baseline at 3. The description adds real value by tying useChrome, premiumProxy, premiumFallback and proxyCountry combinations to their credit costs and describing the fallback billing rule, but it does not elaborate on parameter interactions beyond pricing (e.g., proxyCountry+premiumProxy dependency is only implied).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Extract structured product data ... from an e-commerce product page') and enumerates exactly which fields come back (title, price, currency, images, brand, stock, EAN). It also names concrete target sites and explicitly positions itself above the sibling scrape_raw_html, so the agent can distinguish it without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit routing rule: 'Prefer it to scrape_raw_html for any product page.' That names the alternative and the condition selecting this tool. It stops short of saying when NOT to use it (e.g., product pages on real-estate/booking sites are covered by scrape_real_estate and scrape_booking), so the sibling set is not fully partitioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_search_engineSearch Google or BingARead-only
Get search engine results (organic results, related searches, people also ask...) as JSON. Use it instead of fetching google.com or bing.com pages with scrape_raw_html. Costs 30 credits; failed requests are refunded.
| Name | Required | Description | Default |
|---|---|---|---|
| page | No | Result page, default 1 | |
| engine | No | Default google | |
| search | Yes | The search query | |
| resultLang | No | Language of the results, e.g. en (default), fr | |
| searchType | No | Google only: search (default), maps or trends | |
| domainCountry | No | Two-letter country of the results, e.g. US (default), FR |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint and openWorldHint, so safety is covered; the description adds genuinely new operational context: a concrete cost of 30 credits, a refund policy for failed requests, and the JSON return shape. It does not cover rate limits, blocking/captcha behavior, or result-count limits per page, which keeps it below a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, both dense: capability plus output shape first, then the routing alternative and cost model. No filler and nothing important buried at the end.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 6-parameter, read-only tool with no output schema, the description gives enough (output fields, cost, refund) to call it confidently, and the detailed schema fills in query/engine/locale. Minor gaps are behavioral edge cases like pagination limits or failure modes beyond refunds.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter including enum constraints on engine and searchType is already documented; the description adds no parameter-level detail. Baseline 3 applies since the schema carries the full semantic load.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (get) and resource (search engine results) and enumerates the returned content (organic results, related searches, people also ask). It also distinguishes itself from the closest sibling, scrape_raw_html, so an agent can route without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly tells the agent to use this instead of fetching google.com/bing.com via scrape_raw_html, which is a clear use-condition plus a named alternative. It stops short of stating when not to use it (e.g., non-search pages, or the specialized retail/real-estate scrapers), so it is strong but not exhaustive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_social_mediaScrape a social media pageARead-only
Scrape a public social media profile, post or search (Instagram, TikTok, Facebook, LinkedIn, X/Twitter, Threads). Always use this tool for these networks, never scrape_raw_html. Choose the scraper and give its inputs: instagramProfile (account (name or profile URL); returns the 12 latest posts), instagramPost (url), tiktokProfile (url), tiktokHashtag (hashtag; max_video_count), facebookProfile (url; load_more_items_count), facebookPost (url), facebookOrganization (url), facebookMarketplaceItem (url; country), facebookMarketplaceSearch (keyword, city, country; items_limit), linkedinProfile (url; language), linkedinCompanyProfile (url; language), linkedinPost (url), linkedinSearchResult (keywords; page, language), twitterProfile (url; number_of_tweets), twitterSearch (search; posts_max_count), threadsProfile (url), threadsPost (url). Waits for the result for about a minute, otherwise returns a responseId for get_social_media_result: use that tool to wait longer, as calling this one again is billed again. Costs 30 credits; failed requests are refunded.
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | Profile, post or page URL | |
| city | No | ||
| page | No | ||
| search | No | twitterSearch | |
| account | No | instagramProfile: account name or profile URL | |
| country | No | ||
| hashtag | No | ||
| keyword | No | facebookMarketplaceSearch | |
| scraper | Yes | ||
| keywords | No | linkedinSearchResult | |
| language | No | ||
| items_limit | No | ||
| max_video_count | No | ||
| posts_max_count | No | ||
| number_of_tweets | No | ||
| load_more_items_count | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the readOnlyHint/openWorldHint annotations: ~1 minute wait, a responseId fallback, the warning that re-calling bills again, a 30-credit cost, and refunded failures. Those billing and timeout traits are exactly what an agent needs before committing to a call.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose, then the routing rule, then the scraper inventory — good ordering with no filler. It is a dense single block, though; the 17-value enumeration would scan faster as a list, and repeated parenthetical syntax makes it a wall of text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 16-parameter, no-output-schema tool this is complete: it covers per-scraper inputs, cost, timing, failure/refund behavior, and the recovery tool. Nothing an agent needs to call it once and correctly appears to be missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 31% with 16 parameters, so the description carries the load — and it does, by mapping each scraper enum value to the specific inputs it consumes (e.g. 'facebookMarketplaceSearch (keyword, city, country; items_limit)'). Remaining gaps are minor: no default values, no stated behavior for irrelevant parameters passed alongside a scraper, and no format hints for 'page' or 'language'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Scrape a public social media profile, post or search') and names all six supported networks. It also explicitly separates itself from the scrape_raw_html sibling, so an agent can route correctly without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit exclusion ('Always use this tool for these networks, never scrape_raw_html') and a follow-up path for the timeout case ('use get_social_media_result ... calling this one again is billed again'). Both when-to-use and when-not-to-use are covered, along with the named alternative.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
v1.4.0- Added
get_account - Added
get_usage_history
5 tool updates
v1.1.0- Changed
scrape_booking1 field changed- added
Input schema / properties / premiumFallbackAdded value: +{ + "description": "When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off.", + "type": "boolean" +}
- Changed
scrape_grocery1 field changed- added
Input schema / properties / premiumFallbackAdded value: +{ + "description": "When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off.", + "type": "boolean" +}
- Changed
scrape_raw_html1 field changed- added
Input schema / properties / premiumFallbackAdded value: +{ + "description": "When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off.", + "type": "boolean" +}
- Changed
scrape_real_estate1 field changed- added
Input schema / properties / premiumFallbackAdded value: +{ + "description": "When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off.", + "type": "boolean" +}
- Changed
scrape_retail1 field changed- added
Input schema / properties / premiumFallbackAdded value: +{ + "description": "When the site blocks the request, retry it once with a premium proxy, billed at the premium price only if it succeeds. Default true; false turns it off.", + "type": "boolean" +}
1 tool update
v1.0.1- Changed
scrape_social_media1 field changed- removed
Input schema / properties / posts_numberRemoved value: -{ - "exclusiveMinimum": 0, - "type": "integer" -}
8 tool updates
v1.0.0- First observed
get_social_media_result - First observed
scrape_booking - First observed
scrape_grocery - First observed
scrape_raw_html - First observed
scrape_real_estate - First observed
scrape_retail - First observed
scrape_search_engine - First observed
scrape_social_media
TDQS
Scored across 10 tools
The split between scrape_raw_html and the domain-specific extractors (retail, real estate, booking, grocery, search engine, social) is explicitly documented, including 'prefer X over raw_html' guidance, which removes most ambiguity. The residual overlap is scrape_retail vs scrape_grocery (both extract product title/price/brand from an e-commerce-like page) and scrape_booking vs scrape_real_estate for rental listings, where an agent could plausibly pick either.
Every tool is snake_case with a clear verb_noun form: scrape_<domain> for extraction and get_<resource> for retrieval (get_account, get_usage_history, get_social_media_result). The grouping is immediately legible and the async continuation tool is named to pair with its producer.
Ten tools is a tight, well-earned surface for a scraping service: one generic fetcher, five domain extractors, one search-engine tool, one social-media tool, and account/usage/async-result support. Nothing looks redundant or padding-like, and the set stays well under typical context-bloat thresholds.
The lifecycle is essentially covered: fetch, specialized extraction, account credit state, usage/error history, and an async result poller for the long-running social path. Minor gaps remain — no batch/bulk scraping, no screenshot or rendered-page artefact, and no way to discover which sites/APIs are supported other than the hardcoded examples in descriptions.
Maintenance
Related MCP Connectors
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
Web MCP: scrape/crawl sites, web search, brand assets, app stores, YouTube, Reddit, Hacker News.
Unblocking and fresh web data for agents: URL to Markdown, YouTube, Maps, Amazon, jobs. Pay per call
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables agents to scrape, crawl, map, search, and extract web pages as clean markdown or structured JSON directly through MCP tools.AGPL 3.0
- AlicenseAqualityAmaintenanceEnables AI agents to scrape web pages into Markdown, text, HTML, JSON, PDFs, screenshots, and structured data, while managing batch scraping jobs, persistent browser sessions, usage/request history, and docs lookup through native MCP tools.25120 npm1Apache 2.0
- AlicenseNot gradedqualityBmaintenanceEnables MCP-compatible AI agents to search, scrape, crawl, map, and interact with live web pages, returning clean, structured, agent-ready content. It also supports recurring page monitoring, autonomous deep research, and credit-usage checks.151,287 npm1MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.-