cute-web-scraper
With cute-web-scraper, Claude can scrape, crawl, and extract structured data from websites using plain English instructions.
Page Fetching: Retrieve single or multiple URLs as clean markdown, with optional JavaScript rendering for dynamic pages.
Site Discovery & Analysis: Crawl websites via sitemaps (including indexes and robots.txt) or link-following to discover all pages. Analyze sites to detect platforms (Shopify, WordPress, etc.), locate sitemaps, estimate page count, and determine if JavaScript rendering is needed.
Data Extraction: Extract structured product data (name, price, currency, availability, brand, SKU, rating), email addresses with surrounding context, phone numbers with context, all hyperlinks, and social media profile links across eight platforms (LinkedIn, X/Twitter, Facebook, Instagram, YouTube, TikTok, GitHub, Pinterest). Specialized functions extract entire Shopify store catalogues and collections.
Local Business Search: Leverage OpenStreetMap data to find places by name, description, or business category within a radius, returning addresses, coordinates, phone numbers, websites, and opening hours.
Data Management: Save scraped data into persistent, named tables; run read-only SQL queries to filter, aggregate, group, sort, and clean (deduplicate, merge/drop/rename columns); export tables to CSV or JSON.
Pre-defined Workflows: Use slash commands for common tasks like
scrape_site,scrape_shopify_store,find_contacts, andcompare_prices.Technical Capabilities: MCP server with stdio and HTTP modes for Claude integration, adaptive backoff for polite scraping, and response caching for efficiency.
Provides tools for finding places and local businesses, with details such as name, address, coordinates, phone, website, and opening hours, sourced from OpenStreetMap.
Provides tools for extracting a Shopify store's full catalogue, product variants, and collections.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@cute-web-scraper Find every email address on https://company.com and its contact page"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
cute-web-scraper
An MCP server that gives Claude web scraping powers. Free, local, no API key, no cloud account.
Ask in plain English. It fetches the pages, renders the JavaScript when needed, gets past the blocks, and hands back clean markdown or a queryable table — no selectors, no glue code.
Why this one
It gets in, for free. Four escalating tiers — plain HTTP, browser TLS fingerprints, a real browser, then a stealth browser. ASOS, eBay, Booking.com and Trustpilot all return real data on a home connection, with no proxies, no API key and no per-page cost. Hosted scrapers put their anti-bot behind metered credits and leave it out of their open-source builds; the two fingerprint tiers here are the ones that actually got past those four sites, and they cost nothing.
It doesn't waste your context. Articles are stripped of navigation, cookie banners and footers: a BBC news page goes from 20,513 characters to 3,198. Large results land in a SQLite table you query with SQL instead of pasting into the chat.
It tells the truth about failure. Five of the sites tested served a refusal under a success status — an interstitial under HTTP 200, a bot check under 202 — and one served real content under 403. Block detection weighs the page body, not the status code, so you don't get a stub reported as data.
Whole sites, not single pages. Sitemap discovery, parallel fetching, and 25 tools covering products, contacts, Shopify catalogues, places, screenshots and change tracking.
Documents too. A link to a PDF, .docx or .xlsx is extracted to text and markdown tables rather than silently skipped — and the file type is identified from the bytes, because servers routinely mislabel them.
Install
pipx install git+https://github.com/maccydee/cute-web-scraperChromium is downloaded automatically the first time you use js_render (a one-off ~130MB).
Related MCP server: mcp-server-scraper
Connect it to Claude Code
claude mcp add cute-web-scraper -- cute-web-scraperThen just ask:
Scrape every product from https://example-shop.com and give me a CSV of name and price.Tools
Fetching and discovery
Tool | What it does |
| Search the web and get ranked results, optionally with each result's full text in the same call |
| One URL to clean markdown, with title, status and link count |
| Report the API calls a page makes, with their JSON — read the data source directly |
| Render a page and save a PNG to disk — what it actually looks like, not just its text |
| Many URLs in parallel, returning results and per-URL errors |
| Discover a site's pages via sitemap, falling back to link-following |
| Detect the platform, find the sitemap, report whether JS is needed |
Extraction
Tool | What it does |
| Arbitrary fields via CSS selectors — turns any listing into a table |
| Structured product data (name, price, currency, availability, brand, sku, rating) from JSON-LD, OpenGraph or microdata |
| Email addresses across a list of URLs, with surrounding context |
| Phone numbers across a list of URLs, with surrounding context |
| Every hyperlink, resolved to absolute URLs |
| Social profiles across eight platforms |
| A whole Shopify catalogue, one row per variant |
| A Shopify store's collections and their product counts |
Places and local businesses
Tool | What it does |
| Search by name or description — name, address, coordinates, phone, website, opening hours |
| Every business of a category within a radius of a place |
Change tracking
Tool | What it does |
| Fetch a page and diff it against the last check — new, same or changed |
| Pages being watched, and when each was last seen |
| Stop watching a page |
Result tables
Tool | What it does |
| Saved result tables with row counts and columns |
| One table's columns, row count and a sample |
| Read-only SQL over a saved table — filter, aggregate, group, sort, and optionally save the result as a new table |
| Write a table to CSV or JSON on disk |
| Delete a saved table |
A typical run composes them: analyze_website → crawl_site → fetch_pages → query_table.
Extracting arbitrary fields
extract_by_selector covers everything the fixed extractors do not:
Get the title, price and link from every product on these 40 pages,
save it as `catalogue`, then show me anything under £50.fields maps column names to CSS selectors. row_selector makes each match a row — that is what turns a listing into a table. An @attr suffix reads an attribute instead of text, with href and src resolved to absolute URLs:
{"name": "h3 a@title", "price": ".price_color", "link": "h3 a@href"}Driving a page
fetch_page takes actions, which run before the page is read — cookie gates, "load more" buttons, infinite scroll and search forms:
[{"action": "click", "selector": "#accept-cookies"},
{"action": "scroll_to_bottom", "max_rounds": 10}]Available actions: click, type, press, wait, wait_for, scroll, scroll_to_bottom and click_until_gone. Each reports what it did, so a step that silently matched nothing is visible rather than leaving you guessing.
Reading the API instead of the page
When a site is awkward to parse, inspect_network renders it and reports the requests it made. A JavaScript page almost always loads its data from an endpoint you can fetch directly — cheaper than parsing markup, and it survives redesigns that break selectors:
Inspect the network on this listing page, then fetch whatever JSON endpoint it uses.Watching for changes
Check https://example.com/pricing for changes.track_changes stores a snapshot and reports new, same or changed with a unified diff. That is monitoring without a scheduler — check whenever you like and see only the difference.
Slash commands
The server ships four ready-made workflows, which appear as slash commands in Claude Code: scrape_site, scrape_shopify_store, find_contacts and compare_prices.
Working with large scrapes
Any tool that returns rows accepts save_as. Instead of putting the data in the conversation, it writes a result table and hands back a summary:
Extract the whole catalogue from deathwishcoffee.com into a table called `catalogue`,
then tell me the price range and how many variants are out of stock.Claude calls extract_shopify_store(save_as="catalogue"), gets back a row count and column list, and then answers with query_table:
SELECT COUNT(*) AS variants, MIN(price) AS cheapest,
MAX(price) AS dearest, SUM(available) AS in_stock
FROM catalogueThe table can hold 100,000 rows and none of them enter the conversation. query_table is strictly read-only — it runs against a read-only SQLite handle and rejects anything that is not a SELECT, so a query can never modify or delete saved data.
Tables live in a SQLite file at ~/.cute-web-scraper/results.db (set SCRAPER_DB_PATH to move it).
Cleaning data
query_table also takes save_as, which persists the result as a new table. SQL already expresses the usual cleanup operations, so there's no separate set of edit tools:
SELECT DISTINCT * FROM leads -- deduplicate
SELECT street || ', ' || city AS address FROM leads -- merge columns
SELECT name, phone FROM leads WHERE phone IS NOT NULL -- drop columns and rows
SELECT vendor AS brand FROM catalogue -- renameThe source table is left untouched unless you deliberately target its own name, and the response says replaced_existing_table when you do — so an in-place filter is never a silent loss of rows.
Places and local businesses
find_places looks up a single place; find_places_nearby returns everything of a category within a radius, which is the local lead-generation case:
Find every dentist within 4km of Bath, save it as `leads`,
then tell me how many have a website but no phone number.Categories accept friendly names (cafe, dentist, hotel, solicitor, gym, hairdresser, …) or a raw OpenStreetMap tag like amenity=dentist.
A note on the data source. This is OpenStreetMap, not Google Maps. Google was the obvious target and it does not work: an automated browser gets a cookie-consent interstitial, and once past that, a degraded map shell with no place panel. The stealth tier does not help, because this is a consent wall rather than bot detection — a different problem from the one stealth solves.
OpenStreetMap gives the same fields — name, address, coordinates, phone, website, opening hours, category — through documented open endpoints with no key. The one thing it has no equivalent for is star ratings and review counts, which are Google's own proprietary data.
Both endpoints are volunteer-run. Nominatim's policy of one request per second is enforced internally regardless of SCRAPER_DELAY_MS, and Overpass queries fall through several public mirrors, because the main instance regularly returns 504 under load.
Tool output is also capped at SCRAPER_MAX_INLINE_CHARS (25,000 by default). Past that, a result is truncated with a note pointing at save_as — so a single call can't fill your context by accident.
Example prompts
Export the whole catalogue from deathwishcoffee.com and tell me the price range.
Find all email addresses on https://company.com and its contact pages.
What platform is https://myblog.com on? Does it need JavaScript to scrape?
Scrape these 200 product pages into a table, then show me everything under £50 that's in stock.
Extract the social media links from these 10 agency sites: [urls...]Configuration
Everything is an environment variable, with defaults that work unconfigured.
Variable | Default | Meaning |
|
| Base delay between requests to the same domain |
|
| Maximum parallel requests |
|
| How long a fetched page stays reusable |
|
| Cached pages before least-recently-used eviction |
| unset | Bearer token for HTTP mode |
| unset | Chrome profile to inherit logged-in sessions from |
|
| Retry blocked requests with browser TLS fingerprints |
|
| Last-resort stealth browser for the hardest blocks |
|
| Where result tables are stored |
|
| Ceiling on how much a single tool returns inline |
Batching a long URL list into one table needs mode: "append" on every call after the first, or each batch replaces the last. Rendered pages that come back sparse can be given wait_ms, or better wait_for with a CSS selector.
How it behaves
Main content, not the whole page. Article-shaped pages are run through trafilatura, which isolates the body and drops the surrounding furniture — chosen because on an independent 2,008-page benchmark it scores 0.791 F1 against Readability's 0.674. It is applied per page rather than universally: the same benchmark shows extractors diverging by 20–30 points on product grids and collections, where "main content" is not an article, so listing pages keep the full document. Pass main_content: false to force that anywhere.
Four tiers, escalating only when refused. A plain HTTP client handles most pages. If a site refuses, the request retries with real browser TLS fingerprints (Chrome, then Safari), because some sites fingerprint the TLS handshake itself and no header change gets past them. js_render: true renders in Chromium for single-page apps. As a last resort, a stealth-patched browser handles sites that need JavaScript and reject ordinary automation.
Each tier fixes a different failure, and none is a superset of the others: the TLS tier can't run JavaScript, and Playwright is a detectably automated browser. Every result reports which tier served it. Set SCRAPER_IMPERSONATE=0 or SCRAPER_STEALTH=0 to switch the last two off and let blocks stand.
The last two tiers are evasion, not politeness — they exist to get past bot detection that sites deliberately deployed. They only ever run after a refusal, never on a site that served the page normally.
Adaptive backoff. Requests to the same domain are spaced by SCRAPER_DELAY_MS, measured start to start, so the delay caps the request rate rather than adding to slow responses. When a domain pushes back — a 429, a 403, a Cloudflare challenge — the delay for that domain doubles, up to 60 seconds, and decays back down once requests succeed again. Domains are tracked independently, so scraping two sites at once costs nothing extra.
robots.txt is not enforced. It is read only to locate sitemaps; its Disallow rules are not consulted and there is no setting to change that. The adaptive per-domain delay is this tool's politeness mechanism.
A short cache. Fetched pages are reused for five minutes, so running fetch_pages and then extract_emails over the same URLs does not fetch everything twice.
HTTP mode
The default is stdio, which is what claude mcp add above uses. To run a persistent shared instance instead:
SCRAPER_AUTH_TOKEN=$(openssl rand -hex 16) cute-web-scraper --http --port 8080claude mcp add --transport http cute-web-scraper http://127.0.0.1:8080/mcpIt binds 127.0.0.1 and exposes /mcp plus a /health endpoint. Binding anywhere beyond loopback requires SCRAPER_AUTH_TOKEN, and the server refuses to start without it rather than quietly publishing an open scraper to your network.
Limitations
No proxy rotation and no CAPTCHA solving. This is the real ceiling: the tiers here defeat TLS and browser fingerprinting, not IP reputation. A site that blocks your address or geo-fences its content needs a proxy network, and a paid service with residential proxies will beat this on those. A site that survives all four tiers is reported as blocked rather than guessed at.
LinkedIn and similar may need
SCRAPER_CHROME_USER_DATA_DIRpointed at a logged-in Chrome profile.SCRAPER_DELAY_MS=0removes the polite delay, but backoff still engages when a site pushes back.Phone extraction is deliberately conservative: it requires a country code or a trunk prefix, so it misses some bare local formats rather than returning years and order numbers.
Development
uv sync --extra devuv run pytest -vuv run pytest -m integration -v -suv run ruff check src/ tests/ && uv run mypy src/cute_web_scraper/Unit tests are hermetic and never touch the network. Integration tests hit live sites and are excluded from the default run.
License
MIT — see LICENSE.
Available Tools
24 toolsanalyze_websiteA
Inspect a website before scraping it: detects the platform (Shopify, WordPress, Wix, ...), locates its sitemap, estimates how many pages it has, and reports whether JavaScript rendering is needed.
Inputs: url (required). Returns: result
crawl_siteA
Discover the pages on a website. Prefers the site's sitemap (following sitemap indexes and robots.txt), and falls back to following links. Returns JSON with urls, count, source and truncated. Run this before fetch_pages to scrape a whole site.
Inputs: url (required), limit. Returns: result
drop_tableA
Delete a saved result table. This permanently removes the stored rows; the scraped pages themselves are unaffected.
Inputs: name (required). Returns: result
export_tableA
Export a result table to a file on disk as CSV or JSON, and return its path. Use this to hand data to a spreadsheet or another tool.
Inputs: name (required), fmt, dest_dir. Returns: result
extract_by_selectorA
Extract arbitrary fields from pages using CSS selectors — the general case the fixed extractors do not cover. fields maps output column names to selectors, e.g. {"name": "h1", "price": ".price"}. Set row_selector when a page holds a list: each match becomes a row and the field selectors resolve inside it, which turns a listing into a table. Suffix a selector with @attr to read an attribute instead of text — "a@href" gives the link, resolved to an absolute URL. Pass save_as to store the rows; add mode='append' when batching.
Inputs: urls (required), fields (required), mode, save_as, +2 more. Returns: result
extract_emailsA
Scan a list of URLs for email addresses. Returns JSON with results ({url, value, context}) and errors. Pass save_as='<table_name>' to store results instead of returning them inline.
Inputs: urls (required), mode, save_as, js_render. Returns: result
extract_linksA
Collect every hyperlink from a list of URLs, resolved to absolute URLs. Returns JSON with results ({url, value, context}) and errors. Pass save_as='<table_name>' to store results instead of returning them inline.
Inputs: urls (required), mode, save_as, js_render. Returns: result
extract_phonesA
Scan a list of URLs for phone numbers. Returns JSON with results ({url, value, context}) and errors. Pass save_as='<table_name>' to store results instead of returning them inline.
Inputs: urls (required), mode, save_as, js_render. Returns: result
extract_productsA
Extract structured product data (name, price, currency, availability, brand, sku, image, rating, review_count) from a list of product URLs. Reads the page's own JSON-LD, OpenGraph or microdata rather than guessing at selectors, so it works across most storefronts without configuration. Pass save_as='<table_name>' to store the rows for querying.
Inputs: urls (required), save_as, js_render. Returns: result
extract_shopify_storeA
Extract a Shopify store's catalogue as one row per product variant — price, sku, options, availability, vendor, image and product URL. Reads the store's public products.json, so it needs no rendering and no selectors. Pass save_as='<table_name>' to store the rows (recommended: catalogues are large). max_products caps how many products are pulled.
Inputs: store_url (required), save_as, max_products. Returns: result
extract_social_linksA
Find social media profile links (LinkedIn, X, Facebook, Instagram, YouTube, TikTok, GitHub, Pinterest) across a list of URLs. Returns JSON with results ({url, platform, value}) and errors. Pass save_as='<table_name>' to store results instead of returning them inline.
Inputs: urls (required), mode, save_as, js_render. Returns: result
fetch_pageA
Fetch one web page and return its content as clean markdown with metadata. Set js_render=true for pages that need JavaScript to render (SPAs, infinite-scroll listings, most modern storefronts). If a rendered page still comes back sparse, give it longer with wait_ms, or wait for a specific element with wait_for (a CSS selector) — that is more reliable than a fixed delay. Article-shaped pages have their navigation, cookie banners and footers stripped automatically; set main_content=false to keep the whole page. PDFs are extracted to text.
actions drives the page before reading it (implies js_render). Each is {action, selector, ...}: click, type (text), press (key), wait (ms), wait_for, scroll (times), scroll_to_bottom (max_rounds) for infinite scroll, and click_until_gone (max_clicks) for a 'load more' button. Use it for cookie gates, paginated listings and search forms.
Inputs: url (required), actions, wait_ms, wait_for, +2 more. Returns: result
fetch_pagesA
Fetch many web pages in parallel. Returns JSON with results and errors. Set js_render=true for JavaScript-heavy pages. For more than about 20 URLs, pass save_as='<table_name>' to write the pages into a result table and get back a summary instead of the full text — then use query_table to interrogate it without filling the conversation. When feeding a long URL list through in batches, pass mode='append' on every call after the first, or each batch replaces the last.
Inputs: urls (required), mode, save_as, wait_ms, +2 more. Returns: result
find_placesA
Search for places and local businesses by name or description — 'the British Museum', 'cafes in Shoreditch'. Returns name, address, coordinates, phone, website, opening hours and category. Data comes from OpenStreetMap, so there are no star ratings or review counts; for those you would need a paid Google Places key. Pass save_as='<table_name>' to store the results.
Inputs: query (required), limit, save_as. Returns: result
find_places_nearbyA
Find every business of a category within a radius of a place — 'dentists near Bath', 'cafes within 2km of Shoreditch'. This is the tool for local lead generation: it returns name, address, phone, website and opening hours for each. category accepts friendly names (cafe, dentist, hotel, solicitor, gym, hairdresser, ...) or a raw OpenStreetMap tag like 'amenity=dentist'. Data is OpenStreetMap, so there are no star ratings. Pass save_as='<table_name>' to store the results.
Inputs: near (required), category (required), limit, save_as, +1 more. Returns: result
get_tableA
Inspect one result table: its columns, row count, and a small sample of rows.
Inputs: name (required), sample. Returns: result
inspect_networkA
Render a page and report the API calls it makes, with their JSON responses. A JavaScript site usually loads its data from an endpoint you can read directly — cleaner and far cheaper than parsing rendered markup, and it survives redesigns that break selectors. Use this when a page is hard to scrape, then fetch the endpoint it reveals. Set include_types to widen beyond JSON.
Inputs: url (required), save_as, wait_ms, wait_for, +2 more. Returns: result
list_shopify_collectionsA
List a Shopify store's collections with their product counts. Use this to pick which collections to extract before calling extract_shopify_store.
Inputs: store_url (required). Returns: result
list_tablesA
List saved result tables with their row counts and columns. Result tables are produced by any tool called with save_as.
Inputs: no named parameters. Returns: result
list_trackedA
List the pages being tracked for changes, with when each was last seen.
Inputs: no named parameters. Returns: result
query_tableA
Run a read-only SQL SELECT against saved result tables. This is how you analyse a large scrape without pulling it into the conversation: filter, aggregate, group and sort a table of any size and get back only the rows you asked for. Only SELECT is permitted — the query can never modify saved data. Example: SELECT vendor, COUNT(*) AS n, AVG(price) AS avg_price FROM catalogue GROUP BY vendor ORDER BY n DESC.
Pass save_as='<table_name>' to persist the query's result as a new table. That is how you clean data here: SELECT DISTINCT deduplicates, aliases rename columns, a || ', ' || b AS c merges them, and WHERE drops unwanted rows — all in one step, with the original left untouched unless you deliberately target its name.
Inputs: sql (required), save_as, max_rows. Returns: result
search_webA
Search the web and get back ranked results with titles, URLs and snippets — the way in when you have a question rather than a URL. Feed the urls straight into fetch_pages or extract_by_selector. No API key and no quota.
Inputs: query (required), limit, save_as. Returns: result
track_changesA
Fetch a page and report what changed since the last time it was checked. Returns status 'new', 'same' or 'changed', with a unified diff and line counts when it changed. This is monitoring without a scheduler: check a price, a job board or a policy page whenever you want and see only the difference.
Inputs: url (required), js_render, context_lines. Returns: result
untrackC
Stop tracking a page and forget its stored snapshot.
Inputs: url (required). Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 24 tools
Most tools have clearly distinct purposes: fetching, extracting, searching, tracking, and table operations are well separated. Minor overlap exists between extract_products and extract_shopify_store, and between fetch_page and fetch_pages, but the descriptions clarify their scope enough to avoid serious misselection.
Tool names follow a consistent snake_case verb_noun pattern: extract_*, fetch_*, list_*, find_*, query_table, export_table, drop_table. Even 'untrack' aligns with the tracking tools, making the set predictable and easy to navigate.
24 tools is on the high end, but the count is justified by the server's broad scope: discovery, fetching, extraction, monitoring, local business search, and result-table management. It feels comprehensive rather than redundant, though it approaches the upper limit.
The tool set covers the full scraping lifecycle: discover sites, fetch pages, extract structured data, inspect network calls, track changes, store results, query them, and export them. There are no obvious dead ends or missing core operations for a web-scraping server.
Maintenance
Related MCP Connectors
Web scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.
Web scraping MCP server: any URL to Markdown or JSON, with JavaScript rendering and batch jobs.
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Related MCP Servers
- FlicenseNot gradedqualityNot gradedmaintenanceAn MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.48 npm-
- AlicenseAqualityCmaintenanceMCP server for web scraping — extract clean markdown, links, and metadata from any URL. Free Firecrawl alternative.5119 npm6MIT
- AlicenseNot gradedqualityDmaintenanceOpen-source web scraper and extraction MCP server with JavaScript rendering, markdown output, PDF/DOCX parsing, structured errors, and validated extraction contract diagnostics for agents.2AGPL 3.0
- AlicenseNot gradedqualityBmaintenanceRemote MCP server for web scraping with anti-bot evasion. Provides stealth HTTP fetching, headless browser with Cloudflare bypass, CSS selectors, YouTube transcripts, and Markdown conversion.1MIT