cute-web-scraper

An MCP server that gives Claude web scraping powers. Free, local, no API key, no cloud account.

Ask in plain English. It fetches the pages, renders the JavaScript when needed, gets past the blocks, and hands back clean markdown or a queryable table — no selectors, no glue code.

Why this one

• It gets in. Four escalating tiers — plain HTTP, browser TLS fingerprints, a real browser, then a stealth browser. ASOS, eBay, Booking.com and Trustpilot all return real data, on a home connection with no proxies.

• It doesn't waste your context. Articles are stripped of navigation, cookie banners and footers: a BBC news page goes from 20,513 characters to 3,198. Large results land in a SQLite table you query with SQL instead of pasting into the chat.

• It tells the truth about failure. Five of the sites tested served a refusal under a success status — an interstitial under HTTP 200, a bot check under 202 — and one served real content under 403. Block detection weighs the page body, not the status code, so you don't get a stub reported as data.

• Whole sites, not single pages. Sitemap discovery, parallel fetching, and 24 tools covering products, contacts, Shopify catalogues, places, PDFs and change tracking.

Install

pipx install git+https://github.com/maccydee/cute-web-scraper

Chromium is downloaded automatically the first time you use js_render (a one-off ~130MB).

Related MCP server: mcp-server-scraper

Connect it to Claude Code

claude mcp add cute-web-scraper -- cute-web-scraper

Then just ask:

Scrape every product from https://example-shop.com and give me a CSV of name and price.

Tools

Fetching and discovery

Extraction

Places and local businesses

Change tracking

Result tables

A typical run composes them: analyze_website → crawl_site → fetch_pages → query_table.

Extracting arbitrary fields

extract_by_selector covers everything the fixed extractors do not:

Get the title, price and link from every product on these 40 pages,
save it as `catalogue`, then show me anything under £50.

fields maps column names to CSS selectors. row_selector makes each match a row — that is what turns a listing into a table. An @attr suffix reads an attribute instead of text, with href and src resolved to absolute URLs:

{"name": "h3 a@title", "price": ".price_color", "link": "h3 a@href"}

Driving a page

fetch_page takes actions, which run before the page is read — cookie gates, "load more" buttons, infinite scroll and search forms:

[{"action": "click", "selector": "#accept-cookies"},
 {"action": "scroll_to_bottom", "max_rounds": 10}]

Available actions: click, type, press, wait, wait_for, scroll, scroll_to_bottom and click_until_gone. Each reports what it did, so a step that silently matched nothing is visible rather than leaving you guessing.

Reading the API instead of the page

When a site is awkward to parse, inspect_network renders it and reports the requests it made. A JavaScript page almost always loads its data from an endpoint you can fetch directly — cheaper than parsing markup, and it survives redesigns that break selectors:

Inspect the network on this listing page, then fetch whatever JSON endpoint it uses.

Watching for changes

Check https://example.com/pricing for changes.

track_changes stores a snapshot and reports new, same or changed with a unified diff. That is monitoring without a scheduler — check whenever you like and see only the difference.

Slash commands

The server ships four ready-made workflows, which appear as slash commands in Claude Code: scrape_site, scrape_shopify_store, find_contacts and compare_prices.

Working with large scrapes

Any tool that returns rows accepts save_as. Instead of putting the data in the conversation, it writes a result table and hands back a summary:

Extract the whole catalogue from deathwishcoffee.com into a table called `catalogue`,
then tell me the price range and how many variants are out of stock.

Claude calls extract_shopify_store(save_as="catalogue"), gets back a row count and column list, and then answers with query_table:

SELECT COUNT(*) AS variants, MIN(price) AS cheapest,
       MAX(price) AS dearest, SUM(available) AS in_stock
FROM catalogue

The table can hold 100,000 rows and none of them enter the conversation. query_table is strictly read-only — it runs against a read-only SQLite handle and rejects anything that is not a SELECT, so a query can never modify or delete saved data.

Tables live in a SQLite file at ~/.cute-web-scraper/results.db (set SCRAPER_DB_PATH to move it).

Cleaning data

query_table also takes save_as, which persists the result as a new table. SQL already expresses the usual cleanup operations, so there's no separate set of edit tools:

SELECT DISTINCT * FROM leads                                  -- deduplicate
SELECT street || ', ' || city AS address FROM leads           -- merge columns
SELECT name, phone FROM leads WHERE phone IS NOT NULL         -- drop columns and rows
SELECT vendor AS brand FROM catalogue                         -- rename

The source table is left untouched unless you deliberately target its own name, and the response says replaced_existing_table when you do — so an in-place filter is never a silent loss of rows.

Places and local businesses

find_places looks up a single place; find_places_nearby returns everything of a category within a radius, which is the local lead-generation case:

Find every dentist within 4km of Bath, save it as `leads`,
then tell me how many have a website but no phone number.

Categories accept friendly names (cafe, dentist, hotel, solicitor, gym, hairdresser, …) or a raw OpenStreetMap tag like amenity=dentist.

A note on the data source. This is OpenStreetMap, not Google Maps. Google was the obvious target and it does not work: an automated browser gets a cookie-consent interstitial, and once past that, a degraded map shell with no place panel. The stealth tier does not help, because this is a consent wall rather than bot detection — a different problem from the one stealth solves.

OpenStreetMap gives the same fields — name, address, coordinates, phone, website, opening hours, category — through documented open endpoints with no key. The one thing it has no equivalent for is star ratings and review counts, which are Google's own proprietary data.

Both endpoints are volunteer-run. Nominatim's policy of one request per second is enforced internally regardless of SCRAPER_DELAY_MS, and Overpass queries fall through several public mirrors, because the main instance regularly returns 504 under load.

Tool output is also capped at SCRAPER_MAX_INLINE_CHARS (25,000 by default). Past that, a result is truncated with a note pointing at save_as — so a single call can't fill your context by accident.

Example prompts

Export the whole catalogue from deathwishcoffee.com and tell me the price range.

Find all email addresses on https://company.com and its contact pages.

What platform is https://myblog.com on? Does it need JavaScript to scrape?

Scrape these 200 product pages into a table, then show me everything under £50 that's in stock.

Extract the social media links from these 10 agency sites: [urls...]

Configuration

Everything is an environment variable, with defaults that work unconfigured.

Batching a long URL list into one table needs mode: "append" on every call after the first, or each batch replaces the last. Rendered pages that come back sparse can be given wait_ms, or better wait_for with a CSS selector.

How it behaves

Main content, not the whole page. Article-shaped pages are run through trafilatura, which isolates the body and drops the surrounding furniture — chosen because on an independent 2,008-page benchmark it scores 0.791 F1 against Readability's 0.674. It is applied per page rather than universally: the same benchmark shows extractors diverging by 20–30 points on product grids and collections, where "main content" is not an article, so listing pages keep the full document. Pass main_content: false to force that anywhere.

Four tiers, escalating only when refused. A plain HTTP client handles most pages. If a site refuses, the request retries with real browser TLS fingerprints (Chrome, then Safari), because some sites fingerprint the TLS handshake itself and no header change gets past them. js_render: true renders in Chromium for single-page apps. As a last resort, a stealth-patched browser handles sites that need JavaScript and reject ordinary automation.

Each tier fixes a different failure, and none is a superset of the others: the TLS tier can't run JavaScript, and Playwright is a detectably automated browser. Every result reports which tier served it. Set SCRAPER_IMPERSONATE=0 or SCRAPER_STEALTH=0 to switch the last two off and let blocks stand.

The last two tiers are evasion, not politeness — they exist to get past bot detection that sites deliberately deployed. They only ever run after a refusal, never on a site that served the page normally.

Adaptive backoff. Requests to the same domain are spaced by SCRAPER_DELAY_MS, measured start to start, so the delay caps the request rate rather than adding to slow responses. When a domain pushes back — a 429, a 403, a Cloudflare challenge — the delay for that domain doubles, up to 60 seconds, and decays back down once requests succeed again. Domains are tracked independently, so scraping two sites at once costs nothing extra.

robots.txt is not enforced. It is read only to locate sitemaps; its Disallow rules are not consulted and there is no setting to change that. The adaptive per-domain delay is this tool's politeness mechanism.

A short cache. Fetched pages are reused for five minutes, so running fetch_pages and then extract_emails over the same URLs does not fetch everything twice.

HTTP mode

The default is stdio, which is what claude mcp add above uses. To run a persistent shared instance instead:

SCRAPER_AUTH_TOKEN=$(openssl rand -hex 16) cute-web-scraper --http --port 8080

claude mcp add --transport http cute-web-scraper http://127.0.0.1:8080/mcp

It binds 127.0.0.1 and exposes /mcp plus a /health endpoint. Binding anywhere beyond loopback requires SCRAPER_AUTH_TOKEN, and the server refuses to start without it rather than quietly publishing an open scraper to your network.

Limitations

• No proxy rotation and no CAPTCHA solving. A site that survives all four tiers is reported as blocked rather than guessed at.

• LinkedIn and similar may need SCRAPER_CHROME_USER_DATA_DIR pointed at a logged-in Chrome profile.

• SCRAPER_DELAY_MS=0 removes the polite delay, but backoff still engages when a site pushes back.

• Phone extraction is deliberately conservative: it requires a country code or a trunk prefix, so it misses some bare local formats rather than returning years and order numbers.

Development

uv sync --extra dev

uv run pytest -v

uv run pytest -m integration -v -s

uv run ruff check src/ tests/ && uv run mypy src/cute_web_scraper/

Unit tests are hermetic and never touch the network. Integration tests hit live sites and are excluded from the default run.

License

MIT — see LICENSE.