Competitor Price Tracker

This is a small, complete competitor price tracker. It runs end to end on the ScrapingBee API, and it is small on purpose, so you can read the whole thing and adapt it to your own stack. It is the companion code for the article How to Track Competitor Prices Using Web Scraping. You clone it, point it at your competitors, and it sends you a Slack message when a competitor's price goes below yours.

It reads Amazon and Walmart prices with ScrapingBee's dedicated parsers. For other stores it uses the HTML API with AI extraction. Then it normalizes every response into one shape, saves a snapshot on each run, and alerts you when a competitor is cheaper.

Status: everything here was tested against the live APIs in June 2026, so the code and the numbers were real at that time. It is a reference you own and adapt, not a maintained product, so you stay in control and there is no lock-in. The design is the durable part. The version pins and exact field names are the parts to check against the current ScrapingBee API, because those move over time.

Setup

You need Python 3.10 or newer.

First, get a ScrapingBee API key from https://www.scrapingbee.com/. The free trial needs no card, which is enough to run this whole demo and see it work before you commit to anything.

Then install the dependencies and set your key:

pip install -r requirements.txt          # core deps: requests, beautifulsoup4, python-dotenv
cp .env.example .env                      # put your key in .env, it is loaded automatically
# or skip the file and export the key: export SCRAPINGBEE_API_KEY=...

Related MCP server: steamdt

Use

1. Edit targets.csv. Put one row for each competitor product:

   column	meaning
   our_sku	your internal SKU
   source	amazon (ASIN), walmart (item id), or generic (full URL)
   identifier	the ASIN, item id, or URL
   our_price	your current price, for the undercut comparison
   selector	(optional, generic rows) a CSS selector for the price, when auto-extraction cannot find it
   proxy	(optional, generic rows) premium (25 credits) or stealth (75 credits), for sites behind Cloudflare or DataDome

2. Run it:

   python tracker.py

   What I got running it:

   MOUSE-001    amazon   USD 13.99 (ours 14.99)  <-- UNDERCUT 6.7%
   MOUSE-002    walmart  USD 13.83 (ours 14.99)  <-- UNDERCUT 7.7%
   HOODIE-009   generic  USD 69.0 (ours 75.0)  <-- UNDERCUT 8.0% [OUT OF STOCK]

   The prices are live, so what you see is today's real numbers, not a hardcoded example. Each run appends a snapshot to history.csv. Set SLACK_WEBHOOK_URL if you want the alerts in Slack. Without it, they print to the console. The out-of-stock row is flagged but does not fire an alert, because an out-of-stock competitor is not really undercutting you.

3. Schedule it. The included .github/workflows/track.yml is manual by default, on purpose, so a fresh clone never spends your credits without you asking. To make it run every 6 hours, uncomment the schedule: block and add SCRAPINGBEE_API_KEY and SLACK_WEBHOOK_URL as repo secrets. Each run commits the new snapshot back to the repo, so the price history builds up over time. This is the git-scraping pattern. A plain cron job or a serverless cron works too.

Files

Match before you trust the numbers

A competitor's "Logitech Wireless Mouse" may not be your SKU. Comparing different products is a bug, not an insight. So matcher.py matches in three steps, strongest first. First it matches on GTIN or UPC, when both sides have one (ScrapingBee's Walmart parser returns gtin, and Amazon's product_details often carries a UPC). If there is no shared code, it matches on the model code, which is the field that really tells products apart, because "M185" and "M510" have near-identical titles but are different products. Only if that also fails does it fall back to fuzzy title matching, with a confidence score:

python matcher.py
#  [MATCH   ] conf=0.95 via model :: Logitech M185 Wireless Mouse, 2.4GHz with USB Mini Rece
#  [no match] conf=0.44 via title :: Logitech Silent Wireless Mouse, Blue/Gray, Walmart Excl
#  [no match] conf=0.1  via model :: Logitech M510 Wireless Mouse, 2.4 GHz USB Unifying Rece

Auto-accept the high-confidence matches, and send the needs_review pairs to a human. This matcher is a clean starting point that is honest about where it stops. It gets the common case right, and it does not confuse an M185 with an M510. Real catalogs bring more (thousands of SKUs, missing GTINs, variants, bundles, refurbished versus new), and those need more than a regex and difflib. So build on this skeleton, and rely on GTIN or UPC whenever your data has it, because that path never guesses.

Matching happens once, at setup. It is not in the tracking loop, so you decide what to track one time. build_targets.py does this for you. It searches a keyword, matches each result to your product, and prints the targets.csv rows worth keeping.

python build_targets.py "Logitech M185 Wireless Mouse" "logitech wireless mouse" 14.99
# LOGITECH-B004YA,amazon,B004YAVF8I,14.99    # conf=0.95 via model [ok] M185 Wireless Mouse, 2.4GHz ...

How it behaves

• Generic rows use a deterministic cascade, with AI last. fetch_generic tries a per-site CSS selector you set in targets.csv first, then JSON-LD (schema.org/Product), then an OpenGraph or microdata price meta tag, and only then AI extraction. The first three are deterministic and cost 1 credit. AI is the flaky and more expensive fallback. It sometimes returns HTTP 200 with a plain "Sorry, couldn't get the response from AI" body instead of JSON, and it still charges you. So most sites resolve without AI at all. For the awkward ones, add a selector and the page usually becomes trackable instead of failing. If all four miss, the tracker logs it and moves on instead of crashing.

• For protected sites, escalate the proxy per target. A site behind Cloudflare or DataDome blocks the cheap fetch, so the whole cascade comes back empty. Set proxy=premium (a residential proxy with JS, 25 credits) or proxy=stealth (for the hardest anti-bot, 75 credits) on that one target, and the cascade runs through the right proxy so it can get past the block. This is per-target and opt-in, so you do not pay stealth rates for sites that do not need it.

• Fetches run concurrently. The tracker fans out with a thread pool. Set MAX_WORKERS (default 8) to your plan's concurrency cap. With that, hundreds of SKUs refresh in about the time a sequential loop needs for a dozen. The results are still processed in order, so the alerts and the history stay deterministic.

• Alerts fire on change, not on every run. You are pinged when a competitor newly undercuts you, or drops even further, and is in stock at that moment. You are not pinged every 6 hours about a competitor that has been cheaper all week. Change detection needs the earlier prices, which is why the CI job commits history.csv back (see below).

• Only comparable offers alert. On a marketplace, the headline price is whatever wins the Buy Box. That can be a used unit, a third-party seller, or a multipack, not the item you actually sell. So fetch_amazon reads the Buy Box condition and seller, then records both in every snapshot, which keeps a wrong-offer comparison auditable instead of invisible. A used or multipack offer is logged with a ⚠ note, but it does not fire an undercut alert. A used unit at $7.72 is not an undercut of your new $14.99.

• History accumulates through git-scraping. history.csv is tracked on purpose. The GitHub Actions job checks it out, appends the new run, and commits it back, so the price history builds up in git and you can diff it over time. Running locally appends to it too, which is expected. Two things to know. It adds one commit and grows the CSV on every run, so a year of 6-hourly runs leaves about 1,500 small commits. If that bothers you, squash them or move to a database. Also, the bot pushes to the default branch, so if you protect that branch, point the workflow at a separate data branch or a database instead.

• It does not compare across currencies. Prices are compared only inside OUR_CURRENCY (default USD). There is no FX conversion, so a competitor priced in another currency is recorded and flagged [currency …≠USD, not compared], not mis-compared. This is on purpose. A €63 price is not "cheaper" than your $75, and the tracker will not pretend it is. The one hard case is the currency symbol. € and £ are unambiguous, but $ is shared by USD, CAD, AUD, MXN and more, so a bare $ cannot be told apart on its own. When only a symbol is available, the tracker assumes it is your OUR_CURRENCY, and the _symbol_currency helper still keeps a clearly foreign symbol like ¥ out of the comparison. The dedicated parsers and JSON-LD return an explicit code, so a real CAD is correctly kept separate from USD there. Only the generic symbol-only path has this gap. So track same-market competitors on that path, or use sources that give a currency code, and add FX yourself if you need cross-currency comparison.

• Marketplace prices move, and Walmart varies by store. One item id gave me $13.83, $13.52, and $9.88 across three calls, so do not treat any single fetch as the truth. Walmart's store_id is meant to pin one store for a like-for-like comparison, but every time I tried it, it was slow and only sometimes returned a price. So verify it works for your targets before you rely on it.

• It is US-centric by design. The Walmart parser is US-only, and the worked example uses Amazon.com in USD. Outside the US, use Amazon's domain parameter and the generic path. The Walmart parser will not apply there.

Cost

Check the numbers before you scale. Each run costs credits, and it adds up fast:

The numbers come from the per-call costs. The Amazon and Walmart parsers cost 5 to 15 credits each. The HTML API costs 1 credit without JS and 5 with JS. AI extraction adds 5 more. The free trial's 1,000 credits are enough to evaluate the whole thing and see your real numbers before you pay for anything. For production, size a paid plan to your SKU count, and watch your spend at https://app.scrapingbee.com/api/v1/usage.

Troubleshooting

• A 401, or the message "check SCRAPINGBEE_API_KEY", means the key is missing or wrong. The tracker exits with that message, so it is easy to spot.

• A 429 means you went over your plan's concurrency cap. The session already retries with backoff, so if it keeps happening, lower MAX_WORKERS.

• If pip install fails on a very new or locked-down Python, install just the core: pip install requests beautifulsoup4 python-dotenv. You only need duckdb, fastmcp, and pandas for mcp_server.py.

Tests

python test_tracker.py

The tests cover the parts I kept getting wrong: alert-on-change (no every-run spam), in-stock gating, cross-currency safety, and history accumulation. They use mocked fetchers, so they run offline and cost no credits.

License

It is MIT licensed, so you are free to use it, change it, and ship it. See LICENSE.