Wayback Machine Scraper - Track Website Changes Over Time
ryanclinton--wayback-machine-searchRead-only
Retrieve historical Wayback Machine snapshots to track website changes over time, filter by date or URL, and verify what pages said in the past.
Instructions
Wayback Machine Scraper - Track Website Changes Over Time. Search the Internet Archive's Wayback Machine for historical snapshots of any website. Retrieve archived page metadata -- including timestamps, URLs, MIME types, HTTP status codes, and content hashes -- for up to 10,000 snapshots per run.
When to use: Use when you need archived Wayback Machine snapshots of a URL or domain over time, e.g. to see what a page said on a past date.
Returns: the run's dataset rows as structured JSON (one row per result), plus the run and dataset IDs for paging the full dataset.
Behavior: starts one run of the Apify Actor "ryanclinton/wayback-machine-search" through Apify's hosted MCP gateway (mcp.apify.com) with your own APIFY_TOKEN; the run uses your Apify account. Read-only: it only reads public web data and changes nothing on the source site. Docs: https://apify.com/ryanclinton/wayback-machine-search
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | URL or domain to search for snapshots (e.g., "example.com" or "https://example.com/page"). Ignored if you supply a list under URLs (batch). Console UI prefills "apify.com" as a placeholder; programmatic API runs only use this field if you explicitly send it. Example: "apify.com" | |
| urls | No | Optional: query multiple URLs in one run for comparative competitive intelligence. Each URL gets its own snapshots, change detection, and metrics in the output. The single URL above is added if both fields are populated. | |
| query | No | Filter the output to only snapshots whose change content matches a keyword. Searches changeSummary categories (e.g. "pricing"), keyDiffs, added/removed text, and full content. Pure post-filter — no extra API calls. Example: query "pricing" returns only the snapshots where pricing changed. | |
| dateTo | No | End date filter (YYYYMMDD or YYYY format, e.g., "20231231" or "2023") | |
| monitor | No | Turn on for scheduled monitoring runs. One flag enables: change detection, only-changed snapshots, change intelligence + diffs, alerts on moderate-or-major changes, timeline output, insights record, AND delta-since-last-run state tracking. Compose with a Use Case preset for industry-specific defaults. | |
| useCase | No | Pick a preset to auto-configure filters, change detection, and output mode for a common workflow. Set to "custom" to use the raw inputs below. SEO = monthly buckets + 200-only + change-only. Compliance = closest-date evidence with content. Competitor = monthly buckets + only-changed + alerts on moderate-or-major. Forensics = full-domain crawl, no collapsing. Options: custom = Custom (use the inputs below as-is); seo = SEO history audit; compliance = Legal / compliance evidence; competitor = Com… | custom |
| dateFrom | No | Start date filter (YYYYMMDD or YYYY format, e.g., "20200101" or "2020") | |
| matchType | No | How to match the URL. "exact" = exact URL only, "prefix" = URL prefix match, "host" = all pages on same host, "domain" = all subdomains too Options: exact = Exact URL; prefix = URL Prefix; host = Same Host; domain = All Subdomains. | exact |
| collapseBy | No | Collapse duplicate snapshots. "digest" = unique content only, "timestamp:6" = one per month, "timestamp:8" = one per day, "timestamp:10" = one per hour Options: = No collapsing; digest = Unique content only (digest); timestamp:6 = One per month; timestamp:8 = One per day; timestamp:10 = One per hour. | |
| fastLatest | No | Return the most recent snapshots faster by skipping older index segments. Only useful for domains with hundreds of thousands of snapshots — combine with Max Results to fetch the latest N quickly. | |
| maxResults | No | Maximum number of snapshots to return per URL (max 10,000 in a single CDX query — enable Auto-paginate to exceed). Ignored when Target Date is set (always returns one snapshot). | |
| mimeFilter | No | MIME type filter (e.g., "text/html" for HTML pages only) | |
| outputMode | No | What records to emit. "Snapshots" = one row per snapshot (default). "Timeline" = one row per stable version (consecutive identical-digest snapshots collapsed). "Report" = a markdown report in the key-value store + version records + a summary record. Options: snapshots = Snapshots (one row per archive); timeline = Timeline (one row per distinct version); report = Report (markdown + versions). | snapshots |
| targetDate | No | Return the single snapshot closest to this date — useful for legal/compliance evidence ("what did the page say on 2020-06-15?"). Accepts YYYY-MM-DD or YYYYMMDD. When set, overrides Max Results and returns one snapshot with distanceFromTargetDays. | |
| onlyChanged | No | Filter results to only the snapshots where the page actually changed (plus the first "initial" baseline). Requires Detect Changes. Turns 1,000 raw snapshots into ~50 actionable rows. | |
| useJsRender | No | Enable Playwright rendering for archived pages that return as empty React/Next/Gatsby SPA shells (notably HubSpot, late-2024+ Atlassian). Off (default) uses a cheap HTTP fetch only — fastest and cheapest, but returns empty shells for modern SPA pricing pages. Pair with `forceJsRender` to control whether every snapshot is rendered or only the ones that look like SPA shells. | |
| autoPaginate | No | When the CDX API hits its 10,000-row cap, automatically chunk the query year-by-year and merge the results — letting you pull full multi-year domain histories in one run. Costs more proxy time; only enable when you need the complete archive. | |
| statusFilter | No | HTTP status code filter (e.g., "200" for successful pages only) | |
| detectChanges | No | Compare each snapshot to the previous one and flag rows where the content hash or HTTP status changed. Adds a "changed" boolean and "changeType" string to every result. Free — uses metadata you already have. | |
| forceJsRender | No | If `useJsRender` is on, render every snapshot via Playwright instead of falling back from a cheap HTTP fetch. Use when you already know your URLs are SPA shells and don't want to pay for the cheap-fetch attempt. Ignored when `useJsRender` is off. | |
| generateReport | No | Always write a REPORT.md to the run's key-value store, even when Output Mode is "snapshots". Useful when you want both raw rows and a human-readable summary. | |
| includeContent | No | Fetch and include the archived page text content. Required for the diff/keyDiffs fields under Change Intelligence. Slow — fetches each snapshot individually with polite rate limiting. | |
| includeInsights | No | Push a single "insights" record at the top of the dataset with key events, risk signals, business signals, typed events, coverage, cross-URL comparison, and chartData. Turn off if you only want raw rows. | |
| maxContentFetch | No | Maximum number of pages to fetch content for (only used if Include Page Content is enabled) | |
| monitorStateKey | No | Optional override for the key-value-store key used by Delta mode. Auto-derived from the sorted URL list if not set; provide a stable string here if you want to share state across separate runs that have different URL inputs (e.g. weekly batch + ad-hoc spot-checks). | |
| alertOnMagnitude | No | Emit a single "alert" record at the top of the dataset when changes meet the threshold. Pairs well with scheduling: connect a webhook → Slack/email when the alert record fires. Requires Change Intelligence to compute magnitude. Options: none = No alert (default); any = Any change; moderate = Moderate or major change; major = Major change only. | none |
| deltaSinceLastRun | No | Turn on for stateful monitoring without external storage. The actor remembers the events and snapshot URLs returned in the previous run (in a named key-value store) and filters this run to only the events / snapshots that are NEW since then. First run for a given URL set returns everything as new. | |
| changeIntelligence | No | Augment every changed row with a changeSummary (magnitude: minor/moderate/major, categories: pricing/legal/product/layout/etc., keyDiffs: human-readable change phrases) and a structured diff (added/removed text, change score). Best results require Include Page Content so the actor can diff actual text. | |
| proxyConfiguration | No | Proxy used to reach the Wayback Machine CDX API and archived pages. Defaults to AUTO — Apify's smart router uses datacenter IPs where they work and falls back to residential where blocked. Switch to RESIDENTIAL explicitly if you see persistent 403s. Example: {"useApifyProxy":true} |