Scraper-Health-MCP

Autonomous Self-Healing Scraper Fleet Manager for Bright Data Scraper Studio

The first agent that closes Bright Data's self-healing loop end-to-end without a human in the middle — turning the resume_automation_job endpoint the official demo couldn't find into a one-command "deploy and forget" scraper fleet.

Built for WeMakeDevs Into the Scrape-Verse hackathon (Aug 17–23, 2026).

🎯 The Problem

Web scrapers break silently. When a target site renames a CSS class, moves a field, or redesigns a layout, traditional scrapers return empty/garbage rows with no alarm. The failure is discovered only when a human notices the data is wrong — by which point downstream systems have been consuming bad data for hours or days.

Related MCP server: Reaper MCP

✅ The Solution

Scraper-Health-MCP is an autonomous AI agent that wraps Bright Data's Scraper Studio self-healing API into a fully automated loop:

1. Monitor — Runs scrapers on schedule and health-checks every result

2. Detect — Catches breakage via schema validation, null-field detection, and row-count anomaly detection (Sentinel-style statistical baselines)

3. Heal — Triggers Bright Data's AI self-healing (refactor_template)

4. Auto-Approve — Programmatically accepts the AI's proposed extraction diff via resume_automation_job (the step Bright Data's own demo couldn't do)

5. Verify — Re-scrapes and confirms health is restored

6. Escalate — If healing fails, regenerates the scraper from scratch

🔑 The Technical Unlock

Bright Data's official self-healing demo repo (anil-bd/scraper-studio-self-healing-demo) explicitly states the public API "does not document an endpoint to approve programmatically" and exits with code 3 (awaiting approval) when the heal hits pending_answer, requiring manual UI approval.

The current docs DO expose POST /resume_automation_job with {message: true, auto_save: true} to accept the diff automatically. This means a fully autonomous, hands-off self-healing loop is now buildable — and almost no competitor will have found this.

🤖 MCP Server

The agent exposes 5 MCP tools callable from Claude Code, Cursor, or any MCP-compatible AI agent:

🏗️ Architecture

Register collector (ID + schema + required fields)
        │
        ▼
  Schedule / trigger run  ──►  POST /dca/trigger
        │
        ▼
  Health-check result  ──►  GET /dca/dataset  (schema drift? nulls? row-count anomaly?)
        │
   ├── healthy ─► log success, schedule next run
   └── broken  ─► TRIGGER SELF-HEAL
                    │
                    ▼
          POST /refactor_template  (targeted heal prompt from the failed field)
                    │
                    ▼
          Poll /refactor_template/progress
                    │
                    ├── status:"pending_answer" ─► POST /resume_automation_job
                    │                              {message:true, auto_save:true}   ← the unlock
                    └── done/failed
                    │
                    ▼
          Re-scrape + verify against baseline schema
                    │
                    ├── passes ─► log heal event, resume schedule
                    └── fails  ─► escalate: regenerate scraper from scratch (Workflow 1)
                                  + alert/notification

🛠️ Tech Stack

🚀 Setup

# Clone
git clone https://github.com/0xConsole/scraper-health-mcp.git
cd scraper-health-mcp

# Install
pip install -r requirements.txt

# Run (mock mode — no API key needed)
python -m uvicorn app.main:app --reload

# Run (live mode — with Bright Data API key)
export BRIGHTDATA_API_KEY="your-key-here"
python -m uvicorn app.main:app --reload

Open http://localhost:8000 to see the dashboard.

📡 API Endpoints

🎪 Demo

1. Click "Run Full Demo" on the dashboard

2. The agent health-checks all registered collectors

3. Simulates a site change (breakage) on the first collector

4. Detects the breakage via health check (null fields, row count anomaly)

5. Triggers AI self-healing (refactor_template)

6. Polls until the AI proposes a diff (pending_answer)

7. Auto-approves the diff via resume_automation_job ← the unlock

8. Re-scrapes and verifies health is restored

9. Logs the entire heal event in the timeline

📊 What's Real vs Mocked

📝 License

Apache 2.0

🔗 Links

• Live demo: https://scraper-health-mcp.vercel.app

• GitHub: https://github.com/0xConsole/scraper-health-mcp

• Hackathon: WeMakeDevs Into the Scrape-Verse (Aug 17–23, 2026)

• Bright Data Scraper Studio: https://brightdata.com/scraper-studio

Built by Sentinel Dev · Team: Xayaan Ibrahim (Individual)