ScrapeForge

The first web scraper that learns page layouts and remembers them.
Zero-config structured extraction. One crawl per domain template, never again.

What is this?

ScrapeForge is a self-learning, LLM-powered web scraper that doesn't just extract data , it understands page templates and caches them for reuse.

Traditional scrapers break when a site redesigns. ScrapeForge breaks the cycle:

1. First visit --> LLM analyzes the DOM and generates a CSS extraction schema

2. Pattern learning --> URLs sharing the same layout are automatically grouped (/product/iphone and /product/samsung --> */product/*)

3. Forever after --> Every subsequent URL on that template reuses the cached schema zero LLM cost, zero latency

4. Rot detection --> If a cached schema starts failing, the system detects it empirically and regenerates

No XPath. No manual selectors. No brittle maintenance.

Related MCP server: extracto-mcp

The Problem It Solves

The Problem: Writing and maintaining CSS selectors for web scraping is time-consuming and breaks easily when websites update.

The 10x Solution: ScrapeForge makes data extraction 10x faster by using LLMs to automatically generate, empirically test, and cache robust extraction schemas.

Killer Features

• LLM Schema Generation: Feed it HTML, get a JsonCssExtractionStrategy schema automatically (Gemini, OpenAI, Ollama, any other provider supported)

• Pattern Generalization: Automatically discovers that amazon.in/*/dp/B0C8... and amazon.in/*/dp/B0DF... share one schema

• Empirical Verification: Before trusting a cached schema, it actually tests extraction on the new URL and measures fill-rate

• SQLite Schema Cache: Persistent, timestamped schema storage with domain-scoped lookups

• Production Hardened: Exponential backoff retries, asyncio timeouts, rate limiting, thread-safe file I/O

• MCP Server: Exposed as FastAPI + Model Context Protocol tools. Claude, Cursor, or any MCP client can scrape natively

• Multi-Output: Markdown, raw HTML, or structured JSON in one call

Tech Stack

Why This Is Different

Most "AI scrapers" call an LLM for every single page.
ScrapeForge calls the LLM once per template, then caches and generalizes. That means:

• 1000 product pages on Amazon = 1 LLM call (not 1000)

• Zero maintenance when sites add new articles/products

• Self-healing when layouts drift

Run Instructions (for a clean machine)

Follow these exact step‑by‑step commands to get the system running from scratch.

1. Clone the repository

git clone https://github.com/ajazhussainsiddiqui/ScrapeForge.git
cd ScrapeForge

2. Create and activate a Python virtual environment

python -m venv venv
source venv/bin/activate          # Linux / macOS
# or
.\venv\Scripts\activate           # Windows

3. Install all dependencies

pip install -r requirements.txt

4. Install Playwright browsers (required by Crawl4AI)

playwright install

5. Create the .env configuration file

echo "DATABASE_URL=sqlite:///./schema.db" > .env
echo "LLM_API_KEY=your_api_key_here" >> .env   # only needed for structured extraction

Note: If you only want to test Markdown / HTML crawling, you can leave LLM_API_KEY empty. For structured extraction, provide a Gemini key or anyother provider like OpenAI, Groq (the code expects the provider/model format, e.g. gemini/gemini-2.0-flash).

6. Initialize the database tables

python -c "from db.connection import engine, Base; Base.metadata.create_all(engine)"

7. Start the FastAPI server

uvicorn api:app  # run in terminal from root directory

Your API is now live at http://localhost:8000. The interactive Swagger documentation is available at http://localhost:8000/docs.

Concept Table (5+ key ideas & their location)

Demo Path (works with live data)

Follow this guided walkthrough to see the system in action. No demo data is pre‑seeded , you will generate it in real time.

Note: You can also run this demo directily by executing main.py file with the inputs or follow FastAPI swagger below.

Prerequisites

• Server is running (uvicorn api:app ...).

• You have a valid LLM_API_KEY in .env (for the structured part).

• If you don't have a key, skip Step 4 and just observe the Markdown + schema list.

Open Swagger UI

• Open your browser and go to: http://localhost:8000/docs

• You will see all available endpoints grouped under scrape, schemas, and system.

Health check & raw Markdown scrape

1. Expand the GET /health endpoint. Click "Execute".
   You get: {"status": "ok"} , the server is live.

2. Expand the POST /scrape endpoint. In the request body, paste:

   {
     "url": "https://news.ycombinator.com",    # replace this url with webpage you want to scrape 
     "crawl_type": "markdown",
     "magic": true
   }

   Click "Execute".
   The response returns the full Markdown content of news.ycombinator.com , no API key required.

Generate a structured schema (seed the cache)

1. Back in POST /scrape, change the body to:

   {
     "url": "https://www.bbc.com/news/articles/cwyz11475l1o",
     "crawl_type": "structured",
     "model_provider": "gemini/gemini-2.0-flash",  # configure your LLM model and below its API key
     "api_key": "YOUR_ACTUAL_API_KEY",      
     "magic": true
   }

2. Click "Execute".

   • The system fetches the BBC article HTML.

   • It calls the LLM to generate a CSS extraction schema.

   • It saves that schema to the SQLite database.

   • It runs the schema against the page and returns structured JSON (title, body, images, etc.).
     This takes 15‑30 seconds , this is the "seed" step.

View the cached schema

1. Expand the GET /schemas endpoint. Click "Execute".
   You see a list of cached schemas. The one you just created appears with:

   • domain: bbc.com

   • domain_pattern: a concrete URL (later generalised)

   • example_urls: the URL you just scraped

   • last_successful_used: timestamp

Test the cache with a similar URL

1. Go back to POST /scrape and run structured extraction on a different BBC article, e.g.:

   {
     "url": "https://www.bbc.com/news/articles/c99dym3prl1o",
     "crawl_type": "structured",
     "model_provider": "gemini/gemini-2.0-flash",
     "api_key": "YOUR_API_KEY",
     "magic": true
   }

2. Click "Execute".
   This time, the system skips the LLM, reuses the cached schema, runs the structured crawler, and returns JSON in < 5 seconds. It also automatically updates the example_urls list and touches last_successful_used.

3. (Optional) Expand GET /schemas again. You will see that the example_urls array now contains both URLs , the system is learning and generalising.

Demo complete

In 5 minutes, you have:

• Scraped raw Markdown (no AI).

• Generated a new extraction schema using an LLM (with AI).

• Cached and reused that schema on a second, similar page.

• Explored the API documentation and the database state.

Running Tests

Execute the full test suite with:

pytest tests/ -v

Tests cover:

• Database repository (CRUD, caching).

• Schema validation & similarity scoring.

• URL generalisation and pattern matching.

• Rate limiter and retry logic.

Configuration (config.py)

Key environment variables (set in .env):

Project Structure (high‑level)

.
├── api.py                # FastAPI app, MCP mounting, endpoints
├── main.py               # Orchestration entry point (for CLI / testing)
├── config.py             # Environment variables & constants
├── requirements.txt      # Python dependencies
├── core/
│   ├── crawler.py        # Crawl4AI wrappers (markdown, html, structured)
│   ├── schema_gen.py     # LLM schema generation
│   └── validators.py     # Output validation & failure scoring
├── services/
│   └── schema_service.py # Caching, empirical testing, merging logic
├── db/
│   ├── connection.py     # SQLAlchemy engine & session
│   ├── models.py         # Schema table definition
│   └── repository.py     # CRUD operations
├── utils/
│   ├── async_helpers.py  # RateLimiter, retry_async
│   ├── schema.py         # Jaccard similarity for schemas
│   └── url.py            # Domain extraction, pattern matching
└── tests/                # Unit tests (pytest)