🌐 InfinityScrape MCP: World-Class Web Scraping, Dynamic SPA Rendering & 25-Tool OSINT Intelligence Suite

InfinityScrape MCP is a standalone, production-grade Model Context Protocol (MCP) server engineered to provide AI models (Open WebUI, Claude 3.7, DeepSeek-R1/V3, Antigravity AI, Cursor, LM Studio) with unlimited, high-speed, anti-bot resilient web scraping, dynamic SPA rendering, DuckDuckGo web search, Wayback Machine time-travel, instant YouTube transcription, and precision OSINT / GEOINT location intelligence.

📑 Table of Contents

• 🌟 Why InfinityScrape MCP?

• ⚡ Competitive Comparison

• 🏗️ Architectural Overview

• 🚀 Quick Start & 1-Click Installation

• 🔌 AI Client Integration (Open WebUI, Claude, Cursor)

• 🛠️ Complete 25-Tool Reference Catalog

  • 1. Anti-Bot Web Scraping, Dynamic SPAs & Crawlers (6 Tools)

  • 2. Real-Time Web Search, Research & Archive OSINT (4 Tools)

  • 3. Media, Document & RAG Extraction (4 Tools)

  • 4. Deep Public OSINT & Entity Reconnaissance (4 Tools)

  • 5. Infrastructure, Domain & Network Reconnaissance (5 Tools)

  • 6. Precision GEOINT & Location Intelligence (2 Tools)

• 💻 Command-Line Interface (CLI)

• 🧪 Running Automated Tests

• 📄 License & Authors

Related MCP server: FineData MCP Server

🌟 Why InfinityScrape MCP?

Standard web scrapers often fail on modern websites due to Cloudflare challenges, heavy client-side JavaScript rendering, intrusive cookie consent modals, and rate limits. InfinityScrape solves these problems out-of-the-box:

1. Dual-Engine Scraping Architecture:

   • Fast TLS Engine (primp + httpx): Mimics real Chrome/Safari browser TLS/JA3 fingerprints and HTTP/2 headers to bypass Cloudflare and Akamai challenges in <100ms.

   • Dynamic Headless Browser (Playwright Chromium): Renders complex SPAs (React, Vue, Next.js, Angular), performs infinite scrolling, clicks elements, and executes custom JavaScript.

2. Singleton BrowserPool (~1.8s SPA Renders):

   • Persistent Chromium process with ephemeral context isolation that completely eliminates cold-start latency and avoids memory leaks.

3. Military-Grade Anti-Bot Evasions (100% Undetectable):

   • Cleanly deletes navigator.webdriver from prototype (get: () => undefined).

   • Authentic WebGL Vendor & Renderer spoofing (Google Inc. (NVIDIA) & NVIDIA GeForce RTX 3080).

   • Hardware concurrency (8 cores), device memory (8GB), and authentic window.chrome runtime emulation.

   • Realistic Chrome PDF Viewer plugins and NaCl mimeTypes.

4. 3-Tier Multi-Engine Search Failover (Anti-429 Resilience):

   • Cascades automatically from DuckDuckGo API ➡️ DuckDuckGo HTML Lite ➡️ Bing HTML Fallback. Zero API keys, zero 429 rate-limits.

5. Cloudflare Turnstile & Interstitial Auto-Solver:

   • Detects Turnstile challenge iframes and executes automated coordinate jitter to bypass interstitials.

6. Network-Level Ad & Tracker Elimination:

   • Intercepts and aborts network calls to 35+ ad networks and tracking scripts (doubleclick, criteo, outbrain, google-analytics) before they download, cutting page load time by ~300% and memory usage by 70%.

   • Automatically detects and decomposes OneTrust, Cookiebot, and sticky overlay popups.

7. Wayback Machine Time-Travel:

   • Query internet archive history for any URL across custom date ranges to track competitor pricing changes, deleted pages, and historical copy.

8. Zero-GPU Instant YouTube Transcriber:

   • Extracts complete video/shorts/live transcripts with timestamps ([MM:SS]) in <300ms directly via HTTP streams without downloading video or requiring local GPU Whisper models.

9. Deep Recursive Documentation Crawler:

   • Asynchronous Breadth-First-Search (BFS) crawler with domain locking and path prefix filtering to aggregate entire documentation trees into unified Markdown.

10. State-of-the-Art Public OSINT & GEOINT Reconnaissance:

    • Multi-Signal Confidence Scoring (0% - 100%): Evaluates Name + City + Street + PIN + Org + Role correlation to rank discovered dossiers.

    • 25+ Global Platform Scanners: Scans GitHub, GitLab, StackOverflow, Kaggle, HuggingFace, LeetCode, Codeforces, Dev.to, Medium, Substack, Google Scholar, ResearchGate, Reddit, etc.

    • OpenStreetMap GEOINT: Resolves global addresses down to street/postcode level with GPS coordinates and administrative boundaries.

11. SQLite Persistent Caching Layer:

    • In-memory and SQLite-backed local cache for instant 0ms responses on repeat lookups with configurable TTL.

⚡ Competitive Comparison

🏗️ Architectural Overview

                      ┌─────────────────────────────────────────────────────────┐
                      │    AI Clients: Open WebUI / Claude Desktop / Cursor     │
                      └────────────────────────────┬────────────────────────────┘
                                                   │
                  ┌────────────────────────────────┴────────────────────────────────┐
                  │                                                                 │
                  ▼                                                                 ▼
      [OpenAPI Bridge (Port 8000)]                                     [Stdio JSON-RPC 2.0 Server]
      FastAPI /docs & /openapi.json                                             (server.py)
                  │                                                                 │
                  └────────────────────────────────┬────────────────────────────────┘
                                                   │
                                                   ▼
            ┌──────────────────────┬──────────────────────┬──────────────────────┐
            ▼                      ▼                      ▼                      ▼
     [Fast TLS Engine]     [Playwright Engine]     [OSINT / GEOINT]     [Search & Media]
     • primp JA3/TLS       • Stealth Chromium      • 25+ Platform       • DuckDuckGo Search
     • HTTP/2 Headers      • Ad/Tracker Blocker      Scanners           • Wayback Snapshots
     • <100ms Execution    • Infinite Scroll       • OpenStreetMap      • YouTube (<300ms)
                           • Auto-Dismiss CMPs     • Reverse Geocoding  • Remote PDF Parser
                                                   • Match Confidence
                                                   │
                                                   ▼
                                    ┌──────────────────────────────┐
                                    │ SQLite Caching Layer (0ms)   │
                                    └──────────────────────────────┘

🚀 Quick Start & 1-Click Installation

1. Automated Setup

# Clone the repository
git clone https://github.com/virajverse/infinity-scraper.git
cd infinity-scraper

# Create virtual environment & install
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e .
playwright install chromium

2. Launch FastAPI Bridge (Port 8000)

python openapi_bridge.py

• Interactive Swagger Docs: http://127.0.0.1:8000/docs

• OpenAPI 3.1.0 Schema: http://127.0.0.1:8000/openapi.json

🔌 AI Client Integration

1. Open WebUI (FastAPI Bridge on Port 8000)

1. Ensure the bridge is running (python openapi_bridge.py).

2. In Open WebUI, navigate to Workspace -> Tools -> Add Tool.

3. Import from URL: http://127.0.0.1:8000/openapi.json or use infinity_scraper_suite.

4. All 25 tools are instantly accessible to your agents!

2. Antigravity AI / Claude Desktop (Native Stdio)

Add to your mcp_config.json:

{
  "mcpServers": {
    "infinity-scraper": {
      "command": "python",
      "args": ["-m", "infinity_scraper.server"],
      "env": {
        "PYTHONUNBUFFERED": "1"
      }
    }
  }
}

🛠️ Complete 25-Tool Reference Catalog

1. Anti-Bot Web Scraping, Dynamic SPAs & Crawlers (6 Tools)

2. Real-Time Web Search, Research & Archive OSINT (4 Tools)

3. Media, Document & RAG Extraction (4 Tools)

4. Deep Public OSINT & Entity Reconnaissance (4 Tools)

5. Infrastructure, Domain & Network Reconnaissance (5 Tools)

6. Precision GEOINT & Location Intelligence (2 Tools)

💻 Command-Line Interface (CLI)

InfinityScrape provides a built-in CLI for quick terminal testing:

# Scrape a webpage into Markdown
infinity-scrape scrape "https://news.ycombinator.com" --format markdown

# Search DuckDuckGo from the terminal
infinity-scrape search "Generative Engine Optimization 2026" --limit 5

# Extract YouTube Transcript
infinity-scrape youtube "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

# OSINT Persona Lookup
infinity-scrape osint --name "Linus Torvalds" --platforms github,gitlab

🧪 Running Automated Tests

# Run unit and integration tests
pytest tests/ -v

📄 License & Authors

• Author: Viraj (Founder & CEO, Taliyo Technologies)

• License: MIT License. See LICENSE for details.