🌐 InfinityScrape MCP: World-Class Web Scraping & Deep OSINT Intelligence Suite

InfinityScrape MCP is a standalone, production-grade Model Context Protocol (MCP) server engineered to provide AI models (LM Studio, Claude Desktop, Cursor, Open WebUI, Antigravity AI) with unlimited, high-speed, anti-bot resilient web scraping, dynamic SPA rendering, instant YouTube transcription, and precision OSINT / GEOINT location intelligence.

📑 Table of Contents

• 🌟 Why InfinityScrape MCP?

• ⚡ Competitive Comparison

• 🏗️ Architectural Overview

• 🚀 Quick Start & 1-Click Installation

• 🔌 AI Client Integration (LM Studio, Claude, Cursor)

• 🛠️ Complete 16-Tool Reference Guide

  • 1. Web Scraping & Content Crawling

  • 2. Media, Video & Document Parsers

  • 3. Deep Public OSINT & Entity Reconnaissance

  • 4. Technical, Domain & Network Intelligence

• 🧠 Autonomous AI Agent Playbook (SKILL.md)

• 💻 Command-Line Interface (CLI)

• 🧪 Running Automated Tests

• 📄 License & Authors

Related MCP server: FineData MCP Server

🌟 Why InfinityScrape MCP?

Standard web scrapers often fail on modern websites due to Cloudflare challenges, heavy client-side JavaScript rendering, intrusive cookie consent modals, and rate limits. InfinityScrape solves these problems out-of-the-box:

1. Dual-Engine Architecture:

   • Fast TLS Engine (primp + httpx): Mimics real Chrome/Safari browser TLS/JA3 fingerprints and HTTP/2 headers to bypass Cloudflare and Akamai challenges in <100ms.

   • Dynamic Headless Browser (Playwright Chromium): Renders complex SPAs (React, Vue, Next.js, Angular), performs infinite scrolling, clicks elements, and executes custom JavaScript.

2. Network-Level Ad & Tracker Elimination:

   • Intercepts and aborts network calls to 35+ ad networks and tracking scripts (doubleclick, criteo, outbrain, google-analytics) before they download, cutting page load time by ~300% and memory usage by 70%.

   • Automatically detects and decomposes OneTrust, Cookiebot, and sticky overlay popups.

3. Zero-GPU Instant YouTube Transcriber:

   • Extracts complete video/shorts/live transcripts with timestamps ([MM:SS]) in <300ms directly via HTTP streams without downloading video or requiring local GPU Whisper models.

4. Deep Recursive Documentation Crawler:

   • Asynchronous Breadth-First-Search (BFS) crawler with domain locking and path prefix filtering to aggregate entire documentation trees into unified Markdown.

5. State-of-the-Art Public OSINT & GEOINT Reconnaissance:

   • Multi-Signal Confidence Scoring (0% - 100%): Evaluates Name + City + Street + PIN + Org + Role correlation to rank discovered dossiers.

   • 25+ Global Platform Scanners: Scans GitHub, GitLab, StackOverflow, Kaggle, HuggingFace, LeetCode, Codeforces, Dev.to, Medium, Substack, Google Scholar, ResearchGate, Reddit, etc.

   • OpenStreetMap GEOINT: Resolves global addresses down to street/postcode level with GPS coordinates and administrative boundaries.

6. SQLite Persistent Caching Layer:

   • In-memory and SQLite-backed local cache for instant 0ms responses on repeat lookups with configurable TTL.

⚡ Competitive Comparison

🏗️ Architectural Overview

                      ┌────────────────────────────────────────────────┐
                      │    AI Client (LM Studio / Claude / Cursor)     │
                      └───────────────────────┬────────────────────────┘
                                              │ JSON-RPC 2.0 (Stdio)
                                              ▼
                      ┌────────────────────────────────────────────────┐
                      │          InfinityScrape MCP Server             │
                      │                  (server.py)                   │
                      └───────┬────────────────┬───────────────┬───────┘
                              │                │               │
            ┌─────────────────┴─┐     ┌────────┴────────┐    ┌─┴────────────────┐
            ▼                   ▼     ▼                 ▼    ▼                  ▼
     [Fast TLS Engine]     [Playwright Engine]   [OSINT / GEOINT]   [Media & PDF Engines]
     • primp JA3/TLS       • Stealth Chromium    • 25+ Platform     • YouTube (<300ms)
     • HTTP/2 Stealth      • Network Ad Blocker    Scanners         • Remote PDF Stream
     • <100ms Execution    • Infinite Scroll     • OpenStreetMap    • Table Markdownify
                           • Auto-Dismiss CMPs   • Match Confidence
                                    │
                                    ▼
                      ┌────────────────────────────────┐
                      │ SQLite Caching Layer (0ms TTL) │
                      └────────────────────────────────┘

🚀 Quick Start & 1-Click Installation

Prerequisites

• Python 3.10, 3.11, or 3.12+ installed.

• Windows, macOS, or Linux.

1-Click Setup:

On Windows:

Double-click install.bat or run in PowerShell:

.\install.bat

On Linux / macOS:

chmod +x install.sh
./install.sh

Manual Setup (Any Platform):

# 1. Create virtual environment
python -m venv .venv

# 2. Activate virtual environment
# Windows: .venv\Scripts\activate | Linux/Mac: source .venv/bin/activate

# 3. Install requirements & Playwright browser
pip install -r requirements.txt
playwright install chromium

🔌 AI Client Integration

1. LM Studio (v0.3+)

Go to Settings ➔ Developer ➔ MCP Servers ➔ Edit Config and add:

{
  "mcpServers": {
    "infinity-scraper": {
      "command": "C:/path/to/infinity-scraper/.venv/Scripts/python.exe",
      "args": [
        "-m",
        "infinity_scraper.server"
      ],
      "cwd": "C:/path/to/infinity-scraper",
      "env": {
        "PYTHONUNBUFFERED": "1"
      }
    }
  }
}

2. Claude Desktop

Edit %APPDATA%\Claude\claude_desktop_config.json (Windows) or ~/Library/Application Support/Claude/claude_desktop_config.json (macOS):

{
  "mcpServers": {
    "infinity-scraper": {
      "command": "C:/path/to/infinity-scraper/.venv/Scripts/python.exe",
      "args": [
        "-m",
        "infinity_scraper.server"
      ]
    }
  }
}

3. Cursor IDE

In Cursor Settings ➔ Features ➔ MCP Servers ➔ Add New MCP Server:

• Name: infinity-scraper

• Type: command

• Command: C:/path/to/infinity-scraper/.venv/Scripts/python.exe -m infinity_scraper.server

🛠️ Complete 23-Tool Reference Guide

1. Web Scraping & Content Crawling

2. Media, Social, Video & Document Parsers

3. Deep Public OSINT & Entity Reconnaissance

4. Technical, Domain & Network Intelligence

🧠 Autonomous AI Agent Playbook

InfinityScrape includes an advanced Cognitive Reasoning Framework (skills/infinity-scraper/SKILL.md) that teaches autonomous AI agents how to:

• Dynamically deconstruct user prompts into search and location clues.

• Multi-tool chain across tools (e.g. Search ➔ Filter ➔ Batch Scrape or Geocode ➔ Locality Dork ➔ Profile Extraction).

• Auto-escalate from fast TLS to Playwright headless browser when encountering dynamic React single-page apps.

👉 Read the full agent playbook: skills/infinity-scraper/SKILL.md

💻 Command-Line Interface (CLI)

You can also use InfinityScrape directly from your terminal:

# Scrape a URL to Markdown
python -m infinity_scraper.cli scrape "https://example.com"

# Scrape dynamic SPA with infinite scroll
python -m infinity_scraper.cli scrape "https://news.ycombinator.com" --browser --scroll 3

# Live search and auto-scrape top results
python -m infinity_scraper.cli search "Quantum computing breakthroughs" --scrape --max 4

# Crawl documentation tree
python -m infinity_scraper.cli crawl "https://docs.python.org/3/library/asyncio.html" --pages 5 --depth 2

# Extract remote PDF
python -m infinity_scraper.cli pdf "https://example.com/report.pdf" --pages 10

🧪 Running Automated Tests

Run the comprehensive unit and integration test suite:

python -m tests.test_scraper

Test Coverage:

• ✅ Fast TLS Impersonator

• ✅ Playwright Dynamic Browser

• ✅ DuckDuckGo Live Search

• ✅ HTML Table to Markdown Converter

• ✅ Recursive BFS Documentation Crawler

• ✅ Zero-GPU YouTube Transcript Extraction

• ✅ OSINT SSL, IP Intel & 25+ Platform Presence Check

📄 License & Copyright Protection

This project is licensed under the MIT License (with Mandatory Attribution & DMCA Enforcement).

• Author / Creator: VirajVerse

• Repository: https://github.com/virajverse/infinity-scraper-mcp

• See LICENSE for complete legal terms.