reelminner

💡 Name note: This project's final public name is Reelminner. The Python engine class is Reelminner (see scraper.py), the CLI/GUI and MCP server are branded reelminner, and the GitHub repository is reelminner. The earlier working codename ReelSnipe has been fully retired. Other name ideas are listed in Name options.

📚 Table of Contents

• What is Reelminner

• ✨ Features

• 🧠 How it works

• 🏗️ Project architecture

• 📦 Installation

• 🚀 Quick Start

• 💻 Usage

  • Desktop GUI

  • Command Line (CLI)

  • MCP Server (for AI agents)

  • Python API

• 📊 Output format

• ⚙️ Configuration

• 🗂️ Project structure

• 🧪 Testing & QA

• 📦 Building a standalone EXE

• ⚠️ Legal & ethical disclaimer

• 🤝 Contributing

• 📄 License

• 🆘 Troubleshooting & FAQ

• 🏷️ Name

Related MCP server: Instagram Complete MCP Server

What is Reelminner

Reelminner is an open‑source toolkit that pulls structured data out of Instagram Reels and the profiles that posted them. It is built around a single, reusable engine (Reelminner) that is exposed four different ways:

Everything shares the same parsing, session, and rate‑limit logic, so results are identical no matter which front‑end you use.

✨ Features

• Multi‑source reel parsing — Reelminner reads data from several layers (embedded JSON, GraphQL responses, and a live DOM fallback) so it keeps working even when Instagram changes one of them.

• Owner profile enrichment — for every reel it can auto‑fetch the poster's username, full_name, bio, followers, is_verified, and reels_count.

• Follower count extraction — pulled via Instagram's GraphQL UserByRestrictedView / GraphQLOwnerInfo query, with a DOM fallback and pagination (handles capped follower figures like “1.2M” by scrolling the profile).

• Music metadata — reel audio music_title, music_artist, and music_id.

• Engagement metrics — views, likes, comments, and the direct video_url / thumbnail.

• Session & login management — interactive QR/login, cookie import from EditThisCookie exports, and a 24‑hour session refresh so you don't re‑login constantly.

• Concurrent scraping — a thread pool (--workers, default 3) with polite inter‑request delays (--delay, default 2s) and adaptive back‑off when Instagram throws BLOCKED / RATE_LIMITED.

• Resilient status tracking — every row carries a status code (OK, PARSED_PARTIAL, FAILED, NO_DATA, BLOCKED, RATE_LIMITED) so you know exactly what succeeded.

• Multiple export formats — CSV (default), JSON, and Excel (.xlsx via openpyxl).

• MCP server — five stable tools so an AI agent (Claude, Cursor, etc.) can scrape, check status, import cookies, stop, and export.

• Desktop GUI — built‑in dark theme, paste‑URL box, live results table, right‑click copy URL / open reel, and one‑click export.

• Tested — pytest suite + an end‑to‑end QA harness that enforces data‑quality gates.

🧠 How it works

┌────────────┐   ┌────────────┐   ┌────────────┐   ┌────────────┐
│   GUI      │   │    CLI     │   │  MCP srv   │   │  Python    │
│  gui.py    │   │ scraper.py │   │mcp_server  │   │   import   │
└─────┬──────┘   └─────┬──────┘   └─────┬──────┘   └─────┬──────┘
      └────────────────┴────────────────┴────────────────┘
                       ▼
              ┌───────────────────────┐
              │  Reelminner  │  ← the engine (scraper.py)
              │  • session / cookies   │
              │  • thread pool         │
              │  • adaptive back‑off   │
              └───────────┬───────────┘
                          ▼
              ┌───────────────────────┐
              │  parsers.py            │  ← pure extraction helpers
              │  parse_reel_page / json│
              │  parse_owner / music   │
              │  regex adapters         │
              └───────────────────────┘

1. Normalize the input URL (normalize_reel_url) so /reel/X/ and /reel/s/…/ both work.

2. Load session — apply saved cookies (sessionid, csrftoken, ds_user_id, ig_did, mid, rur) or log in.

3. Fetch & parse the reel page with a layered fallback:

   • parse_reel_page → embedded window.__additionalData / sharedData HTML JSON

   • parse_reel_json → raw GraphQL GQL response

   • parse_graphql_reel → shortcodeMedia object

   • DOM fallback → _extract_text_raw queries the live page for likes / comments / plays / followers via regex adapters.

4. Enrich owner (unless --no-profiles): fetch the profile and read followers, full_name, bio, is_verified, reels_count.

5. Respect limits: sleep delay between requests; if blocked, back off and retry.

6. Write rows to CSV / JSON / Excel with a status per row.

🏗️ Project architecture

Reelminner is a single‑engine, multi‑interface design. One core engine (Reelminner) does all the real work; the GUI, CLI, MCP server, and Python API are thin front‑ends that call into it. This keeps parsing, session handling, and rate‑limiting identical across every entry point.

                         ┌─────────────────────────────┐
        URL(s) in ──────▶│     Reelminner     │  scraper.py
                         │  ── engine / orchestrator ──  │
                         └───────┬───────────┬──────────┘
                  run scrapes    │           │  enrich owner
                                 ▼           ▼
                    ┌────────────────┐  ┌──────────────────┐
                    │   parsers.py    │  │ session + graphql│
                    │ pure extractors │  │ (followers/music)│
                    └───────┬────────┘  └─────────┬────────┘
                            └─────────┬────────────┘
                                      ▼
                            ReelData row + status
                                      ▼
                       CSV / JSON / Excel writers

Module responsibilities

Engine internals (Reelminner)

• Session layer — _SESSION_COOKIE_NAMES (sessionid, csrftoken, ds_user_id, ig_did, mid, rur); _apply_cookies(), _refresh_if_needed() (24h), login() (interactive QR), clear_session().

• Concurrency — scrape() spins up a ThreadPoolExecutor(max_workers=workers); each URL is handled by _worker → _scrape_url, which calls _gather_metadata (reel data) and optionally _gather_article (owner profile). A semaphore + _sleep() enforce politeness; status_code / retcode drive an adaptive retry/back‑off loop when Instagram returns BLOCKED / RATE_LIMITED.

• Parsing pipeline (layered fallback) — inside _gather_metadata the engine tries, in order: parse_reel_page (embedded HTML JSON) → parse_reel_json (raw GraphQL GQL) → parse_graphql_reel (shortcodeMedia) → DOM fallback via the _extract_text_html / _extract_text_raw adapters and the _PATTERNS regex list (likes/comments/plays/followers).

• Profile enrichment — get_follower_count() uses Instagram's GraphQL UserByRestrictedView / GraphQLOwnerInfo query, falling back to the DOM and paginating followers (_fetch_followers with end_cursor) when counts are capped.

• Output — rows are collected as ReelData dicts and written by write_csv (respecting csv_columns), export_json, or export_excel (needs openpyxl).

Why this layout

• Testability — all parsing lives in parsers.py with no browser dependency, so tests/test_parsers.py can assert on saved HTML/JSON fixtures.

• One source of truth — every interface shares the same Reelminner, so a fix in the engine benefits the GUI, CLI, and MCP server simultaneously.

• Safe packaging — the GUI/CLI thin shells mean the PyInstaller EXE only bundles the engine + a minimal UI, keeping the binary small.

📦 Installation

Requirements: Python 3.10+ and the Playwright browser engine.

# 1. Clone
git clone https://github.com/ilovekushgola/reelminner.git
cd reelminner

# 2. (Recommended) create a virtual environment
python -m venv .venv
.venv\Scripts\activate        # Windows
# source .venv/bin/activate   # macOS / Linux

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install the Chromium browser for Playwright
playwright install chromium

GUI only: the desktop app uses tkinter, which ships with standard Python installs. No extra package needed. The GUI is most polished on Windows.

Optional dev/test tools:

pip install -r requirements-dev.txt   # pytest, coverage

💡 Before you start: Reelminner works best with a logged‑in Instagram session — some reels and all owner/follower data require authentication. Run python scraper.py --login once (interactive QR), or import cookies exported from the EditThisCookie browser extension with python scraper.py --import-cookies cookies.json. It only reads public content you're already allowed to view.

🚀 Quick Start

# Scrape a single reel from the command line
python scraper.py "https://www.instagram.com/reel/CxXYZ123/"

# …or many reels from a file (one URL per line)
python scraper.py -f urls.txt -o export.csv

# Launch the desktop GUI
python gui.py

💻 Usage

1. Desktop GUI

python gui.py

• Click Login (optional but recommended — improves success rate).

• Paste one reel URL per line into the box (or Ctrl+A to select all).

• Drag the Workers slider, then click Scrape.

• Watch results appear in the table.

• Right‑click a row to Copy URL or Open Reel.

• Export to CSV / Excel / JSON, or Open results folder.

The last results are auto‑saved to results/_last_results.json.

2. Command Line (CLI)

python scraper.py [URL ...] [options]

# Headless, 5 workers, 1s delay, no profile enrichment
python scraper.py -f reels.txt -w 5 --delay 1 --headless --no-profiles -o out.csv

3. MCP Server (for AI agents)

Reelminner ships an MCP (Model Context Protocol) server so an AI client can drive it.

python mcp_server.py            # stdio transport

Configure your MCP client (.mcp.json is included in the repo):

{
  "mcpServers": {
    "reelminner": {
      "command": "python",
      "args": ["mcp_server.py"],
      "cwd": ".",
      "env": { "RMIN_HEADLESS": "true" }
    }
  }
}

Tools exposed (5, stable):

Environment overrides: RMIN_HEADLESS, RMIN_WORKERS, RMIN_DELAY, RMIN_WITH_PROFILES.

4. Python API

from scraper import Reelminner, write_csv

scraper = Reelminner(workers=3, delay=2.0, headless=True)
rows, report = scraper.scrape(
    ["https://www.instagram.com/reel/CxXYZ123/"],
    with_profiles=True,
)
write_csv(rows, "out.csv")

for r in rows:
    print(r["username"], r["followers"], r["likes"], r["status"])

Key members of Reelminner:

• scrape(urls, with_profiles=True) → (rows, report)

• login() — interactive login

• has_session() / save_cookies_from_file(path) / clear_session()

• write_csv(rows, path), export_json(rows, path), export_excel(rows, path)

• normalize_reel_url(url) — public helper

• csv_columns — the ordered list of output fields

• DEFAULT_STATE_FILE — default storage_state.json

📊 Output format

Each reel becomes one row. The full CSV schema (scraper.csv_columns):

⚙️ Configuration

Cookies / session

• Log in with python scraper.py --login (saves storage_state.json).

• Or export cookies from your browser via the EditThisCookie extension and run python scraper.py --import-cookies cookies.json.

Environment variables (used by MCP server & CLI defaults)

A template is provided: copy mcp.env.example → mcp.env to override MCP defaults.

🗂️ Project structure

reelminner/
├── scraper.py          # Core engine: Reelminner + CLI
├── gui.py              # Tkinter desktop application
├── parsers.py          # Pure extraction helpers (HTML/JSON/music/regex)
├── mcp_server.py       # MCP server (5 tools for AI agents)
├── theme.py            # Dark‑theme styling for the GUI
├── build_exe.py        # PyInstaller build script
├── Reelminner.spec  # PyInstaller spec (one‑file EXE)
├── run_qa.py           # End‑to‑end QA harness with data‑quality gates
├── requirements.txt    # Runtime dependencies
├── requirements-dev.txt# Dev / test dependencies
├── mcp.env.example     # MCP env template
├── .mcp.json           # MCP client configuration
├── assets/             # Icons (icon.ico)
├── docs/               # SKILL.md, E2E test/fix plan
├── skills/             # Agent skill definition
├── tests/              # pytest suite + corpus.txt
└── results/            # Scrape outputs (git‑ignored)

🧪 Testing & QA

# Unit / integration tests
pytest -q

# End‑to‑end data‑quality run (uses your saved session)
python run_qa.py                 # full run over tests/corpus.txt
python run_qa.py --quick         # 1 URL, headless, fast iteration
python run_qa.py --url <reel>    # custom single URL
python run_qa.py --report-only   # show last qa_report.json

The QA harness enforces gates such as parsed‑rate, verified‑rate, non‑empty‑rate, blocked‑rate, and max runtime, and writes results/qa/qa_report.json + qa_results.csv.

📦 Building a standalone EXE

On Windows, produce a portable .exe (no Python needed by end users):

pip install pyinstaller
python build_exe.py

Output: dist/Reelminner.exe (one‑file build via Reelminner.spec).

⚠️ Legal & ethical disclaimer

Reelminner is provided for educational and authorized/personal use only.

• Scraping Instagram may violate its Terms of Service. Use it only on content you own or are permitted to access.

• Respect rate limits (--delay, fewer --workers) and do not use it for spam, harassment, or commercial bulk extraction.

• You are responsible for how you use this tool and for complying with applicable laws (incl. GDPR / privacy regulations) in your jurisdiction.

• The authors are not affiliated with Instagram/Meta and accept no liability.

🆘 Troubleshooting & FAQ

playwright says the browser isn't installed / pages won't open → Make sure you ran both pip install -r requirements.txt and playwright install chromium. Without the Chromium download nothing will launch.

Most fields are empty, or I get BLOCKED / RATE_LIMITED → Log in (python scraper.py --login) or import cookies, then slow down: --delay 4 and fewer workers (-w 1). Instagram throttles anonymous/unauthenticated traffic hardest, so an authenticated session is the single biggest success factor.

A reel returns NO_DATA → The post may be private, deleted, or region‑locked, or Instagram served a login wall. Try again with a logged‑in session.

The GUI window won't open or fonts look wrong → The GUI uses Python's built‑in tkinter. On Windows it's most polished. On Linux/macOS install the Tk package if the window fails to launch (e.g. sudo apt install python3-tk).

ModuleNotFoundError when I run a script → You're likely outside the repo or its virtual environment. cd into the project folder and activate the venv (.venv\Scripts\activate on Windows, source .venv/bin/activate on macOS/Linux) before running python scraper.py.

How do I scrape lots of reels at once? → Put one URL per line in a text file and run python scraper.py -f urls.txt -o out.csv.

Can an AI agent use this? → Yes — run python mcp_server.py and point any MCP client (Claude Desktop, Cursor, etc.) at the included .mcp.json. See MCP Server.

🤝 Contributing

1. Fork the repo and create a feature branch.

2. pip install -r requirements-dev.txt

3. Add/adjust tests in tests/; run pytest and python run_qa.py --quick.

4. Open a pull request describing the change and the QA result.

📄 License

Released under the MIT License — see LICENSE.

🏷️ Name

The project's final public name is Reelminner ("Reel miner"). Earlier internal codenames have been retired. If you fork it you can rename it to anything you like — just update the title in gui.py and this README.