📦 APKPipe

Automated APK & RSS release pipeline pairing Real-Debrid un-restricting, headless JDownloader fallback, Nextcloud storage ingestion, Apprise alerts, Web Dashboard, and native MCP Server.

📖 Table of Contents

• Overview

• Architecture

• Key Features

• Quick Start

  • Docker Compose Deployment

  • Environment Variables

• Download Engines & Resolvers

  • Tier 1: Real-Debrid API

  • Tier 2: Headless JDownloader 2

  • Tier 3: Playwright Scraper Client

• Homelab Integrations

  • Nextcloud Storage & OCC Auto-Scanning

  • Apprise & Ntfy Push Notifications

• Model Context Protocol (MCP) Server

  • Available MCP Tools

  • AI Agent Configuration

• REST API Reference

• Web Dashboard UI

• Development & Testing

• License

🌟 Overview

Maintaining an up-to-date, self-hosted Android application library on Nextcloud or local network storage traditionally requires tedious manual labor: browsing release forums (such as Mobilism), checking version strings, bypassing multi-hoster mirror links (Rapidgator, Uploady, DropGalaxy), unzipping archives, renaming APKs, moving files to proper directories, triggering Nextcloud file indexing, and tracking release history.

APKPipe completely automates this lifecycle end-to-end:

1. Monitors RSS feeds periodically for new application releases.

2. Matches releases against a customizable Watchlist with regex patterns, releaser whitelists ([Balatan], [derrin], [RockMODS]), and SemVer version gating.

3. Resolves download links via a 3-tier strategy (Real-Debrid API $\rightarrow$ JDownloader 2 $\rightarrow$ Playwright Scraper).

4. Downloads and unpacks archives (.zip, .rar, .7z, .tar) to extract APK binaries.

5. Organizes binaries into clean, standardized directory structures ({DOWNLOAD_DIR}/{AppName}/{AppName} v{Version} [{Releaser}].apk).

6. Dispatches Nextcloud OCC scan commands (occ files:scan) so files immediately appear in your mobile Nextcloud app.

7. Pushes rich status alerts to Discord, Telegram, Pushover, or Ntfy via Apprise.

8. Exposes native MCP Server tools and a Web Dashboard for human management and autonomous AI agent operations.

📐 Architecture

flowchart TD
    subgraph Sources["Feeds & Control Interfaces"]
        RSS["Mobilism RSS Feeds\n(Apps, Games, Custom)"]
        WebUI["FastAPI Web Dashboard\n(Alpine.js + Tailwind)"]
        MCP["MCP Server Tools\n(AI Agents / mcp-router)"]
    end

    subgraph APKPipeCore["APKPipe Core Engine"]
        Scheduler["AsyncIO / Periodic Poller"]
        Parser["Feed & Post Parser"]
        Matcher["Watchlist Matcher\n(Regex, Releaser Filter, SemVer)"]
        DB[("SQLite Database\n(Watchlist, Feeds, Tasks, History)")]
        Resolver["Tiered Link Resolver"]
        Downloader["Stream Downloader & Extractor"]
        Organizer["File Organizer & Sanitizer"]
        Notifier["Apprise / Ntfy Client"]
    end

    subgraph Resolvers["Download Engines"]
        RD["Real-Debrid API (Tier 1)\n(/unrestrict/link)"]
        JD["Headless JDownloader 2 (Tier 2)\n(MyJDownloader API / Watch Dir)"]
        Scraper["playwright-csharp-scraper (Tier 3)\n(Captcha / Browser Evaluator)"]
    end

    subgraph StorageNotify["Homelab Storage & Alerts"]
        NextcloudFS["Nextcloud Storage\n(/downloads/{AppName}/...)"]
        AppriseAPI["Apprise API (8000) / Ntfy"]
        NextcloudOCC["Nextcloud OCC Scan\n(docker exec occ files:scan)"]
    end

    RSS --> Scheduler
    Scheduler --> Parser
    Parser --> Matcher
    Matcher <--> DB
    WebUI <--> DB
    MCP <--> DB
    Matcher --> Resolver

    Resolver -->|RD Supported Link| RD
    Resolver -->|RD Unsupported / Direct| JD
    Resolver -->|JS / Captcha Protected| Scraper

    RD --> Downloader
    JD --> Downloader
    Scraper --> Downloader
    Downloader --> Organizer
    Organizer --> NextcloudFS
    Organizer --> NextcloudOCC
    Organizer --> Notifier
    Notifier --> AppriseAPI

✨ Key Features

• 🔄 Autonomous Feed Ingestion: Periodic polling of multiple RSS / ATOM feed sources with deduplication.

• 🎯 Fine-Grained Matcher: Target specific apps by name or regex, require minimum version thresholds, and filter by trusted releaser tags.

• ⚡ Multi-Tier Link Unrestricting:

  • Tier 1 (Real-Debrid): Direct high-speed API un-restricting for 50+ filehosters.

  • Tier 2 (JDownloader 2): Headless MyJDownloader cloud API or local folder linkgrabber.

  • Tier 3 (Playwright Scraper): Headless browser scraper fallback for dynamic landing pages.

• 📦 Automated Archive Extraction: Transparently handles .zip, .rar, .7z, and .tar.gz archives, extracting nested .apk binaries.

• 📂 Nextcloud OCC Integration: Seamless placement into Nextcloud user directories with automated occ files:scan invocation.

• 🔔 Multi-Channel Homelab Notifications: Push notifications via Apprise API and Ntfy with release metadata, file sizes, and download duration.

• 🤖 Native Model Context Protocol (MCP): 8 built-in tools compatible with Claude Desktop, Cursor, and mcp-router.

• 💻 Responsive Web Dashboard: Clean dark/light theme built with Alpine.js and Tailwind CSS.

🚀 Quick Start

Docker Compose Deployment

The recommended way to run APKPipe in production is using Docker Compose.

1. Create a docker-compose.yaml file:

services:
  apkpipe:
    image: ghcr.io/spelech/apkpipe:latest
    container_name: apkpipe
    restart: unless-stopped
    ports:
      - "8429:8000"
    environment:
      - APKPIPE_APP_NAME=APKPipe
      - APKPIPE_HOST=0.0.0.0
      - APKPIPE_PORT=8000
      - APKPIPE_DATABASE_URL=sqlite+aiosqlite:////data/apkpipe.db
      - APKPIPE_DOWNLOAD_DIR=/downloads
      - APKPIPE_STAGING_DIR=/data/staging
      - APKPIPE_POLL_INTERVAL_SECONDS=900
      - APKPIPE_REAL_DEBRID_API_TOKEN=your_real_debrid_token_here
      - APKPIPE_NEXTCLOUD_URL=http://nextcloud:80
      - APKPIPE_NEXTCLOUD_OCC_COMMAND=docker exec nextcloud-aio-nextcloud sudo -u www-data php /var/www/html/occ files:scan --path="{path}"
      - APKPIPE_APPRISE_URL=http://apprise:8000/notify/apprise
      - APKPIPE_NTFY_TOPIC=apkpipe-alerts
    volumes:
      - ./data:/data
      - /drives/storage/Nextcloud/APKs:/downloads
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      - "caddy=apk.wileyriley.com"
      - "caddy.reverse_proxy={{upstreams 8000}}"
      - "caddy.import=tinyauth"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 15s
    networks:
      - default
      - caddy

networks:
  default:
    name: apkpipe_net
  caddy:
    external: true

2. Launch the stack:

docker compose up -d

3. Open your browser to http://localhost:8429 (or https://apk.wileyriley.com if routed via Caddy).

Environment Variables

⚡ Download Engines & Resolvers

Tier 1: Real-Debrid API

APKPipe natively authenticates with Real-Debrid's REST API /unrestrict/link. When a forum post contains mirrors (Rapidgator, Uploady, DropGalaxy, Mega, Katfile, etc.), Real-Debrid un-restricts the link into a high-speed, direct CDN URL.

To configure Real-Debrid:

1. Generate an API token at real-debrid.com/apitoken.

2. Set APKPIPE_REAL_DEBRID_API_TOKEN=your_token in your environment or via the Web UI Settings tab.

Tier 2: Headless JDownloader 2

For hosters not supported by Real-Debrid or requiring specialized decrypters, APKPipe connects to headless JDownloader 2 instances via:

• MyJDownloader Cloud API: Submits download packages directly to your connected device.

• Linkgrabber Watch Directory: Drops .crawljob instructions into a shared folder.

Tier 3: Playwright Scraper Client

When download landing pages are protected by JavaScript redirection or cloud protection, APKPipe dispatches page evaluation tasks to playwright-csharp-scraper (http://scraper:8080), which evaluates the DOM and extracts final direct download mirrors.

🔗 Homelab Integrations

Nextcloud Storage & OCC Auto-Scanning

APKPipe formats and places files into your Nextcloud storage directory:

/downloads/
├── Spotify/
│   └── Spotify v8.9.18.534 [Balatan].apk
└── Nova Launcher/
    └── Nova Launcher v8.0.18 [Prime].apk

To inform Nextcloud of newly downloaded files without waiting for Nextcloud background jobs, APKPipe executes occ files:scan:

• Docker Exec (Recommended): Pass Docker socket /var/run/docker.sock and set:

  APKPIPE_NEXTCLOUD_OCC_COMMAND=docker exec nextcloud-aio-nextcloud sudo -u www-data php /var/www/html/occ files:scan --path="{path}"

• CLI / Host Exec: Direct command invocation if APKPipe runs directly on the host.

Apprise & Ntfy Push Notifications

APKPipe emits structured event notifications on:

• 🔍 feed_matched: Release discovered matching a watchlist rule.

• ⬇️ download_started: Unrestricting link and initiating streaming download.

• ✅ download_completed: Archive extracted, file placed, OCC scan complete.

• ❌ download_failed: Error details and failure cause.

Supported notification endpoints:

• Apprise API: http://apprise:8000/notify/apprise

• Ntfy: https://ntfy.sh/your-topic or self-hosted Ntfy server.

🤖 Model Context Protocol (MCP) Server

APKPipe exposes a native MCP 2024-11-05 Server allowing AI assistants (Claude Desktop, Cursor, Antigravity mcp-router) to manage watchlists, inspect feeds, trigger polls, and download releases autonomously.

Available MCP Tools

AI Agent Configuration

Add APKPipe to your claude_desktop_config.json or mcp-router:

{
  "mcpServers": {
    "apkpipe": {
      "command": "python",
      "args": ["-m", "apkpipe.mcp.server"],
      "env": {
        "APKPIPE_DATABASE_URL": "sqlite+aiosqlite:////data/apkpipe.db",
        "APKPIPE_DOWNLOAD_DIR": "/downloads"
      }
    }
  }
}

Or connect via Streamable HTTP / SSE endpoint:

http://localhost:8429/mcp/sse

🌐 REST API Reference

The interactive OpenAPI / Swagger UI is available at http://localhost:8429/docs.

Watchlist Endpoints

• GET /api/watchlist - List all watchlist items with optional search/filtering.

• POST /api/watchlist - Add a new application to watchlist.

• GET /api/watchlist/{id} - Retrieve a specific watchlist entry.

• PUT /api/watchlist/{id} - Update a watchlist item.

• DELETE /api/watchlist/{id} - Remove a watchlist entry.

Feed Sources Endpoints

• GET /api/feeds - List all configured RSS feed sources.

• POST /api/feeds - Add a new RSS feed source.

• POST /api/feeds/{id}/poll - Trigger immediate polling of a specific feed.

• POST /api/feeds/poll-all - Trigger polling across all active feed sources.

Downloads & History Endpoints

• GET /api/downloads/queue - List active tasks in the download pipeline.

• GET /api/downloads/history - List completed, failed, and historic downloads.

• POST /api/downloads/manual - Trigger an immediate download from direct/mirror URL.

• POST /api/downloads/{task_id}/retry - Retry a failed download task.

System & Health Endpoints

• GET /health - System health check probe.

• GET /api/settings - Retrieve current application settings.

• POST /api/settings - Update runtime configuration.

🖥️ Web Dashboard UI

APKPipe includes a built-in Web UI accessible on port 8429:

• Dashboard (/): Overview metrics, active polling status, recent downloads, and system storage.

• Watchlist (/watchlist): Add, edit, filter, and remove monitored applications.

• Feed Sources (/feeds): Manage RSS URLs, polling frequencies, and trigger manual syncs.

• History (/history): Audit trail of completed downloads, execution times, file sizes, and error traces.

• Settings (/settings): Configure Real-Debrid tokens, Nextcloud OCC paths, and notification hooks.

🛠️ Development & Testing

Local Setup

# Clone repository
git clone https://github.com/spelech/apkpipe.git
cd apkpipe

# Create and activate Python virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install development dependencies and editable package
pip install -r requirements-dev.txt
pip install -e .

Running Test Suite & Coverage

Enforcing ≥ 80% line coverage:

# Run pytest with coverage report
pytest --cov=src/apkpipe --cov-report=term-missing tests/

Local Server Launch

# Start FastAPI application with live reloading
uvicorn apkpipe.main:app --host 0.0.0.0 --port 8000 --reload

📄 License

This project is licensed under the MIT License. See LICENSE for details.