# 🐧 TuxScout (`tuxscout`)

**Cross-Platform AI Agent Reconnaissance, Media & Deep Research Suite (Linux • Windows • macOS)**

[![npm version](https://img.shields.io/npm/v/tuxscout.svg?style=flat&color=cb3837)](https://www.npmjs.com/package/tuxscout)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
[![Node.js >= 18](https://img.shields.io/badge/Node.js-%3E%3D18.0.0-brightgreen.svg)](https://nodejs.org/)
[![Platform](https://img.shields.io/badge/Platform-Linux%20%7C%20Windows%20%7C%20macOS-orange.svg)](#quick-installation)
[![MCP Ready](https://img.shields.io/badge/MCP-Stdio%20Server-purple.svg)](#mcp-model-context-protocol-server-tx-mcp)

**TuxScout** is a high-performance, ultra-low-memory AI Agent Toolkit engineered for Linux servers, Docker containers, Windows workstations, and macOS environments. It provides:
- **Global CLI Binaries** (`tuxscout`, `tx-search`, `tx-imdb`, `tx-pdf`, `tx-skill`, `tx-images`, `tx-social`, `tx-news`, `tx-extract`, `tx-fetch`, `tx-browser`, `tx-yt`, `tx-research`, `tx-serve`, `tx-mcp`)
- **Built-in AI Agent Skill (`SKILL.md`)** for Claude Desktop, Antigravity, Cursor, Cline, Windsurf, OpenClaw, and custom LLM agents (`tx-skill` / `tuxscout skill`)
- **Universal Proxy IP Support** (`HTTP`, `HTTPS`, `SOCKS5` / `SOCKS5h`, credentials, rotation pools, `--proxy` flag, `TUXSCOUT_PROXY` env var) across **all** tools
- **Model Context Protocol (MCP) Server** (`tx-mcp` — zero-dependency JSON-RPC 2.0 over stdio)
- **HTTP REST API Server** (`tx-serve` — drop-in compatible with Delphos web/image gateways)
- **TypeScript / Node.js SDK** (`import { webSearch, searchImdb, searchPdf, imageSearch, socialIntel, newsIntel, smartExtract, fetchMarkdown, fetchWithPuppeteer, youtubeTool, deepResearch, setGlobalProxy } from 'tuxscout'`)

---

## ⚡ Quick Installation

### 1. Official NPM Package (Global Install)
```bash
npm install -g tuxscout
tuxscout setup
```
Or run directly without installing:
```bash
npx tuxscout search "Gemini 4 release notes"
```

### 2. Windows PowerShell One-Liner (`irm`)
```powershell
irm https://raw.githubusercontent.com/SuparvaCode/tuxscout/main/install.ps1 | iex
```

### 3. Linux (Ubuntu/Debian/Fedora/Arch) & macOS One-Liner (`curl`)
```bash
curl -fsSL https://raw.githubusercontent.com/SuparvaCode/tuxscout/main/install.sh | bash
```

### 4. Direct Git Install
```bash
npm install -g git+https://github.com/SuparvaCode/tuxscout.git
tuxscout setup
```

### 4. Clone & Local Install
```bash
git clone https://github.com/SuparvaCode/tuxscout.git
cd tuxscout
npm install && npm run build && npm install -g .
tuxscout setup
```

---

## 🛠️ Included Tools & Architecture

### 1. Multi-Engine Internet Web Search & Advanced Operators (`tx-search`, `tx-imdb`, `tx-pdf`)
- **Engines:** DuckDuckGo HTML, DuckDuckGo Lite, Brave Search, Yahoo Search (with manual `/_bv` redirect & `YBV` cookie jar persistence), Wikipedia API, Bing, **IMDb**, **ArXiv**, and **OpenAlex**.
- **Advanced Search Operators:**
  - Vertical Search Modes (`--type`): `'general' | 'pdf' | 'imdb' | 'academic' | 'code' | 'docs' | 'books'`
  - Structured Filters: `--site` (single or comma-separated domains), `--exclude-site`, `--filetype`, `--pdf`, `--imdb`, `--intitle`, `--inurl`, `--exact`, `--exclude`, `--time` (`day | week | month | year | any`), and `--region`.
- **Dedicated IMDb Search & Rich Dossier Extraction (`tx-imdb`):**
  - Searches IMDb for movies, TV series, actors, and directors (`tx-imdb "Inception"`), or extracts full movie/TV/person dossiers (`tx-imdb tt1375666 --extract` or `tx-extract imdb:Inception`) including rating, votes, genres, runtime, release date, directors, cast, plot, keywords, awards, and filmography.
- **Dedicated Multi-Engine PDF Search & Zero-Dependency PDF Parser (`tx-pdf`):**
  - Combines web `filetype:pdf` search with direct **ArXiv** & **OpenAlex** open-access PDF feeds (`tx-pdf "Attention Is All You Need"`).
  - Extracts full text and metadata (`Title`, `Author`, `PageCount`) directly from remote `.pdf` URLs using Node's native `zlib` stream decompressor (`tx-pdf <pdf-url> --extract` or `tx-extract pdf:<url>`) with zero external PDF dependencies.
- **4-Layer Relevance & Quality Filter:**
  1. **Intent Classification:** Distinguishes `news` / release queries from `evergreen` and `navigational` queries.
  2. **Subject vs. Freshness Separation:** Prevents generic news portals from hijacking queries containing words like `"latest"` or `"upcoming"`.
  3. **Landing-Page Rejection:** Filters out bare marketing homepages when the query asks for news, releases, or changelogs.
  4. **Engine-Level Relevance Verdict:** Detects off-topic SERPs (`ratio < 0.34`) and automatically fails over to the next engine.
- **Multi-Query Parallel Execution:** Accepts multiple `--query` angles, executes them concurrently, and merges results round-robin by rank.

### 10. Universal Proxy IP & Rotation Pool Support (`--proxy`)
- Works across **all** tools: Web Search, IMDb, PDF, Image Search & Download, Social Intel, News Intel, Smart Extract, Markdown Fetch, Puppeteer Headless Browser (`--proxy-server` + `page.authenticate`), YouTube (`yt-dlp --proxy`), and Deep Research.
- **Flexible Proxy Formats (Zero Extra Dependencies):**
  - `http://host:port` or `http://user:pass@host:port`
  - `https://host:port` or `https://user:pass@host:port`
  - `socks5://host:port` or `socks5h://user:pass@host:port` (native RFC 1928/1929 SOCKS5 tunnel with remote DNS)
  - Shorthand `host:port`, `host:port:user:pass`, or `user:pass@host:port`
  - **Proxy Rotation Pools:** Pass a comma-separated string (`"http://1.1.1.1:8080,socks5://2.2.2.2:1080"`) or `string[]` in the SDK to automatically rotate and fail over across proxies.
  - **Environment / Global Config:** Set `TUXSCOUT_PROXY`, `ALL_PROXY`, `HTTPS_PROXY`, or `HTTP_PROXY`, or call `setGlobalProxy(...)` in Node.js.

### 2. 5-Engine Live-Verified Image Search & Parallel Downloader (`tx-images`)
- **Engines:** Bing Async Image Grid, DuckDuckGo (`vqd` token + `i.js`), Yahoo Image Tiles, Wikimedia Commons API, and Openverse CC API.
- **Live Magic-Byte URL Probing (`verifyUrls: true` by default):** Over-fetches candidates and probes the first 512 bytes in parallel (`Range: bytes=0-2047`) to confirm `HTTP 200/206` and valid binary image magic bytes (`JPG`, `PNG`, `WEBP`, `GIF`, `AVIF`, `SVG`, `BMP`, `TIFF`). Broken `404` links, `403` hotlink-blocked URLs, watermarked stock hosts (`gettyimages`, `shutterstock`, `istockphoto`, `alamy`, `pinterest`), and HTML error pages are automatically filtered out.
- **Parallel Verified Downloader (`--download`):** Saves `N` verified images concurrently to disk with automatic `Referer` spoofing, thumbnail fallback, and a `manifest.json` index.

### 3. Fast Website-to-Markdown Fetcher (`tx-fetch`)
- Zero-DOM-dependency Readability-style main content extractor (`<article>`, `<main>`, noise/nav/footer/ad stripping).
- Converts HTML into clean, token-efficient GitHub-Flavored Markdown (preserving fenced code blocks with language tags, Markdown tables, headings, lists, blockquotes, and resolved absolute links).
- **Automatic JS-Wall / SPA Escalation:** Detects Cloudflare challenges, `<noscript>` SPA shells, or empty `#root`/`#app` containers and automatically escalates to the Low-RAM Puppeteer fetcher.

### 4. Ultra-Low-Memory Puppeteer JS Fetcher (`tx-browser`)
- **0 MB Idle RAM:** Singleton browser pool automatically closes headless Chromium/Chrome/Edge/Brave after 45s of inactivity.
- **Concurrency Semaphore & Page Recycling:** Caps simultaneous tabs (max 3) and recycles the browser every 25 pages to eliminate memory leaks.
- **Aggressive Request Interception:** Blocks `image`, `media`, `font`, `websocket`, and 35+ ad/analytics/tracker domains at the network layer—reducing page memory by **60–80%** and speeding up loads by **3–5x**.
- **Built-in Stealth Evasions:** Spoofs `navigator.webdriver`, `window.chrome`, plugins, and languages.

### 5. YouTube Video, Music, Search & Fast Transcript Tool (`tx-yt`)
- Powered by `yt-dlp` + `ffmpeg` (auto-discovers system binaries or self-bootstraps `yt-dlp` into `~/.tuxscout/bin/`).
- **Strict Long-Form vs. Shorts Filter (`videoType: 'long' | 'shorts' | 'all'`):**
  - By default (`--type long`), YouTube searches return **only normal long-form videos** (`> 60s`) and exclude YouTube Shorts.
  - Pass `--shorts` (or `--type shorts`) to search **only YouTube Shorts** (`<= 60s`), or `--type all` for both.
- **Fast Single-Track Transcript Extractor (`--mode transcript` / `--transcript`):** Extracts clean timestamped Markdown + plain text captions (`WEBVTT`, `SRT`, `JSON3`, `SRV1/XML`) in **~3–4 seconds** without hitting YouTube multi-track HTTP 429 rate limits.
- **High-Quality Video & Audio Downloads:** Supports `mp4`, `mkv`, `webm` video downloads (up to `2160p`/`1080p`/`720p`) and `mp3`, `m4a`, `opus`, `flac`, `wav` audio extraction.

### 6. 4-Stage Parallel Deep Research Pipeline (`tx-research`)
- **Stage 1 (Query Expansion):** Classifies intent and generates 2–4 orthogonal search angles.
- **Stage 2 (Parallel Multi-Engine Discovery):** Executes all angles concurrently across search engines (+ optional verified Image & YouTube discovery) and selects domain-diverse top URLs.
- **Stage 3 (Parallel Tiered Content Extraction):** Fetches candidate pages concurrently in Markdown, escalating JS-walled pages to Low-RAM Puppeteer and topping up failed pages from backup candidates.
- **Stage 4 (Passage Ranking & Synthesis):** Scores paragraphs by query-term & factual density and compiles a citation-backed Markdown Research Report (`[1]`, `[2]`, ...).

### 7. Zero-Browser Social Media Reconnaissance (`tx-social`)
- Extracts concise, structured JSON + Markdown from social platforms in `<1s` **without opening a browser window**:
  - **X / Twitter (`twitter`):** Single tweet/thread inspection (full text, quoted tweet, poll choices, direct `.mp4`/image media URLs, likes/retweets/replies/views), user profile + recent tweets hydrated newest-first by Snowflake ID, and topic tweet search.
  - **Instagram (`instagram`):** User profile (followers, following, post count, bio, HD avatar) + parallel discovery & hydration of recent posts/reels, single post/reel inspection (full caption, exact likes & comments, publish date, direct CDN media URL), and topic search.
  - **Reddit (`reddit`):** Subreddit feeds (`hot`, `new`, `top`), full post + nested comment tree extraction, and Reddit search via cookie-bootstrapped Atom feeds.
  - **GitHub (`github`):** Repository overview + latest releases + raw README excerpt, Issue/PR discussion thread + comments, and GitHub Trending repositories (`daily`, `weekly`, `monthly`, filterable by language).
  - **TikTok (`tiktok`):** Video caption, author metadata, and thumbnail via oEmbed + crawler metadata.

### 8. Live News & Tech Intelligence Aggregator (`tx-news`)
- Multi-source live news engine combining **Google News RSS** (`when: '1h' | '24h' | '7d' | '30d'`), **Hacker News Algolia API** (frontpage, search, and full nested story comment trees via `--hn-id`), **Reddit News** subreddits, and curated RSS feeds (**TechCrunch, ArsTechnica, The Verge, Lobsters, Phoronix, BleepingComputer, The Hacker News, MIT Tech Review AI, BBC World**) across 11 categories (`top`, `world`, `tech`, `ai`, `security`, `science`, `business`, `crypto`, `linux`, `opensource`, `hn`).

### 9. Zero-Browser Smart Knowledge Extractor (`tx-extract`)
- Turns any URL or shorthand target into structured JSON + Markdown without launching a browser:
  - **StackOverflow (`so:` or URL):** StackExchange API question body + top/accepted answers with code blocks.
  - **ArXiv (`arxiv:` or URL):** ArXiv Atom API paper metadata, authors, categories, PDF link, and full abstract.
  - **Wikipedia (`wiki:` or URL):** Wikimedia REST v1 structured summary + thumbnail.
  - **NPM (`npm:` or URL) & PyPI (`pypi:` or URL):** Package version, description, weekly downloads, dependencies, CLI binaries, and repo links.
  - **Social & Video URLs:** Automatically routes X/Twitter, Instagram, Reddit, GitHub, TikTok, HackerNews, and YouTube URLs to their native JSON/API extractors.
  - **Generic Webpages:** Extracts embedded `application/ld+json` (Schema.org), Next.js `__NEXT_DATA__` hydration state, HTML `<table>` data as JSON objects, RSS feeds, OpenGraph metadata, and clean Markdown.
  - **Automatic `tx-fetch` Interception:** Passing social or knowledge URLs to `tx-fetch` automatically routes through `smartExtract` (`method: 'api'`).

---

## 💻 CLI Usage & Examples

```bash
# System diagnostics & binary setup
tuxscout setup
tuxscout doctor

# Print AI Agent Skill Guide & CLI Cheatsheet
tuxscout skill
tx-skill --json

# Export JSON tool schemas for AI Agents / LLM Function Calling
tuxscout schema

# 1. Web Search & Advanced Search Operators
tx-search "Linux kernel 6.12 release notes" -n 5
tx-search --query "Node.js v24 features" --query "Node 24 vs Bun benchmarks" --json
tx-search "ebpf packet filter" --type pdf --site arxiv.org,kernel.org --exact "XDP" --exclude windows --time year

# 1b. IMDb Movie/TV/Actor Search & Full Dossier Extraction
tx-imdb "Inception" -n 5
tx-imdb tt1375666 --extract
tx-extract "imdb:Interstellar"

# 1c. PDF Search & Zero-Dependency PDF Text Extraction
tx-pdf "Attention Is All You Need" -n 5
tx-pdf "https://arxiv.org/pdf/1706.03762.pdf" --extract
tx-extract "pdf:https://arxiv.org/pdf/1706.03762.pdf"

# 1d. Using Universal Proxy IP (HTTP / HTTPS / SOCKS5 / Rotation Pool) with ANY command
tx-search "linux kernel" --proxy socks5://127.0.0.1:1080
tx-social "@nasa" --proxy 192.168.1.10:8080:myuser:mypass
tx-research "quantum error correction" --proxy "http://p1.example.com:8080,socks5://p2.example.com:1080"

# 2. Image Search (live-verified URLs) & Parallel Download
tx-images "Linux Tux penguin" -n 5
tx-images "mountain landscape wallpaper" --download --out ./wallpapers -n 5

# 3. Fast Website-to-Markdown Fetch (auto-escalates to JS browser if needed)
tx-fetch https://en.wikipedia.org/wiki/Linux --max-chars 5000
tx-fetch https://news.ycombinator.com --json

# 4. Ultra-Low-RAM Puppeteer JS Browser Fetch
tx-browser https://example.com --scroll --screenshot ./page.png
tx-browser https://react.dev --wait networkidle2

# 5. YouTube Search (Long-Form vs Shorts), Transcript, Audio & Video Download
tx-yt "linux kernel tutorial" -n 5                     # Default: Long-form videos only (no Shorts)
tx-yt "linux tips" --shorts -n 5                       # YouTube Shorts only (<=60s)
tx-yt "linux" --type all -n 5                          # Both Long-form and Shorts
tx-yt "https://www.youtube.com/watch?v=jNQXAC9IVRw" --mode transcript
tx-yt "https://www.youtube.com/watch?v=jNQXAC9IVRw" --mode info
tx-yt "https://www.youtube.com/watch?v=jNQXAC9IVRw" --mode audio --format mp3 --out ./music
tx-yt "https://www.youtube.com/watch?v=jNQXAC9IVRw" --mode video --height 720 --out ./videos

# 6. 4-Stage Parallel Deep Research
tx-research "Compare SQLite WAL mode vs DuckDB for embedded analytics" --depth quick
tx-research "Latest breakthroughs in solid state batteries" --depth standard --images --yt --save ./report.md

# 7. Zero-Browser Social Media Reconnaissance (X/Twitter, Instagram, Reddit, GitHub, TikTok)
tx-social "https://x.com/jack/status/20"                      # X/Twitter single tweet & thread stats
tx-social "@nasa" --platform twitter -n 5                     # X/Twitter profile + newest tweets
tx-social "@nasa" --platform instagram -n 5                   # Instagram profile + recent posts/reels
tx-social "https://www.instagram.com/p/DWq-l0gDegd/"          # Instagram post/reel full caption & stats
tx-social "r/linux" --sort hot -n 10                          # Reddit subreddit feed
tx-social "gh:trending" --lang typescript                     # GitHub trending repositories
tx-social "torvalds/linux"                                    # GitHub repo overview, releases & README

# 8. Live News & Tech Intelligence Aggregator
tx-news --category ai -n 10                                   # Live AI news (Google News + HN + Tech RSS)
tx-news "Linux kernel 6.14" --when 7d -n 10                   # Topic news search within last 7 days
tx-news --category hn -n 15                                   # Hacker News frontpage
tx-news --hn-id 8863                                          # Full Hacker News story + comment tree

# 9. Zero-Browser Smart Knowledge Extractor
tx-extract "https://stackoverflow.com/questions/11227809"     # StackOverflow question + top answers
tx-extract "so:node fetch timeout"                            # StackOverflow search + accepted answers
tx-extract "arxiv:1706.03762"                                 # ArXiv paper abstract, authors & PDF link
tx-extract "npm:tuxscout"                                     # NPM package metadata & weekly downloads
tx-extract "pypi:fastapi"                                     # PyPI package metadata & dependencies
tx-extract "wiki:Linux_kernel"                                # Wikipedia structured summary
```

---

## 🔌 MCP (Model Context Protocol) Server (`tx-mcp`)

TuxScout includes a built-in, zero-dependency MCP server over `stdio`. After installing globally, add `tx-mcp` to your AI Agent client (`claude_desktop_config.json`, Cursor, Windsurf, Cline, OpenClaw, etc.):

```json
{
  "mcpServers": {
    "tuxscout": {
      "command": "tx-mcp"
    }
  }
}
```

### Exposed MCP Tools
| Tool Name | Description |
|-----------|-------------|
| `web_search` | Multi-engine Web Search with 4-layer relevance filtering & parallel query angles |
| `image_search` | 5-engine Image Search with live parallel magic-byte URL verification |
| `image_download` | Parallel verified image downloader with Referer spoofing & `manifest.json` |
| `fetch_markdown` | Fast Website-to-Markdown converter with automatic Puppeteer JS fallback |
| `fetch_puppeteer_js` | Ultra-low-RAM Headless Chromium JS renderer with request interception |
| `youtube_tool` | YouTube Search (`long` / `shorts` / `all`), fast Transcripts, Info, Audio & Video downloader |
| `deep_research` | 4-Stage Parallel Multi-Angle Search & Deep Research dossier generator |
| `social_intel` | Zero-browser Social Media scraper for X/Twitter, Instagram, Reddit, GitHub & TikTok |
| `news_intel` | Live News & Tech Intelligence aggregator (Google News, Hacker News, Reddit, Curated RSS) |
| `smart_extract` | Zero-Browser Smart Knowledge Extractor (StackOverflow, ArXiv, Wiki, NPM, PyPI, JSON-LD/Tables) |

---

## 🌐 HTTP REST API Server (`tx-serve`)

Start the HTTP API server:
```bash
tx-serve --port 8787
# Optional API key / secret protection:
AGENT_TOOLS_SECRET="your-secret-key" tx-serve --port 8787
```

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/health` | Health check, RAM usage, and detected system binaries |
| `GET` | `/schema` | Full OpenAI / Anthropic / MCP JSON tool schemas |
| `GET/POST` | `/search` | Multi-engine Web Search (`{ query, queries?, n?, engine?, searchType?, site?, filetype?, pdfOnly?, timeRange?, proxy? }`) |
| `GET/POST` | `/imdb` | IMDb Movie/TV/Person Search or Dossier Extraction (`{ query, id?, extract?, n?, proxy? }`) |
| `GET/POST` | `/pdf` | Multi-engine PDF Search or Direct PDF Text Extraction (`{ query, url?, extract?, n?, maxChars?, proxy? }`) |
| `GET/POST` | `/images` | Live-verified Image Search (`{ q, n?, safe?, verifyUrls?, download?, outputDir? }`) |
| `POST` | `/images/download` | Parallel verified image downloader (`{ query?, images?, outputDir, count? }`) |
| `GET/POST` | `/fetch` | Website-to-Markdown (`{ url }` or batch `{ urls: string[], autoPuppeteer?: true }`) |
| `GET/POST` | `/puppeteer` | Low-RAM Headless Chromium JS fetch (`{ url, waitUntil?, scrollToBottom?, screenshotPath? }`) |
| `GET/POST` | `/yt` | YouTube tool (`{ input, mode?: 'search'|'transcript'|'video'|'audio'|'info', videoType?: 'long'|'shorts'|'all' }`) |
| `POST` | `/research` | 4-Stage Parallel Deep Research (`{ query, subQueries?, depth?: 'quick'|'standard'|'deep' }`) |
| `GET/POST` | `/social` | Zero-browser Social Media scraper (`{ target, platform?, action?, limit?, sort?, language? }`) |
| `GET/POST` | `/news` | Live News & Tech Intelligence (`{ query?, category?, when?, limit?, hnItemId? }`) |
| `GET/POST` | `/extract` | Zero-browser Smart Knowledge Extractor (`{ target, limit?, maxChars? }`) |

---

## 📦 Node.js / TypeScript SDK Usage

```typescript
import {
  webSearch,
  imageSearch,
  downloadImages,
  fetchMarkdown,
  fetchWithPuppeteer,
  youtubeTool,
  deepResearch,
} from 'tuxscout';

// 1. Web Search
const search = await webSearch({ queries: ['Linux 6.12 kernel', 'Linux 6.12 benchmarks'], n: 10 });

// 2. Live-Verified Image Search
const images = await imageSearch({ query: 'Tux penguin', n: 5, verifyUrls: true });

// 3. Fast Website to Markdown
const page = await fetchMarkdown({ url: 'https://en.wikipedia.org/wiki/Linux', maxChars: 8000 });

// 4. YouTube Search (Long-form only) & Fast Transcript
const videos = await youtubeTool({ input: 'linux kernel tutorial', mode: 'search', videoType: 'long', searchLimit: 5 });
const transcript = await youtubeTool({ input: 'https://www.youtube.com/watch?v=jNQXAC9IVRw', mode: 'transcript' });

// 5. Multi-Stage Deep Research
const dossier = await deepResearch({ query: 'Compare io_uring vs epoll performance', depth: 'standard' });
console.log(dossier.reportMarkdown);
```

---

## 📄 License

Released under the [MIT License](./LICENSE).