Skip to main content
Glama
OmniExtract MCP

OmniExtract MCP

README.md
# OmniExtract MCP & Apify Actor (`omni-extract-mcp`)

[![Apify Store](https://img.shields.io/badge/Apify_Store-poweragy%2Fomni--extract--mcp-00C853?logo=apify)](https://apify.com/poweragy/omni-extract-mcp)
[![MCP Protocol](https://img.shields.io/badge/MCP-2024--11--05-blue)](https://apify.com/poweragy/omni-extract-mcp)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**Universal, Stateless, Cookie-Free Content & Transcript Extractor for AI Agents (Claude Desktop, Cursor, Windsurf, OpenClaw, n8n, Make, and Apify x402).**

šŸ‘‰ **Run Instantly in the Cloud (Zero Setup, Pay-Per-Result / x402 Ready)**: [**https://apify.com/poweragy/omni-extract-mcp**](https://apify.com/poweragy/omni-extract-mcp)

---

## Why OmniExtract?

Standard `WebFetch` and browser tools fail with `403 Forbidden`, login walls, or empty JavaScript shells when given social media or video URLs. **OmniExtract** converts complex links into clean, LLM-ready **Markdown + Timestamped Transcripts + Watermark-Free Media URLs** in one call:

1. **X (Twitter) Posts & Long-Form X Articles**: Full Draft.js block rendering (`# / ## / ###`, ordered/unordered lists, code blocks, blockquotes, quoted posts) and media extraction **with zero personal cookies required**.
2. **Douyin & TikTok Watermark-Free Extraction**: Pure-algorithm `x-secsdk-web-signature` signing (`ttwid` + `uifid` + MD5 salt) + 302 shortlink resolution + binary MIME magic sniffing (`WebP`/`JPEG`/`PNG`) — **zero headless browser overhead**.
3. **YouTube & Bilibili 3-Layer Subtitle Engine**: Extracts native & auto-generated VTT/SRT captions into deduplicated full-text transcripts and timestamped segments in ~2 seconds without downloading heavy video files.
4. **Anti-Bot Web & GitHub Markdown**: Automatic GitHub raw README/blob fast-path, semantic HTML-to-Markdown conversion, bot-wall detection, and Jina Reader fallback.

---

## Option 1: Hosted Cloud API & MCP via Apify Store (Recommended)

No local dependencies (`ffmpeg` / `yt-dlp`) or server maintenance required:

- **Apify Store Page**: [https://apify.com/poweragy/omni-extract-mcp](https://apify.com/poweragy/omni-extract-mcp)
- **REST API Call**:
  ```bash
  curl -X POST "https://api.apify.com/v2/acts/poweragy~omni-extract-mcp/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "urls": [
        "https://x.com/ai_chuhai/status/2037437653071835440",
        "https://v.douyin.com/0kH2cQVrf0o/"
      ],
      "include_transcript": true
    }'
  ```

---

## Option 2: Local MCP Server (Claude Desktop / Cursor / Windsurf)

```bash
pip install git+https://github.com/solarfans/omni-extract-mcp.git
```

Add to your `claude_desktop_config.json` or `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "omni-extract": {
      "command": "python3",
      "args": ["-m", "omni_extract.mcp_server", "--mode", "stdio"]
    }
  }
}
```

### Exposed MCP Tools
- `omni_extract_url(url, include_transcript=True, output_format="markdown")`: Extract a single URL into structured Markdown or JSON.
- `omni_extract_batch(urls, include_transcript=True)`: Concurrently extract up to 10 URLs in parallel.
- `omni_check_quota(api_key="")`: Check remaining API key credits or daily quota.

---

## Option 3: Self-Hosted HTTP / Remote MCP Gateway

```bash
python3 -m omni_extract.mcp_server --mode http --port 8080
```

- `GET /v1/health`: Health & capability check
- `POST /mcp`: Streamable HTTP MCP JSON-RPC 2.0 endpoint
- `POST /v1/extract`: Direct REST single URL extraction
- `POST /v1/extract/batch`: Direct REST concurrent batch extraction
- `GET /v1/quota`: Check remaining API key credits or daily anonymous free quota

---

## Option 4: Quick Directory Integration (Smithery / mcpservers.org / Cursor / Claude Desktop)

### Hosted Apify MCP Server (Zero Local Install)
Connect directly using Apify's hosted MCP endpoint (`https://mcp.apify.com/?actors=poweragy/omni-extract-mcp`):

```json
{
  "mcpServers": {
    "omni-extract-cloud": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://mcp.apify.com/?actors=poweragy/omni-extract-mcp"],
      "env": {
        "APIFY_TOKEN": "YOUR_APIFY_TOKEN"
      }
    }
  }
}
```

### Troubleshooting & Common Error Fixes (FAQ)
- **[Fix: X (Twitter) Articles Truncated / JS Shell](./docs/fix-x-twitter-article-scraping-truncated.md)**: Standard HTTP scrapers stop at the X JS shell or truncated preview. Pass the full `x.com/<user>/status/<id>` URL to `omni_extract_url` — OmniExtract resolves the underlying Draft.js blocks (`article.content.blocks`) and entity media automatically without cookies.
- **[Fix: Douyin Watermark & Cookie Expired Errors](./docs/fix-douyin-watermark-cookie-expired-mcp.md)**: Pass `v.douyin.com` or `douyin.com/video/...` links directly. OmniExtract resolves 302 redirects and computes `x-secsdk-web-signature` natively, returning clean Markdown + direct media URLs without headless browser overhead.
- **[Fix: YouTube Transcript Blocked on Cloud IPs](./docs/fix-youtube-transcript-ip-blocked-mcp.md)**: Runs a 3-layer native + `yt-dlp` caption pipeline without downloading full MP4 streams.
- **Empty `{}` health checks vs billed items**: Empty input `{}` runs execute a read-only connectivity check and skip PPE item billing (`apify_ppe_skipped: true`), while real extraction runs emit transparent `charged` / `charge_failed` counters in the Actor summary.
- **Registry manifests included**: [`smithery.yaml`](./smithery.yaml) and [`server.json`](./server.json).

TDQS

A3.6/5.0

Scored across 3 tools

Disambiguation5/5

Each tool targets a distinct operation: single-URL extraction, batched extraction, and quota checking. There is no realistic way to confuse batch extraction with a single extraction or with a credit check, and the descriptions reinforce the boundary.

Naming Consistency5/5

All three names follow a strict server-prefixed verb_noun pattern: omni_extract_url, omni_extract_batch, omni_check_quota. The shared 'omni_' namespace plus consistent verb forms makes the set fully predictable.

Tool Count5/5

Three tools is a tight, well-scoped surface for a stateless extraction service, and each one earns its place. Nothing redundant, nothing gratuitous.

Completeness4/5

The core workflow is fully covered: extract one, extract many, and monitor credits. Minor gaps exist, such as no way to select output format/depth, inspect supported-platform info, or check the status of a long-running batch, but agents can work around these.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues