OmniExtract MCP
# OmniExtract MCP & Apify Actor (`omni-extract-mcp`)
[](https://apify.com/poweragy/omni-extract-mcp)
[](https://apify.com/poweragy/omni-extract-mcp)
[](https://opensource.org/licenses/MIT)
**Universal, Stateless, Cookie-Free Content & Transcript Extractor for AI Agents (Claude Desktop, Cursor, Windsurf, OpenClaw, n8n, Make, and Apify x402).**
š **Run Instantly in the Cloud (Zero Setup, Pay-Per-Result / x402 Ready)**: [**https://apify.com/poweragy/omni-extract-mcp**](https://apify.com/poweragy/omni-extract-mcp)
---
## Why OmniExtract?
Standard `WebFetch` and browser tools fail with `403 Forbidden`, login walls, or empty JavaScript shells when given social media or video URLs. **OmniExtract** converts complex links into clean, LLM-ready **Markdown + Timestamped Transcripts + Watermark-Free Media URLs** in one call:
1. **X (Twitter) Posts & Long-Form X Articles**: Full Draft.js block rendering (`# / ## / ###`, ordered/unordered lists, code blocks, blockquotes, quoted posts) and media extraction **with zero personal cookies required**.
2. **Douyin & TikTok Watermark-Free Extraction**: Pure-algorithm `x-secsdk-web-signature` signing (`ttwid` + `uifid` + MD5 salt) + 302 shortlink resolution + binary MIME magic sniffing (`WebP`/`JPEG`/`PNG`) ā **zero headless browser overhead**.
3. **YouTube & Bilibili 3-Layer Subtitle Engine**: Extracts native & auto-generated VTT/SRT captions into deduplicated full-text transcripts and timestamped segments in ~2 seconds without downloading heavy video files.
4. **Anti-Bot Web & GitHub Markdown**: Automatic GitHub raw README/blob fast-path, semantic HTML-to-Markdown conversion, bot-wall detection, and Jina Reader fallback.
---
## Option 1: Hosted Cloud API & MCP via Apify Store (Recommended)
No local dependencies (`ffmpeg` / `yt-dlp`) or server maintenance required:
- **Apify Store Page**: [https://apify.com/poweragy/omni-extract-mcp](https://apify.com/poweragy/omni-extract-mcp)
- **REST API Call**:
```bash
curl -X POST "https://api.apify.com/v2/acts/poweragy~omni-extract-mcp/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://x.com/ai_chuhai/status/2037437653071835440",
"https://v.douyin.com/0kH2cQVrf0o/"
],
"include_transcript": true
}'
```
---
## Option 2: Local MCP Server (Claude Desktop / Cursor / Windsurf)
```bash
pip install git+https://github.com/solarfans/omni-extract-mcp.git
```
Add to your `claude_desktop_config.json` or `.cursor/mcp.json`:
```json
{
"mcpServers": {
"omni-extract": {
"command": "python3",
"args": ["-m", "omni_extract.mcp_server", "--mode", "stdio"]
}
}
}
```
### Exposed MCP Tools
- `omni_extract_url(url, include_transcript=True, output_format="markdown")`: Extract a single URL into structured Markdown or JSON.
- `omni_extract_batch(urls, include_transcript=True)`: Concurrently extract up to 10 URLs in parallel.
- `omni_check_quota(api_key="")`: Check remaining API key credits or daily quota.
---
## Option 3: Self-Hosted HTTP / Remote MCP Gateway
```bash
python3 -m omni_extract.mcp_server --mode http --port 8080
```
- `GET /v1/health`: Health & capability check
- `POST /mcp`: Streamable HTTP MCP JSON-RPC 2.0 endpoint
- `POST /v1/extract`: Direct REST single URL extraction
- `POST /v1/extract/batch`: Direct REST concurrent batch extraction
- `GET /v1/quota`: Check remaining API key credits or daily anonymous free quota
---
## Option 4: Quick Directory Integration (Smithery / mcpservers.org / Cursor / Claude Desktop)
### Hosted Apify MCP Server (Zero Local Install)
Connect directly using Apify's hosted MCP endpoint (`https://mcp.apify.com/?actors=poweragy/omni-extract-mcp`):
```json
{
"mcpServers": {
"omni-extract-cloud": {
"command": "npx",
"args": ["-y", "mcp-remote", "https://mcp.apify.com/?actors=poweragy/omni-extract-mcp"],
"env": {
"APIFY_TOKEN": "YOUR_APIFY_TOKEN"
}
}
}
}
```
### Troubleshooting & Common Error Fixes (FAQ)
- **[Fix: X (Twitter) Articles Truncated / JS Shell](./docs/fix-x-twitter-article-scraping-truncated.md)**: Standard HTTP scrapers stop at the X JS shell or truncated preview. Pass the full `x.com/<user>/status/<id>` URL to `omni_extract_url` ā OmniExtract resolves the underlying Draft.js blocks (`article.content.blocks`) and entity media automatically without cookies.
- **[Fix: Douyin Watermark & Cookie Expired Errors](./docs/fix-douyin-watermark-cookie-expired-mcp.md)**: Pass `v.douyin.com` or `douyin.com/video/...` links directly. OmniExtract resolves 302 redirects and computes `x-secsdk-web-signature` natively, returning clean Markdown + direct media URLs without headless browser overhead.
- **[Fix: YouTube Transcript Blocked on Cloud IPs](./docs/fix-youtube-transcript-ip-blocked-mcp.md)**: Runs a 3-layer native + `yt-dlp` caption pipeline without downloading full MP4 streams.
- **Empty `{}` health checks vs billed items**: Empty input `{}` runs execute a read-only connectivity check and skip PPE item billing (`apify_ppe_skipped: true`), while real extraction runs emit transparent `charged` / `charge_failed` counters in the Actor summary.
- **Registry manifests included**: [`smithery.yaml`](./smithery.yaml) and [`server.json`](./server.json).
TDQS
Scored across 3 tools
Each tool targets a distinct operation: single-URL extraction, batched extraction, and quota checking. There is no realistic way to confuse batch extraction with a single extraction or with a credit check, and the descriptions reinforce the boundary.
All three names follow a strict server-prefixed verb_noun pattern: omni_extract_url, omni_extract_batch, omni_check_quota. The shared 'omni_' namespace plus consistent verb forms makes the set fully predictable.
Three tools is a tight, well-scoped surface for a stateless extraction service, and each one earns its place. Nothing redundant, nothing gratuitous.
The core workflow is fully covered: extract one, extract many, and monitor credits. Minor gaps exist, such as no way to select output format/depth, inspect supported-platform info, or check the status of a long-running batch, but agents can work around these.