Omni Scraper MCP

What this is

Omni Scraper is an integration and orchestration layer, not a claim to have rebuilt every upstream scraper. It gives agents one local MCP server that can:

• collect public and explicitly authorized data;

• normalize records into one NormalizedDocument schema;

• run the vendored Last30Days research engine;

• discover upstream provider tools dynamically;

• pass through provider capabilities without loading every tool schema upfront;

• evaluate documents with TypeSafe Jev through OpenRouter;

• keep credentials local and out of Git.

Related MCP server: Witness

What this is not

This repository does not implement:

• TLS/JA3/JA4 fingerprint spoofing;

• CAPTCHA-solving code;

• human-mimicry automation;

• direct anti-bot bypass;

• hidden affiliate or credential handling.

It uses official APIs, RSS, normal browser sessions and explicitly authorized provider services. Scraped content is always treated as untrusted data.

Core features

Trader briefing

trader_brief combines Polymarket, Alpaca ETFs, Hyperliquid, Treasury yields, important macro events, major-company earnings and SEC filings. Jev selects/checks evidence and a bounded writer produces a short Dutch ES/NQ + crypto brief with history and optional Telegram delivery. The dedicated pipeline has a conservative EUR 4.50 monthly cutoff under a EUR 5 budget; general provider tools are outside that cap. See setup, limitations and commands.

Quickstart

git clone https://github.com/Trilokx/omni-scraper-mcp.git
cd "OMNI SCRAPER"

uv venv --python 3.11
uv pip install -r requirements-dev.txt
uv run playwright install chromium

Copy-Item .env.example .env
# Fill in only the providers you actually use.

Run the CLI:

uv run python cli.py sources
uv run python cli.py run `
  --query "AI agents" `
  --sources hackernews,github,arxiv,techmeme,rss `
  --jev-preset market_opportunity_scanner

Run the central MCP:

uv run python mcp_server.py

Register it with Codex:

codex mcp add omni-scraper -- `
  "$PWD\.venv\Scripts\python.exe" `
  "$PWD\mcp_server.py"

Central MCP tools

The curated tools remain stable and small:

run_omni_scraper
run_last30days
last30days_status
scrapecreators_search
run_apify_actor
brightdata_search
brightdata_scrape
evaluate_jev
check_omni_providers
list_omni_sources

For full upstream coverage, use the dynamic provider passthrough:

list_provider_capabilities(provider?)
list_provider_tools(provider)
call_provider_tool(provider, tool_name, arguments)

This keeps the agent context small while allowing access to the upstream provider catalogue on demand. See docs/PROVIDERS.md.

Providers and endpoints

Provider credentials live in .env and are never committed. The full matrix is in docs/PROVIDERS.md.

Last30Days attribution

Last30Days is vendored under vendor/last30days-skill at commit 349ca444b4fda466e74d471dffa2aff36bb997f1, version 3.25.0, under the MIT license.

• Upstream: mvanhorn/last30days-skill

• Role here: multi-source research engine.

• Omni contribution: central MCP facade, provider discovery, normalized data layer, provider passthrough, and Jev evaluation.

• Full credits: docs/ATTRIBUTION.md

This repository does not claim ownership of the Last30Days engine or its authors' work.

Affiliate disclosure

Some provider signup links may be affiliate links. When enabled, Omni Scraper may receive compensation at no additional cost to the user.

• Bright Data: affiliate programme available.

• Apify: affiliate programme available.

• ScrapeCreators: no public affiliate programme found on 2026-09-20.

• Last30Days: MIT open source; no affiliate programme found.

• OpenRouter: no public affiliate programme found on 2026-09-20.

• TypeSafe: no public affiliate programme found on 2026-09-20.

Affiliate status never affects source ordering, provider selection or evaluation results. Personal tracked affiliate links are not committed to the public repository. Use environment variables or the ignored config/affiliate_links.local.json overlay. See docs/AFFILIATES.md for terms and verification dates.

Verification

uv run pytest -q
uvx ruff check .
uvx mypy . --ignore-missing-imports
uv run python -m compileall -q connectors pipelines skills workflows cli.py mcp_server.py

The test suite covers normalization, deduplication, Jev evaluation, routing, ScrapeCreators parsing, Apify normalization, Bright Data parsing, Last30Days normalization, provider registry metadata and the MCP tool surface.

Provider maintenance

Provider MCPs, tool catalogs, documentation and upstream repositories change over time. The repository includes:

uv run python -m scripts.check_provider_drift
uv run python -m scripts.update_provider_snapshots

config/provider_snapshots.json stores the current tool count, sorted tool-name hash and upstream commit for each provider. The scheduled .github/workflows/provider-drift.yml job detects drift. See docs/MAINTENANCE.md.

Security

• Never commit .env or provider tokens.

• Treat all scraped and provider-returned content as untrusted data.

• Use provider credentials only through local environment variables or a secret manager.

• Do not enable browser-cookie access without explicit operator consent.

• Report security issues according to SECURITY.md.

License

MIT. See LICENSE. Upstream projects retain their respective licenses and attribution notices.