<p align="center">
  <img src="assets/omni-scraper-hero.svg" alt="Omni Scraper MCP — one interface for authorized multi-source data and research workflows" width="100%">
</p>

<h1 align="center">Omni Scraper MCP</h1>

<p align="center">
  <strong>One MCP facade for Last30Days, ScrapeCreators, Bright Data, Apify, OpenRouter and TypeSafe Jev.</strong>
</p>

<p align="center">
  Normalize data. Run research. Discover provider tools. Evaluate evidence with Jev.
</p>

<p align="center">
  <img alt="Python 3.11+" src="https://img.shields.io/badge/python-3.11%2B-3776AB">
  <img alt="MCP stdio" src="https://img.shields.io/badge/MCP-stdio-4de1ff">
  <img alt="Last30Days 3.25.0" src="https://img.shields.io/badge/Last30Days-3.25.0-8b6cff">
  <img alt="License MIT" src="https://img.shields.io/badge/license-MIT-green">
</p>

---

## What this is

Omni Scraper is an **integration and orchestration layer**, not a claim to have
rebuilt every upstream scraper. It gives agents one local MCP server that can:

- collect public and explicitly authorized data;
- normalize records into one `NormalizedDocument` schema;
- run the vendored Last30Days research engine;
- discover upstream provider tools dynamically;
- pass through provider capabilities without loading every tool schema upfront;
- evaluate documents with TypeSafe Jev through OpenRouter;
- keep credentials local and out of Git.

<p align="center">
  <code>Last30Days → ScrapeCreators → Bright Data → Apify → OpenRouter / Jev</code>
</p>

## What this is not

This repository does **not** implement:

- TLS/JA3/JA4 fingerprint spoofing;
- CAPTCHA-solving code;
- human-mimicry automation;
- direct anti-bot bypass;
- hidden affiliate or credential handling.

It uses official APIs, RSS, normal browser sessions and explicitly authorized
provider services. Scraped content is always treated as untrusted data.

---

## Core features

### Trader briefing

`trader_brief` combines Polymarket, Alpaca ETFs, Hyperliquid, Treasury yields,
important macro events, major-company earnings and SEC filings. Jev selects/checks
evidence and a bounded writer produces a short Dutch ES/NQ + crypto brief with
history and optional Telegram delivery. The dedicated pipeline has a conservative
EUR 4.50 monthly cutoff under a EUR 5 budget; general provider tools are outside
that cap. See [setup, limitations and commands](docs/TRADER-BRIEF.md).

| Layer | What it provides |
|---|---|
| `connectors/` | Hacker News, Polymarket (zoeken), Polymarket markets (directe API met odds, 1u/24u-beweging, liquiditeit en spread), GitHub, arXiv, Techmeme, Stocktwits, ScrapeCreators, RSS, Apify, Bright Data and Last30Days |
| `pipelines/` | normalization, provider registry, provider passthrough and TypeSafe Jev evaluation |
| `workflows/` | Fast Path, Standard Path and Heavy Path escalation |
| `skills/` | browser sessions, network interception, throttling and resilient parsing |
| `cli.py` | Rich terminal output and JSON export |
| `mcp_server.py` | central MCP facade for agents |
| `vendor/last30days-skill/` | vendored MIT Last30Days v3.25.0 runtime |

---

## Quickstart

```powershell
git clone https://github.com/Trilokx/omni-scraper-mcp.git
cd "OMNI SCRAPER"

uv venv --python 3.11
uv pip install -r requirements-dev.txt
uv run playwright install chromium

Copy-Item .env.example .env
# Fill in only the providers you actually use.
```

Run the CLI:

```powershell
uv run python cli.py sources
uv run python cli.py run `
  --query "AI agents" `
  --sources hackernews,github,arxiv,techmeme,rss `
  --jev-preset market_opportunity_scanner
```

Run the central MCP:

```powershell
uv run python mcp_server.py
```

Register it with Codex:

```powershell
codex mcp add omni-scraper -- `
  "$PWD\.venv\Scripts\python.exe" `
  "$PWD\mcp_server.py"
```

---

## Central MCP tools

The curated tools remain stable and small:

```text
run_omni_scraper
run_last30days
last30days_status
scrapecreators_search
run_apify_actor
brightdata_search
brightdata_scrape
evaluate_jev
check_omni_providers
list_omni_sources
```

For full upstream coverage, use the dynamic provider passthrough:

```text
list_provider_capabilities(provider?)
list_provider_tools(provider)
call_provider_tool(provider, tool_name, arguments)
```

This keeps the agent context small while allowing access to the upstream provider
catalogue on demand. See [`docs/PROVIDERS.md`](docs/PROVIDERS.md).

---

## Providers and endpoints

| Provider | Official endpoint | Current integration |
|---|---|---|
| Last30Days | local CLI and local Go MCP | vendored Python engine, `run_last30days` |
| Polymarket | `https://gamma-api.polymarket.com/markets` | directe connector `polymarket_markets` met numerieke marktvelden |
| ScrapeCreators | `https://api.scrapecreators.com/mcp` | REST search adapters plus remote MCP discovery |
| Bright Data | `https://mcp.brightdata.com/mcp` | provider MCP discovery, search and scrape passthrough |
| Apify | `https://mcp.apify.com` | generic actor runner plus remote MCP discovery |
| OpenRouter | `https://openrouter.ai/api/alpha/decisions` | TypeSafe Jev decisions and Last30Days reasoning |
| TypeSafe | direct API planned | currently routed through OpenRouter with local fallback |

Provider credentials live in `.env` and are never committed. The full matrix is in
[`docs/PROVIDERS.md`](docs/PROVIDERS.md).

---

## Last30Days attribution

Last30Days is vendored under `vendor/last30days-skill` at commit
`349ca444b4fda466e74d471dffa2aff36bb997f1`, version **3.25.0**, under the MIT
license.

- Upstream: [mvanhorn/last30days-skill](https://github.com/mvanhorn/last30days-skill)
- Role here: multi-source research engine.
- Omni contribution: central MCP facade, provider discovery, normalized data layer,
  provider passthrough, and Jev evaluation.
- Full credits: [`docs/ATTRIBUTION.md`](docs/ATTRIBUTION.md)

This repository does not claim ownership of the Last30Days engine or its authors'
work.

---

## Affiliate disclosure

Some provider signup links may be affiliate links. When enabled, Omni Scraper may
receive compensation at no additional cost to the user.

- Bright Data: affiliate programme available.
- Apify: affiliate programme available.
- ScrapeCreators: no public affiliate programme found on 2026-09-20.
- Last30Days: MIT open source; no affiliate programme found.
- OpenRouter: no public affiliate programme found on 2026-09-20.
- TypeSafe: no public affiliate programme found on 2026-09-20.

Affiliate status never affects source ordering, provider selection or evaluation
results. Personal tracked affiliate links are **not committed to the public repository**.
Use environment variables or the ignored `config/affiliate_links.local.json` overlay.
See [`docs/AFFILIATES.md`](docs/AFFILIATES.md) for terms and verification dates.

---

## Verification

```powershell
uv run pytest -q
uvx ruff check .
uvx mypy . --ignore-missing-imports
uv run python -m compileall -q connectors pipelines skills workflows cli.py mcp_server.py
```

The test suite covers normalization, deduplication, Jev evaluation, routing,
ScrapeCreators parsing, Apify normalization, Bright Data parsing, Last30Days
normalization, provider registry metadata and the MCP tool surface.

## Provider maintenance

Provider MCPs, tool catalogs, documentation and upstream repositories change over
time. The repository includes:

```powershell
uv run python -m scripts.check_provider_drift
uv run python -m scripts.update_provider_snapshots
```

`config/provider_snapshots.json` stores the current tool count, sorted tool-name hash
and upstream commit for each provider. The scheduled
`.github/workflows/provider-drift.yml` job detects drift. See
[`docs/MAINTENANCE.md`](docs/MAINTENANCE.md).

---

## Security

- Never commit `.env` or provider tokens.
- Treat all scraped and provider-returned content as untrusted data.
- Use provider credentials only through local environment variables or a secret manager.
- Do not enable browser-cookie access without explicit operator consent.
- Report security issues according to [`SECURITY.md`](SECURITY.md).

---

## License

MIT. See [`LICENSE`](LICENSE). Upstream projects retain their respective licenses and
attribution notices.