# OmniMCP — Universal Web-to-MCP Gateway

**Turn any website into a first-class MCP toolset in seconds.**

OmniMCP is a universal MCP server that transforms any URL into a rich set of MCP tools. It can scrape content, perform deep crawls, run LLM-powered structured extraction, auto-detect and mount REST APIs via OpenAPI specs, and more — all exposed through a single SSE endpoint that any MCP client can connect to.

## Philosophy

Modern AI agents are incredibly powerful at reasoning, but they are still fundamentally blind to the live web. Most agents either:
- Rely on brittle, pre-built scrapers for a handful of known sites, or
- Have no web access at all.

OmniMCP solves the "last mile" problem: **any website → structured, tool-callable data** without writing custom code for each domain.

It embodies three core principles:
- **Universality** — One server, any site.
- **Zero-config intelligence** — LLM extraction + automatic OpenAPI mounting when available.
- **Production-grade reliability** — Caching, retries, JS rendering, anti-bot hardening, and clean Markdown output.

## Pain Points Solved

| Pain Point | How OmniMCP Solves It |
|------------|-----------------------|
| Agents can't access dynamic/JS-heavy sites | Playwright + crawl4ai with `wait_for_js`, user simulation, overlay removal |
| Every new site requires custom scraper code | `scrape_url`, `crawl_site`, `smart_extract_page`, `css_extract_page` work on any URL |
| REST APIs are invisible to agents unless manually integrated | `connect_site` auto-detects OpenAPI/Swagger specs and mounts them as native MCP tools |
| Repeated scraping wastes tokens and time | Built-in SQLite cache with configurable TTL (default 1h) |
| Structured data extraction is inconsistent | Optional LLM-based extraction (`smart_extract_page`) or deterministic CSS selectors |
| Agents need both raw content and precise data | Multiple extraction modes (Markdown, JSON schema, CSS, keyword search) |

## Use Cases

### 1. Research & Competitive Intelligence
```python
# Agent wants latest pricing from a competitor
result = await scrape_url("https://competitor.com/pricing")
```

### 2. Automated Monitoring
Deep crawl a documentation site nightly and diff changes.

### 3. Dynamic API Integration
Point `connect_site` at a new SaaS product → instantly gain 20–80 new MCP tools from their OpenAPI spec.

### 4. Structured Data Extraction at Scale
Use `smart_extract_page` with natural language instructions + JSON schema to turn any page into typed data.

### 5. Internal Tooling for Agents
Give your agent a "web superpower" without maintaining dozens of site-specific scrapers.

## Features

- **10 production tools** (see table below)
- **FastMCP 3.3.1** SSE transport (compatible with Claude Desktop, Cursor, Hermes, etc.)
- **crawl4ai 0.8.6** backend with full JavaScript rendering
- **Automatic OpenAPI mounting** — turns REST APIs into MCP tools
- **LLM-powered extraction** (optional, via your local oMLX endpoint)
- **SQLite result cache** with namespace-based invalidation
- **Proxy support** for anti-bot sites
- **Docker-ready**

### Tool Reference

| Tool                    | Description                                                                 | Best For                     |
|-------------------------|-----------------------------------------------------------------------------|------------------------------|
| `connect_site`          | Auto-detects OpenAPI spec or falls back to scrape mode                      | First contact with new domain |
| `scrape_url`            | Clean Markdown + metadata from any URL                                      | General browsing             |
| `crawl_site`            | Deep crawl with depth & page limits                                         | Documentation, blogs, sites  |
| `smart_extract_page`    | Natural-language + optional JSON schema extraction via LLM                  | Structured data (pricing, products, etc.) |
| `css_extract_page`      | Fast, deterministic CSS selector extraction                                 | High-volume, stable layouts  |
| `search_page`           | Keyword search with surrounding context                                     | Finding specific information |
| `get_links`             | All internal + external hyperlinks                                          | Graph building, discovery    |
| `fetch_api_endpoint`    | Raw HTTP client (GET/POST/PUT/DELETE)                                       | Direct API calls             |
| `list_tools`            | Currently mounted tools                                                     | Self-inspection              |
| `clear_cache`           | Namespace or full cache flush                                               | Forcing fresh data           |

## Quick Start

```bash
# 1. Clone
git clone https://github.com/shagghiesuperstar/omni-mcp.git
cd omni-mcp

# 2. Create venv (Python ≥ 3.10 required)
python3 -m venv .venv
source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install browser
playwright install chromium

# 5. Run crawl4ai post-install
crawl4ai-setup

# 6. Configure (optional but recommended)
cp .env.example .env
# Edit .env and set LLM_BASE_URL if you want smart_extract_page to use your local LLM

# 7. Run
python server.py
# → SSE endpoint live at http://0.0.0.0:8055/sse
```

## Hermes Agent Integration

Add to your `config.yaml`:

```yaml
mcp_servers:
  omni_mcp:
    transport: sse
    url: http://localhost:8055/sse
```

Cross-machine (M4 ↔ M5 via Tailscale):

```yaml
mcp_servers:
  omni_mcp:
    transport: sse
    url: https://your-m5-tailscale-hostname:8055/sse
```

## Advanced Usage Examples

### LLM-Powered Structured Extraction

```python
result = await smart_extract_page(
    url="https://example.com/products",
    instruction="Extract all product names, prices, and availability as a list of objects",
    schema='{"type":"object","properties":{"products":{"type":"array","items":{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"string"},"in_stock":{"type":"boolean"}}}}}}'
)
```

### Deep Crawl with Limits

```python
pages = await crawl_site(
    base_url="https://docs.example.com",
    max_depth=3,
    max_pages=50
)
```

### Auto-Mounting an API

```python
await connect_site("https://api.example.com")
# → Instantly mounts all endpoints as MCP tools under the domain prefix
```

## Opportunities & Future Directions

- **Agent-native web layer** — Become the default web ingestion layer for every major agent framework.
- **Self-updating tool registry** — Automatically discover and mount new APIs as sites publish OpenAPI specs.
- **Multi-modal extraction** — Combine LLM + vision for screenshot-based or PDF-heavy sites.
- **Enterprise deployment** — Add authentication, rate limiting, and audit logging for internal agent fleets.
- **Community scrapers** — Allow users to contribute high-quality CSS/LLM extraction templates.

## License

MIT — use freely in personal and commercial agent projects.

---

**Built with precision by Pi (Superstar_ops) • May 2026**