OmniMCP — Universal Web-to-MCP Gateway

Turn any website into a first-class MCP toolset in seconds.

OmniMCP is a universal MCP server that transforms any URL into a rich set of MCP tools. It can scrape content, perform deep crawls, run LLM-powered structured extraction, auto-detect and mount REST APIs via OpenAPI specs, and more — all exposed through a single SSE endpoint that any MCP client can connect to.

Philosophy

Modern AI agents are incredibly powerful at reasoning, but they are still fundamentally blind to the live web. Most agents either:

• Rely on brittle, pre-built scrapers for a handful of known sites, or

• Have no web access at all.

OmniMCP solves the "last mile" problem: any website → structured, tool-callable data without writing custom code for each domain.

It embodies three core principles:

• Universality — One server, any site.

• Zero-config intelligence — LLM extraction + automatic OpenAPI mounting when available.

• Production-grade reliability — Caching, retries, JS rendering, anti-bot hardening, and clean Markdown output.

Pain Points Solved

Use Cases

1. Research & Competitive Intelligence

# Agent wants latest pricing from a competitor
result = await scrape_url("https://competitor.com/pricing")

2. Automated Monitoring

Deep crawl a documentation site nightly and diff changes.

3. Dynamic API Integration

Point connect_site at a new SaaS product → instantly gain 20–80 new MCP tools from their OpenAPI spec.

4. Structured Data Extraction at Scale

Use smart_extract_page with natural language instructions + JSON schema to turn any page into typed data.

5. Internal Tooling for Agents

Give your agent a "web superpower" without maintaining dozens of site-specific scrapers.

Features

• 10 production tools (see table below)

• FastMCP 3.3.1 SSE transport (compatible with Claude Desktop, Cursor, Hermes, etc.)

• crawl4ai 0.8.6 backend with full JavaScript rendering

• Automatic OpenAPI mounting — turns REST APIs into MCP tools

• LLM-powered extraction (optional, via your local oMLX endpoint)

• SQLite result cache with namespace-based invalidation

• Proxy support for anti-bot sites

• Docker-ready

Tool Reference

Quick Start

# 1. Clone
git clone https://github.com/shagghiesuperstar/omni-mcp.git
cd omni-mcp

# 2. Create venv (Python ≥ 3.10 required)
python3 -m venv .venv
source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Install browser
playwright install chromium

# 5. Run crawl4ai post-install
crawl4ai-setup

# 6. Configure (optional but recommended)
cp .env.example .env
# Edit .env and set LLM_BASE_URL if you want smart_extract_page to use your local LLM

# 7. Run
python server.py
# → SSE endpoint live at http://0.0.0.0:8055/sse

Hermes Agent Integration

Add to your config.yaml:

mcp_servers:
  omni_mcp:
    transport: sse
    url: http://localhost:8055/sse

Cross-machine (M4 ↔ M5 via Tailscale):

mcp_servers:
  omni_mcp:
    transport: sse
    url: https://your-m5-tailscale-hostname:8055/sse

Advanced Usage Examples

LLM-Powered Structured Extraction

result = await smart_extract_page(
    url="https://example.com/products",
    instruction="Extract all product names, prices, and availability as a list of objects",
    schema='{"type":"object","properties":{"products":{"type":"array","items":{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"string"},"in_stock":{"type":"boolean"}}}}}}'
)

Deep Crawl with Limits

pages = await crawl_site(
    base_url="https://docs.example.com",
    max_depth=3,
    max_pages=50
)

Auto-Mounting an API

await connect_site("https://api.example.com")
# → Instantly mounts all endpoints as MCP tools under the domain prefix

Opportunities & Future Directions

• Agent-native web layer — Become the default web ingestion layer for every major agent framework.

• Self-updating tool registry — Automatically discover and mount new APIs as sites publish OpenAPI specs.

• Multi-modal extraction — Combine LLM + vision for screenshot-based or PDF-heavy sites.

• Enterprise deployment — Add authentication, rate limiting, and audit logging for internal agent fleets.

• Community scrapers — Allow users to contribute high-quality CSS/LLM extraction templates.

License

MIT — use freely in personal and commercial agent projects.

Built with precision by Pi (Superstar_ops) • May 2026