Molavi Web Intelligence Engine (MWIE)

One Engine. Complete Web Intelligence.

English | فارسی | Türkçe | Azərbaycan | العربية

1. Project Introduction

Molavi Web Intelligence Engine (MWIE) is an open-source, self-hosted web intelligence platform designed for Technical SEO, Generative Engine Optimization (GEO), AI Search Indexing, and Autonomous AI Agents.

Modern search has fundamentally transitioned from simple keyword matching to generative AI engines (Perplexity, ChatGPT Search, Google Gemini Grounding). Existing web scrapers focus either only on raw HTML extraction or simple Markdown conversion without understanding semantic entity graphs, citations, or technical SEO health.

MWIE unifies high-throughput HTTP crawling, headless JavaScript rendering, resilient DOM parsing, technical SEO scorecards, and AI-readiness analysis into a modular engine accessible via REST API and native Model Context Protocol (MCP).

Related MCP server: google-seo-mcp

2. Core Features

• Unified Multi-Adapter Extraction:

  • Native Async HTTP Adapter: Ultra-fast (sub-20ms) scraping with zero browser overhead.

  • Playwright Headless Browser Adapter: Renders client-side dynamic JavaScript single-page apps (SPAs).

  • Firecrawl Clean-Room Adapter: Decoupled REST adapter for self-hosted or cloud Firecrawl clusters.

• Intelligent Evidence-Based Routing:

  • Automatically selects the optimal extraction backend based on page type, JavaScript requirements, and real-time performance telemetry.

  • Seamless automatic fallbacks to alternative adapters upon network or rendering anomalies.

• Resilient Content & Markdown Generation:

  • Heuristic boilerplate noise removal (headers, footers, navigation, cookie banners).

  • Produces hierarchical, clean Markdown optimized for LLM prompts and RAG chunking.

• Native Model Context Protocol (MCP) Server:

  • Exposes 12 native tools directly to AI agent platforms (Hermes, Hamzad, Claude Desktop, Cursor).

• Cost & Resource Efficiency:

  • 100% self-hosted and open-source. Requires zero paid APIs to operate.

3. SEO-Native Intelligence Capabilities

Technical SEO is a first-class citizen inside MWIE:

• Indexability & Crawl Control: Meta robots (noindex, nofollow), X-Robots-Tag, HTTP status code validation.

• Metadata Auditing: Meta title and description length evaluation against search snippet thresholds.

• Canonical Consistency: Rel canonical extraction and cross-check against target request URLs.

• Heading Hierarchy: Deep analysis of H1-H6 tags, single H1 compliance, and semantic nesting.

• Image Accessibility: Total image audit, alt text detection, and missing alt reporting.

• Link Graph Intelligence: Separation and counting of internal vs external URLs.

• Structured Data: Full Schema.org JSON-LD extraction, microdata detection, and schema type mapping.

• Social Tags: Open Graph (og:*) and Twitter Cards (twitter:*) completeness analysis.

• Internationalization: Hreflang tag parsing and target language validation.

• Content-to-HTML Ratio: Diagnostic ratio calculation highlighting text density vs boilerplate code.

• Weighted SEO Scorecard: Generates an objective 0–100 score accompanied by actionable diagnostic recommendations.

4. GEO-Native Intelligence Capabilities

Generative Engine Optimization (GEO) prepares content for generative discovery:

• Brand & Entity Recognition: Extracts Brand, Organization, Person, and Product entities from JSON-LD and content text.

• Citation Intelligence: Maps external references, source links, blockquote citations, and author attribution.

• Answer-Oriented Q&A Extraction: Automatically discovers FAQPage structures and interrogative heading/answer pairs.

• Semantic Content Chunking: Splits body copy into clean, token-estimated chunks tailored for vector embeddings and RAG pipelines.

• AI-Readiness Scorecard: 0–100 score quantifying entity density, citation grounding, Q&A clarity, and chunkability for generative search engines.

5. Architecture Overview

┌────────────────────────────────────────────────────────────────────────┐
│                        AI Agents & Client Apps                         │
│           (Hermes, Hamzad, Claude Desktop, Cursor, REST Clients)       │
└──────────────────┬─────────────────────────────────┬───────────────────┘
                   │ MCP (Stdio/SSE)                 │ REST API (HTTP)
┌──────────────────▼─────────────────────────────────▼───────────────────┐
│               Molavi Web Intelligence Engine (MWIE) Core               │
├────────────────────────────────────────────────────────────────────────┤
│ ┌──────────────────────────┐    ┌────────────────────────────────────┐ │
│ │  Intelligent Router      │    │  Request Queue & Robots Manager    │ │
│ └────────────┬─────────────┘    └────────────────────────────────────┘ │
│              │                                                         │
│ ┌────────────▼───────────────────────────────────────────────────────┐ │
│ │                   Unified Extraction Adapters                      │ │
│ │  • Native HTTP Adapter    • Playwright Adapter   • Firecrawl API   │ │
│ └────────────┬───────────────────────────────────────────────────────┘ │
│              │ Raw HTML                                                │
│ ┌────────────▼───────────────────────────────────────────────────────┐ │
│ │              Parsing & Content Cleaning Pipeline                   │ │
│ │  • Noise Stripper    • HTML Parser   • Markdown Generator          │ │
│ └────────────┬───────────────────────────────────────────────────────┘ │
│              │ Clean DOM & Text                                        │
│ ┌────────────▼────────────────────────┬──────────────────────────────┐ │
│ │    Technical SEO Analyzer           │     GEO & Entity Analyzer    │ │
│ │  • Canonical & Metadata             │  • Brand Entity Extraction   │ │
│ │  • Headings & Image Alts            │  • Citations & Author Map    │ │
│ │  • JSON-LD Structured Data          │  • Q&A Knowledge Blocks      │ │
│ │  • 0-100 Weighted Scorecard         │  • AI-Readiness Scorecard    │ │
│ └─────────────────────────────────────┴──────────────────────────────┘ │
│                                                                        │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │   Memory / Redis TTL Cache   •   SSRF Security & Allowlist Gates   │ │
│ └────────────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┘

6. Installation & Quick Start

Prerequisites

• Python 3.10+

• Linux, macOS, or Windows

Installation

git clone https://github.com/tmolavi/molavi-web-intelligence.git
cd molavi-web-intelligence
python3 -m pip install -e .

For development (including testing tools):

python3 -m pip install -e ".[dev]"

Quick Start via Python SDK

import asyncio
from mwie.crawler.engine import crawl_engine
from mwie.core.models import ExtractionOptions


async def main():
    options = ExtractionOptions(render_js=False, extract_seo=True, extract_geo=True)
    result = await crawl_engine.extract_single("https://molavi.pro", options)

    print(f"Title: {result.seo.meta_title}")
    print(f"SEO Score: {result.seo.seo_score}/100")
    print(f"AI Readiness: {result.geo.ai_readiness_score}/100")
    print(f"Entities: {[e.name for e in result.geo.entities]}")


asyncio.run(main())

7. REST API Usage

Start the REST API server:

uvicorn mwie.api.app:app --host 0.0.0.0 --port 8000

Visit interactive API documentation at: http://localhost:8000/docs

Example: Extract & Analyze URL

curl -X POST http://localhost:8000/api/v1/extract \
  -H "Content-Type: application/json" \
  -d '{"url": "https://molavi.pro"}'

Example: Technical SEO Audit

curl -X POST http://localhost:8000/api/v1/seo \
  -H "Content-Type: application/json" \
  -d '{"url": "https://molavi.pro"}'

Example: Compare Competitor Pages

curl -X POST http://localhost:8000/api/v1/compare \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://molavi.pro", "https://example.com"]}'

8. Model Context Protocol (MCP) Integration

MWIE provides a native Python MCP server exposing 12 tools for autonomous agents:

Adding MWIE to Claude Desktop

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "molavi-web-intelligence": {
      "command": "python3",
      "args": ["-m", "mwie.mcp.server"]
    }
  }
}

9. Docker Deployment

Deploy with Docker:

docker build -t molavi-web-intelligence .
docker run -p 8000:8000 molavi-web-intelligence

Or using Docker Compose:

docker compose up -d

10. Security & Responsible Crawling

MWIE includes strict built-in security features:

• SSRF Protection: By default, RFC 1918 private networks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), localhost (127.0.0.1), and loopback addresses are strictly blocked.

• Robots.txt Adherence: Automatically respects robots.txt disallow rules.

• Domain Allowlists/Blocklists: Configurable in settings or via environment variables.

11. Empirical Benchmarks

Empirical measurements on Linux ARM64 (100 iterations per component):

• HTML Parsing & Noise Cleaning: 13.10 ms avg latency (76.4 ops/sec, ~4.7 MB peak RAM)

• Markdown Generator: 2.56 ms avg latency (390.7 ops/sec, ~2.6 MB peak RAM)

• Technical SEO Analyzer: 11.25 ms avg latency (88.9 ops/sec, ~2.4 MB peak RAM)

• GEO & Entity Graph Engine: 8.56 ms avg latency (116.8 ops/sec, ~6.4 MB peak RAM)

See docs/BENCHMARKS.md for full reproducible details.

12. License & Attribution

• Released under the Apache License 2.0. See LICENSE for terms.

• Independent clean-room architecture developed by Taqi Molavi.

• Third-party research references and licenses are documented in CREDITS.md and THIRD_PARTY_LICENSES.md.

13. Author & Ecosystem

Author: Taqi Molavi
Website: https://molavi.pro
Ecosystem Integrations: Hermes Agent | Hamzad Digital Twins