# Media2HTML: A VLM-Free Pure Compiler for Multimodal Agents

Disclaimer: Please understand this is V0.01 and still very experimental. It is the product of a personal theory and nothing more.

A tool that converts images, videos, and audio into structured, cacheable HTML representations optimized for text-based LLM reasoning.

**V0.0.1 — Pure Compiler Architecture**

---

## Concept

Modern multimodal AI agents need to reason about media files, but LLMs are fundamentally text-based. Traditional approaches — passing raw base64 images or unstructured VLM captions — suffer from massive token overhead, hallucination risk, and GPU contention.

Media2HTML eliminates all of this. It extracts structured facts (objects, text, colors, spatial relations, semantic tags) using deterministic algorithms, then compiles them into semantic HTML that LLMs understand natively. **Zero Vision-Language Models in the runtime loop.**

**Key benefits:**
- **(To be dertimined) token reduction** vs. raw base64 images
- **0 GB VRAM** — all extraction runs on CPU, leaving GPU free for the text LLM
- **100% deterministic** — same input always produces the same output
- **Zero hallucination** — no generative model in the pipeline
- **Instant caching** — repeated file analysis returns in <1ms

---

## V0.0.1 Stack

| Component | Technology | Purpose |
|-----------|-----------|---------|
| **Object Detection** | YOLO-World + SAHI | Open-vocabulary detection with sliced inference for 4K+ images |
| **Z-Axis Layout** | Depth-Anything V2 | Foreground / midground / background categorization |
| **Semantic Tags** | CLIP (openai/clip-vit-base-patch32) | Zero-shot scene tagging (rain, night, cherry blossom, etc.) |
| **Text Extraction** | RapidOCR | CPU-only OCR with paragraph grouping |
| **Visual Maps** | ASCILINE | Token-optimized 512-cell grid for visual reasoning |
| **Audio** | Whisper + Pyannote | Transcription and speaker diarization |
| **Video** | scenedetect + FFmpeg | Scene detection with keyframe extraction |

---

## Setup

### 1. Install System Dependencies

```bash
sudo apt install ffmpeg portaudio19-dev
```

### 2. Install Python Packages

```bash
pip install -e .
```

### 3. Environment Variables (Optional)

```bash
export HF_TOKEN="hf_..."  # Required for Pyannote audio diarization
```

---

## Usage

### As a Python Library

```python
from media2html import media_to_html

# Image
html = media_to_html("path/to/image.jpg", mode="rich")

# Video (with interleaved audio)
html = media_to_html("path/to/video.mp4", mode="compact")

# Audio
html = media_to_html("path/to/audio.wav", mode="minimal")
```

### As an MCP Server

```bash
# Start the MCP server
python3 -m media2html.mcp_server

# Or run directly
python3 media2html/mcp_server.py
```

Agents call the `transcode_media` tool to analyze any local media file.

---

## Modes

| Mode | Content | Token Count |
|------|---------|-------------|
| `minimal` | Caption + top objects + colors | ~200 |
| `compact` | + OCR + visual grid | ~800 |
| `rich` | + spatial relations, semantic tags, depth scene, accent colors | ~2500 |

---

## Output Format

Media2HTML produces semantic HTML that leverages LLM pretraining priors:

```html
<image-summary width="1376" height="768" source="photo.png">
  <caption>Scene contains: car, person, umbrella. Text detected: 'LOFUGA'. Dominant color: #303048. Tags: rain, wet road.</caption>
  <spatial-graph>
    <rel>car is left of person</rel>
    <rel>person is above car</rel>
  </spatial-graph>
  <semantic-tags>
    <t>rain</t>
    <t>wet road</t>
  </semantic-tags>
  <scene>
    <foreground>car</foreground>
    <background>person, umbrella</background>
  </scene>
  <accent-colors>
    <c hex="#e0c4c4" pct="0.04" label="vibrant"/>
  </accent-colors>
  <objects>
    <obj label="car" bbox="0.277,0.438,0.758,1.000"/>
  </objects>
  <text-regions>
    <t bbox="0.350,0.750,0.412,0.794">LOFUGA</t>
  </text-regions>
  <colors>
    <c hex="#303048" pct="0.08"/>
  </colors>
</image-summary>
```

---

## Architecture

```
media2html/
├── pyproject.toml           # Package configuration
├── requirements.txt         # Dependencies
├── README.md                # This file
├── TECHNICAL_WHITEPAPER.md  # Full architecture documentation
├── test_media2html.py       # Test suite
└── media2html/
    ├── __init__.py          # Package exports
    ├── cache.py             # Disk-based caching (diskcache + MD5)
    ├── html_builder.py      # Data model + HTML generation
    ├── pipeline.py          # Main extraction pipeline
    ├── mcp_server.py        # MCP server for agent integration
    └── extractors/
        ├── __init__.py
        ├── vision.py        # YOLO-World + Depth-Anything + CLIP + RapidOCR
        ├── audio.py         # Whisper + Pyannote
        └── video.py         # Scene detection + keyframes
```

---

## Performance

| Metric | Value |
|--------|-------|
| Token reduction vs. raw base64 | to be determined |
| Extraction latency (minimal) | ~200ms | Estimated
| Extraction latency (rich) | ~700ms | Estimated
| GPU VRAM required | 0 GB (all CPU) |
| Deterministic output | 100% | Estimated
| Hallucination risk | 0% | Estimated
| Cache hit latency | <1ms | Estimated

---

## License Creative Commons